Working WITH the Robots • Perfect Audience Blog

As advertisers, we interact with data-driven automation, machine learning, and artificial intelligence on a daily basis. Google Adwords, Facebook Advertising, Perfect Audience, and other major ad platforms all use advanced technology to drive results. How does it work? And how can you as an agency owner or marketer truly use these tools to your advantage? Watch the webinar and Q&A below to understand the basics of Advertising AI – no experience or background required.

Speakers:

Shiv Gupta
Founder, U of Digital
Previously: AOL, Criteo

Dr. Dan Becker
Founder, Decision.ai
Previously: Kaggle (Google), DataRobot, FTC Economist

Jalali Hartman
Director, Perfect Audience A.I. Lab
Previously: Robauto.ai, Yovia, MarketingExperiments

View the slide deck:

AI-Lab-How-to-Optimize-for-AI-Webinar-Slides

Webinar Transcription:

Hey, everybody, happy Thursday. This is Jalali Hartmann, I’m the Director of the AI Lab for perfect audience. We’re just giving everybody a second to log on here, and we’re going to get started. It should be able to hear us and sees blue screen in the background slide.

Everybody like about, it looks like people still logging on here will give me about 30 more seconds.

Eric’s going to go ahead and make some interesting kick us off here.

Sounds good.

All right.

So, yeah. Thanks for everybody for coming. Very excited about today’s topic, and its panel.

Just to give you everybody some background and why we put this together in pretty short order, I might add. I saw this article that was written a couple of weeks ago in ad exchanger.

And I was fascinated by it, and I’d send it over to Jalali, who heads up our AI lab, and the subject of the article. Or, it was really, what caught me. It’s like, you know, calling BS and AI. Like. I gotta click on that. I gotta look at it, read it.

And the problem is real, right?

The, you know, machine learning and AI is a hard thing to solve when you’re trying to do it right, but it’s extraordinarily valuable when you do.

So, it’s a mix of human and machines to make this thing work and to, and to do it right and to get it right.

And so, really, we wanted to bring on some experts, and and we’ve got 2 or 3 of them on the phone now, doctor Dan Becker, he’s a leading expert on machine learning is really helped innovate and educate thousands through Google and and FTC and, you know, sort of the list goes on.

And then Dan has worked with different types of iterations on TensorFlow, Just, you know, the kind of background and the kind of mind, you know, that we get to bring to the table on these events, just know, is amazing to me. So, thank you, Dan, for, for joining us. And and Shift … is also, he’s sort of old school advertising. You know, it came out of AOL and advertising dot com and ….

And so, you know, he knows the subject inside and out and, and has been, you know, in the Industry four, for quite awhile and has been sort of leading the charge and some of this stuff. So, appreciate, you’re also joining us.

Uh.

So let’s just start in right now, what’s sort of get right into it. So, Shiv, you know, can you tell us a little bit about why it is that you wrote the article, Why you’re calling AI, you know, BS, you know? I kinda what was the thought behind that?

Yeah, for sure. So first of all, thanks for having me on Hi everybody, excited to be here and excited, excited to be talking to you guys about this topic. You know, my experience to your point, Eric. I’ve always been an ad tech, right, and I’ve primarily been on the sales side of ad tech, So I was selling at … dot com at AOL and …. And, you know, frankly, when you’re selling in this industry, you kind of figure two things out Pretty quickly.

You figure out the things that you’re talking about and in what ways they are genuine and sincere and, you know, in which ways maybe they’re a little misguided or a little bit, you know, like, a little bit of misdirection. Then, you very, very quickly figure out what the competitors are doing, as well, in terms of Misdirection. Right, Because that’s, That’s Your job. You’re selling against them all the time. And after leaving …, I started a company called U of Digital, and we are focused on bringing structured education to the space now. And in that effort, you know, I’ve, I’ve been able to, first of all, have a much more objective viewpoint of the industry without like sitting at one company and having to sell one technology or solution. And I’ve also, you know, one of our missions at the company is to make digital advertising a better space through education, right? And so, that’s kind of what led me to write this.

I constantly feel like, you know, part of our mission is to illuminate some of the things in the industry that are that are think are troubling and causing, causing inefficiencies. And, you know, one of the main things in this article that I talk about is the difference between real AI and misdirection. Right? And so that was one of the main reasons. I wrote this, I’m excited to be here in particular Because I know you know you guys with Jolly on the on the line and with doctor Jim Packer on the line. Like you guys are, actual experts, right. I, I’ve just been an ad tech, I’m an ad tech guy. And I have some thoughts and opinions that I wrote about in this article. But I’m excited to be here and learn from you guys because your knee deep in the AI itself. I have a bunch of questions for you guys throughout today that I’ll probably probably be bringing up about definitions. And differentiation between things like neural networks, and machine learning, and AI and deep learning, right? There’s all these terms at the industry loves to throw around, I’d love to learn, I’m excited to just learn from or more, from, from you guys here today.

Awesome.

And so We have Jalali on the phone, too, and as many of you already know, and it looks like people are, keep joining the webinar, jalali heads up. Our AI Lab, here at perfect audience, and has just, you know, is really driven the, the technology forward, as we’ve been building some of these things out over the last nine months now. So we’re super excited to have him on board, so Jalali, you, want to, you want to kick us off here?

Yeah, so, thanks for opening up. So we’re gonna get ready to Dan’s on here, and Dan is someone that I’ve known for, actually, my whole career, and he’s just, he’s brilliant. But he’s brilliant, in particular, Machine learning. So he’s one of the foremost experts I would consider, I want to have them on here because it’s not necessarily just looking at it from an advertiser perspective, but how does this actually work so that maybe we as a community, we could get a little bit more aligned in terms of understanding what we’re even talking about. So, before we get going like, one quote that really stuck out to me and got me started in this years ago, was, the CEO of Google came out and said, he thought he always be more impactful than electricity.

If you kinda really think about that, that’s a tremendous statement. What I think is that it’s actually probably bigger than that. We also are kind of using it the wrong way or thinking about the wrong way or how misperceptions of how it works. So with that, I’m going to turn over to Dan. Dan’s going to do some definitions, and he has a quick quiz. A machine learning’s route, because you’re Wheaties this morning now. He’s just gonna go through some kind of basic terms and shows from his perspective, like what is and what isn’t possible.

So I know there’s been a few questions about definitions I have on the lower right here.

Really, a Venn diagram of AI and machine learning. Deep learning neural nets this in a moment. Because people just want to know what are the same. So, starting on the outer most part of this, L, so, unfortunately, AI has a definition, which is ambiguous, and then different people have different definitions, which is in contrast to, some things are further into the circle, which are different types of AI, which are very concrete definitions. I and many others now think of AI is just, you get a computer to do something, that’s pretty stark. And that’s, you know, not not a terribly useful definition. Different people might think smarter, not smart, but that, that’s a pretty common definition, and a lot of people in research are now using it. It’s, it’s easy for us to understand. Let’s get more concrete, though.

Mean, between 90 and 95% of AI applications today are using machine learning. And that is something which is really well defined. And after this slide, I’m going to show you just a couple of examples of what is machine learning. What are the parts of it? How do they fit together? But machine learning is really a way for us to look at patterns a historical data and use that to make predictions. Prospectively. I’ll go into more detail about that. The other one that you hear frequently is deep learning. Deep learning is a type of machine learning. It’s just a very specific way of looking at the patterns in historical data, and using that to make predictions.

The thing, which is potentially special about deep learning, is that, I guess they’re probably Yeah, all right, two main things that are important to know.

The first is that we’ve tried machine learning with images, and with video, but really, with the visual information for decades. And we’ve only become reasonably good at it, using this particular type of data, images and video.

And the same is true of text and audio, Really only become good at that in the last seven years for images, and even less than that for text. And that’s because of deep learning techniques. Again, just using historical data to make predictions on new about what? How to classify new data. You’ll also be kept neural networks. Neural networks is really. I think that’s a synonym for deep learning once. Upon a time decades ago, when they were trying to figure out how to how to make this work, it was inspired by connections to how they thought the human brain worked. Now, most people who really know what they’re doing, I think that, that’s not a useful analogy. It’s just a term that’s less leftover from the past, and you should be careful not to get too sidetracked. When people talk about, oh, it’s like a replacement human.

It’s not, It’s doing some something which, you know, certainly, you could almost call pattern recognition. If you go to the next slide, I’ll show you the parts of how machine learning works. And deep learning is not that much different.

So this is just a table of data, You know, you might stored in a database. You might store it in a CSV file. You can even have this type of data stored in in a spreadsheet. And there are a few terms that are really essential to getting the right mindset for how machine learning works. So first is unit of observation that we’ve got.

In this case, a row, it’s just, how do we divide up the data from when we want to make predictions. What sort of the unit that we’re using to make predictions. In this case, we could say, it’s each row. What does it mean? It could be the behavior of a given customer on a given date. So in this case, these first two columns are how we define the unit of observation. And deep learning. Frequently, one image, we make prediction for one image at a time. Whenever you, whenever you’re making these predictions, you want to make sure that your … every time we make a prediction for every time we look at that one unit and say, what are the patterns across these units. What is what does that sort of atomic unit that we’re working in?

The second piece is, I’ve said that this is useful for prediction. So we need something called a prediction target. In this case, that could be how much is someone going to spend.

Because it’s a given customer on a given date. How much is this customer going to spend on a state?

In this case, spend would be the predicted target.

And the last is everything that is used to make that prediction.

So here we say, we got some historical data, a bunch of customer date pairs for each of them. We have, how much they spent.

And then we’ve got different things that we could use to find patterns that help us, to text. And I’ll show you in the next slide.

I really like the simplest algorithm that I know. it’s not, really.

It’s not the best way to make these predictions. Is the worst place. But I think I don’t get the ideas in your head. Or, alright, how could we divide data used throughout the day to find patterns in it and then make predictions moving forward. So this is something called a decision tree. Again, it’s a pretty naive way of doing machine learning, but we start with 50,000 customers. And then we’ve got an algorithm, which was breaks them into different groups based on their behaviors. Does airway, which is sensible answer the first time that we split our data into two groups? We can say, All right.

For each row in that dataset, did they visit more than two times F? So we we have a group, which is on the right. If they did, the less than two times, we have a booth. It’s on the left. So, here, you have 10000 people on the date of their business. in a dataset, they had made, at most, two previous visits. The site, If we want to make a prediction, or someone in that sort of bucket of users, will just take the average value of people in our historical data that went to to that branch of the tree. In the future. You need to make a prediction for how much for this person spent if they come to our website today. That made us the two visits. They go down that branch of the tree or prediction, seven bucks. You can also have some next thing here. And in many cases, this one that’s more deeply and say, well, at 40,000 people, who make more than two visits, And within that group, we say, well, what’s the total in the previous purchases?

And then we could yeah. People who wanted to visit never made previous purchase.

On average, when they came on any given day, spent $11 on that day. And that’s how we make a prediction for someone in this group. again, know, oh, we can do things that are that are much savvier. We can make a bunch of these different trees that split in different ways than the average of those. There are better ways you can make these predictions. But at art, we’re just finding the patterns in the historical data we collected.

Then, from that, when we need to make a prediction for someone, we look at this model.

Sorry, given the patterns, what’s our prediction?

What’s the common people in that, in that sort of bucket or behavior in the past? That’s how I make predictions. So quick to kind of giving you the structure, I say, yeah, let’s go to the next slide.

I don’t want to do, but I want to just briefly make a clarification about what do these models do, and what are the limitations? So you saw in the last slide, really, what we’re doing is prediction.

There are many thing is that you see, where you are actually doing, something quite clever. It’s not obvious the best prediction, but in every case, machine, learning, around 99.9% of cases, we’ve just added a very clever way to to use prediction. Sometimes. It seems unlike prediction to, let’s say, a chatbot.

That seems like it’s not predicting, but actually, it’s collected historical data. How did a personal response, I get a text? And then it responds in a similar way. So it’s, It’s doing a prediction for, how would a human to have responded? No.

If you have probably the most famous AI model, this AlphaGo is this gave this program that could be the best humans and support can go to predicting for each potential move. Am I going to wait if I make them? So it’s it’s really focused on prediction.

In many cases, that’s not the prediction is not enough to tell you what you should actually do. You need to bring in other sources of knowledge, And in many cases, that’s knowledge that lives in someone’s head, You know, what’s our markup? Maybe that’s something where you could go and ask someone, but it’s not in your predicted data. And so, to go from, I can do site predicted to I could do something which is prescriptive, It helps me achieve. Some business level KPI is a technology that really is not widespread, it happens to be, you know, I work, at Decision AI or and founded, this is an AI. And that is exactly a tool so that you can bring in Arabic got some models that make predictions. But then, I could say, what are the other sources of data I have?

Where you get the stress that knowledge I have integrate those, and try and come up with better decisions, that optimize bottom level, bottom line, business, KPIs. So let’s do a quick quiz. Hey, doctor, doctor Becker, do you mind if I ask a quick question here, before we get into the quiz. Yeah. So, so I think this is the previous slide, is a really important point, right, Of machine learning, is predictive, right. In nature, versus prescriptive. And I think, like, you know, one thing I was trying to get to, what the article that I wrote, was trying to figure out where to call BS, right? And I think pretty much everyone in the advertising space, or the ad tech industry, is out there, touting some sort of machine learning, AI type of solution. But I know, my personal belief is, a lot of them don’t really have that, and it’s, it’s more so smoke, and mirrors, and maybe there’s some very, very basic algorithms behind the scenes, as opposed to actual AI or machine learning. So, I would love to hear from you, as the expert. You know, I think this is really helpful, predictive versus prescriptive.

But are there any other ways, or how would you specifically advise folks to be able to make a distinction between AI? and then, you know, maybe just a basic algorithm, or some some very basic kind of calculations that are happening in the background?

Yeah, this comes back. When I add that, when I had that Venn diagram, I had to be tough.

I think that we overemphasize whether it’s the importance of whether something is AI or not. Sure. I’ve got a problem, and someone could say, if I need to nail nail them, project construction, so I’m going to say I have an AI solutions on because I have a hammer, or whether it’s AI or her. I just chair, is it going to get this thing adults?

And so, I think that, uh, know, what do you say, is it the astronaut?

I think, I think we’re better off calling the astronaut BS on does it solve this problem well? And if the thing that powers it, AI are not AI.

Aye.

Could maybe be academically interesting, but I actually think it’s frequently a mistake for us to even worry about that. If someone says, I’ve got this thing, it’s AI.

That’s probably not enough for me to say, because it could be AI, could be doing that. As far as me, being the best human it go at that can help you.

No cell more, probably not.

Yeah, so, I absolutely agree on the outcomes piece. That’s the most important thing that I met, that, you know, ultimately matters, and, honestly, because there is so much, I think, mister Action, not just around AI, but around a lot of things in this industry. I think, you know, buyers more than ever need to just be focused on, hey, does this solve my problem?

But, my only counter to that, my only caveat would be like, I think, you know, if you’re if you’re a buyer and you have a pic of, you know, 10, 10 different solutions to solve one specific problem, and you can kind of understand that one of those solutions or many of those solutions have a tendency to misdirect. Maybe more than some others. I think that could be part of a decision making process, right? Because I think still, in this industry, even though so much is automated so, much is also still relationship driven, entrust driven, right? And, so, that’s the only reason I asked. But, I absolutely 100% agree with your point, that fire, should be focused on outcomes, more than anything.

Yup.

I was recently at a pitch competition.

Every startup, at the end of it, kinda said, Oh, and we are developing AI, or we use AI, It’s that kind of thing that I think we have to be careful for it. Because it’s, You can ask them, what does it do, what is it going to do it? Because, really, they have no idea.

So, I think that’s to your point shifts in advertising, you do your making these decisions.

And I think one of the things we’re really trying to do with all of this, project in general, is just, let’s cut, let’s break it down to what really matters. That is getting a profitable conversion, or getting you’re getting business out of the marketing spend. So that’s, kind of what we want to focus on, but there’s a lot of work that goes into it. So, OK, So go back to you, Dan.

I guess, Mine in the right place, again.

Yeah, So let me let me say your phrase, that’s ship, sort of touched on this issue of when it’s predicted, that sort of thing, more of a predictive or prescriptive?

Yeah, I think that there are cases where good predictions. If I’m not know, if I’m going to predict how much someone’s going to The thing about two different websites, one, is, I’m going to predict how much time was spent today. And they’re never going to come back. And I, that’s just thereby explained, it really is a one-time purchase.

If I can predict how much they’re going to spend today, it’s probably actually a good measure of, customer lifetime value of the Same thing. If I mark up to 100%, there’s not. A lot of other information I need to bring in. Just. A prediction could be enough.

Yeah, I have another business.

And I’ve got a lot of different products that I sell Anda.

They have different markups and I’m trying to build a relationship.

And the value of having someone come to make a purchase today is not just the value of that purchase today. But it’s the fact that now that you’re with me, they’ve got some other customer lifetime value, even the value of forming that relationship depends on what’s the way that I interact with them in the future.

Now, it’s sort of things are easy to predict. There’s no simple answer to that.

And that’s when you say we need to do something which embeds a lot of other knowledge. And there’s no way to automate that stuff. Like if If the technology to selling due to use it, well, it needs to reach inside your head. It’s like, how are you going to change how you How you market to your customers, that teacher. Yeah. Somebody who says, We’re going to tell you the customer. lifetime value, They’re not getting that information out to you.

It just can’t be, even consensually can’t be done.

Just, so you have a, you want to do a quick?

this is, you know, I think we always learn best, like, thinking about these hands-on. So let’s do a couple of quick questions. I’ll give you three to NaN to think about it. And then I’ll just tell you if it’s possible or not possible. So all of these quizzes do, you think this can be done?

So first one, predict, how much someone will spend with you in the next six months.

So can this be done or not be done? This is something that’s probably, Maybe even the most, and I might actually, It’s not ad tech, but it’s finally been dead. for super long time marketing economical use, case. Absolutely. It can be done. What is the way done? Oh, we look at a bunch of people who have made their first purchase more than six months ago. For each person. We collect data. We say how much they spend a subsequent six months.

That’s our training data, We use patterns before that predict that. Somebody new comes to our website. We make a prediction surface stuff, Andrew, Everyone on this call might be familiar with this, and it’s very possible, OK.

Predict whether a user will purchase a product that you just added to your inventory for the first time. So, totally new project, product, Totally new products?

Uh, can you make predictions about who’s going to purchase this? Took a few seconds. A couple seconds to think about it.

Aye.

Know, this is, the answer here is, is mostly that you cannot, maybe you could use a standard and it’s similar to a previous product, and I’m just going to ignore the differences. And now, because it would they purchase the previous product?

You know, the thing that Amazon does, because they are such a large scale, is they say, we’re just going to show, where does have recommended to a million people over the next day.

And now we do have historical data, but for someone who operated at a smaller scale, for a totally new offering, you just need to, to run experiments and see who’s gonna buy it, because the historical data doesn’t exist in ML or AI, can’t help with that.

Also, been just so far, we’re just about to get into a Q&A, so if you have questions for either any of these people, just start posted into the chat window, so. Awesome, so, yeah, so, I’m gonna go through, actually, this is where we have one more, so.

Last one, right.

Finally, the friction that will cause the user to tell their friends about your product.

Yeah, This is, first of all, you don’t see what people told their friends about your product. So, that’s sort of organic growth.

It is, it’s hard to measure it and as a result, it’s hard to predict.

So, it’s kinda hearing like scales important. And, so, you have lots of data. And a little bit of human input is important. Is that kind of summarize.

Yeah.

Yeah.

Being able to figure out what the previous pattern is, then you make the extent of what the rate prediction target is.

Uh, we can skip this one, I think.

Keep open. So that’s, That’s interesting, I think. I think, hopefully, you guys are getting And there’s some questions coming in. Hopefully, you’re getting like, a general overview of kind of, what’s what’s from machine learning in general. I want to just do a couple of things. A couple of problems that I see when I look at this is, first thing is everybody’s black box. So there, there is a AI running, but you don’t necessarily know what it is. Right? So, you’re trusting that Google is driving the best possible conversions for you?

Who knows, Right? So that’s one problem. I’m not saying that’s good or bad, or everybody has a right to their technology. That’s one problem with.

The second thing is, we get very confused, and I think this is Dan Dan’s points is you can get correlation is not causation actually should have talked about this in his article Which is not smarter will give a link to it?

But it’s this is a common one that I’ve seen for years and starts ice cream sales correlated with shark attacks Well, yeah, there They have similar pattern, but they’re not necessarily related. Right? The other problem that I run into all the time with us is It just doesn’t work. Like it’s supposed to like, it doesn’t work like you wanted to here’s an example. This is Google’s image classifier, which is, it is an amazing tool, you upload an image, and it can tell you all kinds of stuff about it.

You can see it’s not. This chart was a bracelet, right? So it’s like doing its thing. But if you’re not involved in it, or you’re not watching, or you don’t know what you’re looking at, you can get some real false positives. So real quick, I’m going to show you an example we did, but there’s there’s present SEO people on here. This was my first exposure to, like, Big data, is trying to solve Google’s SEO. So, they obviously have an algorithm. It’s fairly simple is based on who else thinks you’re popular. Most basically, I just wanted to see what it looks like on the scene. So, you don’t get so mystified by all this stuff. Really, this is just a Python running an open source thing called not be that helps. You were vigorous data And it’s running that algorithm, right?

So that’s what it kinda looks like on the back end, And when you put it together, you can kind of start to see how some of these guys, somebody’s players, like, what their algorithm is, this, is this, actually off their website, fitbits website, But they’re basically the same.

When you go and try to create an ad, we’re gonna, we’re going to charge you, based on what other people have done. With similar as, what we think the, the action will be, which are bidding, and how good, Yeah, They have like 4 or 5 variables. Behind that, they have all kinds of information from past ads. They’re using that, combined with how your campaign works, to try to try to optimize.

So we have a perfect audience, And it’s not a perfect pitch by any means, is that it’s an engine that’s an intelligent engine that’s designed to kind of optimize your ads. So it shows ads to people who have already visited your site. Because I’m finds new ones so on. So the way his word, and I just won’t be as transparent. So you can see this example, so it starts, for Dan’s point. We have an advantage that we have in the middle of hundreds of millions of shoppers every month. So we have some data. We know, roughly, kind of, these people are, we have audience data build-up, Wave Activity, that data, but then what it does is simply is just taking out the stuff that’s not work.

Right, so it’s saying this ad was shown here and never did anything, or whatever. Let’s remove it, and it just goes through that systematic process until it gets better and better results.

And the nice thing about this, and this is like, how, this is how it should be. So, there’s a, usually, complex algorithms run in the background in real time. You don’t have to know anything about that. I don’t even have to know anything about that. I have to know what I’m trying to optimize for.

I have to make sure it’s set up in a way where it could work, right? So, if I’m optimizing for a conversion goal, for example, but I never get any conversions, maybe my conversion goal is too far down that funnel and never fires in the early part of the campaign.

It can’t kind of training and learn. So, that’s a problem, and just when you’re as a marketer, as you’re kind of thinking about it, this is a real examples of Fortune 5000 customers.

They basically have all these campaigns running all these different ads, years and years, been, trying to manually optimize and get the right combination. And then, boom, we just turn the AI on.

You can kind of see where there are about 25 average CPA before the target of 10. You can actually see over there was a system error where I actually accidentally human error when I turned it off.

And then it started to drive it back down, and then they introduce new products as a whole new funnel. Now, write the models are suddenly changed, that changes the whole thing, but that’s the basic thing, So it’s looking for like conversion goal. In my case, it’s saying what impression we tie back to that, and most likely, it’s using a mix of stuff that already knows, stuff that it’s learning as I go. So, the reason I chose this, there’s two things. one, we can, we can take a advertiser off the street and get them optimize, it works faster and better. The more you, more data you feed it, is because we need kind of information, and it works better if you have somebody actually managing, figuring out what is the right target So on, here’s, you may have seen this ad as you guys were searching around. This was designed to follow people, the basic human slip. You can kind of see it had about 10 days to run not enough data, really kicking **** the blue is the clicks. So same span, same ad.

Lose the clicks, rather than conversion, So starting to figure out what’s working, Right, So if I ran this another couple of months, or if I run a much bigger set, of data, would have worked a lot better.

You guys probably, Just as an example, you guys probably all due to Google Adwords or Facebook advertising or something like that, the title of this webinar was How to optimize. We want to hit on that here, And we’re just about ready to get it, get into questions here, so put your questions. But when you’re in Google, this is kind of something to think about, and this is, we’re actually Eric and I, and Kathleen, because you’re in charge of the marketing, we all are working on this ourselves, right? We have this campaign, Google’s driving traffic.

We’ve been moving that conversion goal further and further into the funnel, So that started, it was optimizing for leads. Then we found the leads were kind of fraudulent, or a bunch of fake or a bunch of junk, right? It was like a false positive. So when you move it to sales, and now are all the way down to, like, the person actually logged in to the account, use, the system. That’s telling the AI on Google’s case to optimize for that. So just, when you’re setting this stuff up, that’s kind of something we can do. So we’re just gonna get it. We’re gonna, we’re gonna let you guys kinda ask your questions, This, is all gonna be available, recording, We’re going to center around every including. The big thing is to start with your goal, right? So, start at the top of your funnel. We just got, I just got off call The new advertiser.

They have like a six step registration process and they have no data yet. So, the kind of the recommendation is, let’s turn on AI, but let’s make the conversion goal. They just visited a key page, right? So we’ll start to get lots of results. Then we start to test and incrementally change channels, and I think this is one thing we get a lot of questions about. Attribution in advertising and Facebook. Get the credit for Google, drove the lead, and we were, They saw our banner, and so this is Big, master, it’s a spaghetti bowl in the way that you get around those. You systematically sort of testing different funnels, 1 by 1, 1, by one. So a couple of tips like that, I want to go ahead and just open up the questions who ever got some questions coming in?

And you guys can just hop on these. So first one is from Brian.

Brian asks, At what point do determinant model has to be retrained?

Is it from a set deviation from the expected values? So Dan, you want to take that one?

Yeah, so, the most important part is that you have a way of getting the data over, getting a change in, uh, model accuracy over time.

You know, it’s quite common, Unfortunately, for people to build a model, then, deploy it, and it just, it’s getting used. But, they’re not able to bring the data back in about, it’s still functioning well. And so, the most important part is to have the infrastructure, so that, you know, if your ads are not performing as well, you are aware of that.

Then the second part of that, which is, alright, so you can see whether it’s performing worse or better. There’s, no, that’s just a judgement call us, no simple answers, or threshold where you say, above a certain numbers, OK, and below, that number is not, OK, or, or vice versa.

OK, Eric, I know you have some questions for, Dan said so interesting about this is, Eric is the general manager of perfect audience, and he’s kinda tasked with how should the machine learning, how should this product evolves? So I know you had some questions, While we have these guys on here, do you want to ask those while the rest of people are coming on?

Yeah, I think I’d be remiss if I didn’t ask like the experts, you know, it helps. It helps us, right. I mean we’re we’re very deep into overhauling some of our own underlying tech.

Both just on the product side, and, and, and so, know, it’s interesting, like, earlier on, Dan, You’re, you’re sort of talking about, you know, how like to compare, you know, And and one of the things that I was thinking about, you know, for us, is, I Would say historically, we’ve been, when I say we, I mean, perfect audience has had been sort of guilty of putting some fairly sophisticated, you know, algorithms in place, right, But nothing, Nothing overly sophisticated.

So we, I mean, we’re guilty, right, you know, of of sort of playing the AI card, and, and that, to me, it really, it sort of bothered me a little bit.

And, and, of course, when I say we isn’t perfect audience.

SharpSpring, which is the The The parent company of Perfect Audience acquired perfect audience. Last November we closed November 21st. And so, you know, when we came in, we looked at the business. We said this is a great business. There’s, there’s tons of really fundamental things going on that I’m really excited about. But one of the things that we really wanted to overhaul was, you know, what was driving the conversions that, that our customers were getting.

And so we’ve spent a lot of time and resources and money and development hours really wrapped around a newer version of the underlying engine that drives perfect audience.

And, maybe, Jalali, I’ll talk a little bit about that, but, Dan, you know, I guess one of my questions to you is, OK, so, it’s not, no.

It’s not necessarily binary, right? So, in other words, either you are doing it, or you’re not doing it.

There’s also degrees with which you are more and more effective, And so, over time, as other tech, you know, catches up, you know, other competitors to us, you know, they get better and better.

You know, we, you have to be able to use AI if you’re going to be able to compete over time, Right? You have to, you know, utilize these large datasets.

And I remember you and I were talking, I don’t know, 4 or 5, 6 months ago, something like that, about this exact subject. So I was just kinda curious, like, what your thoughts were that, like, what does that look like to you?

Like, what should I is the GM of perfect audience sort of be focused on right now? Yeah. It’s gonna come back to, which, we’re talking about ship earlier. Maybe the distinction is not, where’s their AI?

But you don’t want a cheating AI to drive your business goals, You want to say, all right, I’ve got some funnel where I’ve got, what are the metrics for that, what are the metrics that we want to move, then, now, AI is going to be of service to those metrics, so, you know, whether it’s conversion rates are, or CPA, whatever, whatever it is, You’re going to say, Alright, I want to make that, here’s, here’s the baby, that’s where do I place an ad, and, I do think that there is room at that point, to say, you’ve got all this data. I mean, you guys are in a unique position to have that amount of data. I think that now, especially some of it, some of, it is a prediction problem of.

Given which app to a place, it’s going to have the highest.

That’s going to have the highest click through rate or whatever it is you’re optimizing for.

It’s over.

I can code is now.

I think of it not as starting what adu AI do I want, but rather, what Peter schools, and I tried to move, and then after that, the AI should fall into place anywhere. Where, what is it that I need to predict?

Because that’s where machine learning is so effective way to predict.

So, I’m OK with business goals, I love that. I think there’s, there’s something to, about, you know, being able to drink your own champagne, right, you know.

So we for our own campaigns, we use perfect audience, all right, and, and so figuring out what’s working and what’s not working and being able to iterate. And then building around that, is, and building around, or, you know, the KPIs that are the most important to us. You know? If I’m putting my marketing hat on for a second, what are the most important things for me, right? And then those things translate across, you know, every other marketer. That’s great. So I showed you, so just sort of shifting the subject real quick, so you are the …, right, and, and so you have years of Inside knowledge and the industry then you’ve seen sort of shifting that’s occurring, you know, in the landscape over the, over the last, you know, few years especially.

And and so what what do you think people need right now? And what’s what’s sort of?

Like, what should be, you know people working on right now.

Yeah, that’s a that’s a big question, Eric, We probably spend a few hours on that one, but I’ll try. I’ll try to keep it short. Yeah.

I mean, actually, we’ve run workshops where we walk through, like trends and things that are important right now in the industry. I think top of mind for everyone right now is data privacy, right. So data privacy is obviously super important to consumers, You know, the government is, is talking about it a lot. We have some regulation now in California, you know, we have some in other states as well.

So as data privacy kind of continues to evolve the landscape, you know, it’s going to affect how everybody advertises online, right? And that means third party cookies are going away. That means the … from Apple is going away, right. And it impacts all the targeting the measurement. The planning, the way buyers buy online today, right? And so data, privacy, super top of mind. You know, you go on, any of any one of the big trades, any given day the week and it’s, you know, top top five to, 10 articles are talking about something data privacy related. So, I’d say that’s number one, you know, adjacent to that. I think you also have the discussion around, around things like measurement around things like the walled gardens, around things like identity, right? All these things, again, I say adjacent because they’re very much adjacent to the data privacy conversation. But I think that that data privacy conversation then spawns until a lot of different other areas like this. Right? So I think I think that’s top of mind.

I think what’s important to recognize, especially for this audience here today and, and for you guys as perfect, perfect audience is like some of its noise. Right. Like, it’s important to keep your head down and continue to kind of execute on your vision for yourself and for your customers. And I think you can keep doing that even with everything that’s going on. And I think a lot of that comes down to what doctor Becker just sat around. You know, focused on outcomes, If you can focus on outcomes, you’re aligned to your customers and what they want to do. And that’s at the end of the day, what a buyer cares about most, right. So, I think it’s important for you guys, as Somebody that’s selling a technology in the space, to help your cost for you guys, first and foremost, to stay focused on what’s important to your customers. And then, also to remind your customers about what’s important to write, because they’re, they’re out there reading the trades every day to, and, and getting all that, that kind of, tangential noise in their brands, so that’s that’s what I would say.

Yeah. I think that’s, I think that’s dead on a Just the performance and attribution.

You know, the At the end of the day, you know, there you’re exactly right. There’s a ton of noise. There’s a ton of things that you could be looking at, and, you know, it’s funny.

We, you know, we, I don’t know, three days ago. We were just talking about this in a product meeting We were talking about some of the **** things that you can sort of bring to the table, You, know, in terms of new feature development and other things.

But, really, you know, I think, rightly so, you know, the focus for us anyway has been, how can we get better performance, you know, for the campaigns that we’re running, and just, stay, sort of, like, singularly focused on that. And then that, you know, sort of leads to so many other things.

But, you know, I think attribution is way up there. Right? I mean, attribute and really understanding where that conversion is coming from, and being able to make sure that your dollars are being the most effectively spent. That’s, that’s critical.

So, all these points, Digital always point earlier, right about causation versus correlation, right? Like, I think, 5, 10, 15 years ago in this space, everyone was kind of obsessed with, OK, how granular can I get with attributing, you know, X dollars to why, you know, impression or touch point? And I think the industry over the last few years, partly because of some of the data privacy shifts, has realized, like, OK, I actually we’re, we’ve been solving for the wrong problem, right? We’ve been trying to assign credit for way too long, as opposed to seeking out true causation. And I think that’s, that’s, you know, that’s, that’s an evolution of the space. That’s been really positive because the industry is kinda pushed us in that direction. As cookies go away as the walled gardens get higher. You can’t just, you know, attribute. You can’t tag everything and attribute value to every single touch point. So, how do we get smarter as a community? Well, OK.

We need to actually zoom out and seek out causation, not correlation yep. That’s that’s absolutely right. So so Jalali we’re getting a ton of these questions are coming in. Do you want to you want to take us through some of them?

Yeah, sure, so this correlation or causation versus correlation is interesting to me. I saw it described as, I’m not picking on Facebook, and all this tremendous platform, but they somebody, somebody was describing as the handout coupons at the cash register line.

Right?

So, it’s like a, I’m not saying that’s good or bad, but that’s different than Going out and finding people that would be potential buyers, right?

So, that leads to, Daniel, actually has a great question, ties this, they seem for Facebook’s algorithm, you need 50 results in seven days to exit the learning phase, and set a proper, probably deliberateness.

So he’s asking, like, how does our platform working in person, But just, I guess, to touch on that, so, 50 results in seven days for this learning phase, basically, I think what he’s referring to is just this initial period where it spins up campaign, and tries to determine kind of, how it’s gonna work and determine which placements to show. Now, I do think, in Facebook’s case, they actually do do use a fair amount of historical data. So you could come into it blind.

It would have a sense based on similar product category, and et cetera, et cetera. In terms of perfect audience, we don’t have a set amount of time. It’s not it’s it’s a continuous optimization.

So in the beginning is going to be your poorest performance. And depending on kind of how many conversions you’re getting is a function of impressions, it’s going to start to improve. And there should be no limit to how far could improve, obviously, of market cap. But that’s a great question, Daniel.

Does anybody else have anything to add? Certainly pull up another one here, I think I think your other slide, can You go back a couple of slides, just real quick?

I mean, that’s sort of what you, That’s exactly what was happening here.

Go, let’s see one more, I think.

Yeah, there you go, so.

that’s cool, There we go, So that’s exactly what’s happening here, right, is, oh, you know, if you think about this in terms like a time series, know, you know, the, the algorithm, you know zero data, you’re starting like clean slate, right, And over time what’s happening is, is, as that data is being sort of adjusted and figured out, right? Which, you know, which publishers, which ads are working better, you know, and cutting out? The stuff that doesn’t work, you know, methodically and as quickly as possible, you start seeing lower and lower CPA’s. So, yeah, I mean, that’s like that’s this is like the graph that sort of answers that question.

Yeah, and it’s obviously, from my perspective, it’s, all right. I’m fairly experienced optimizing. These can work with it. I can never could have done this without this model. Can never do it accidentally, turning it off in the middle of the campaign? Yeah, yeah. Yeah, going back out, so. Yeah, it was a very human health, and they can also get the middle of it, so. Yeah, great questions, let’s just so OK, How does machine learning compared to regression modeling?

It’s a good question.

Yeah, so, uh, all right. So, regression modeling can that phrase can get used in two senses.

The one which you probably mean is, is that for a long time, people were running regressions in the sense of a linear regression, logistic regression. Machine learning is doing the same thing, but it just does it more estimates. It makes more accurate predictions.

It does those because of the way in which it picks up interactions between different variables, but it’s just doing the same concept for the same thing, but does it more accurately.

Some people make it distinction, I think, is not the way that this section is. The question that, this question is asked this distinction, Some people ask for, think of it, in terms of regression, which is predicting a number of risk classification, and that machine learning can do either of those.

But, I think, you really mean 10 years ago, people were using yeah! The software called … data, or SPSS to run regressions. Machine learning is doing the same thing more accurately.

So, the most basic way to describe it, in, as this course links, you’re gonna see a pattern from some historical behavior, then your machine learning is going to kind of try to match up those patterns, right?

Or find things that are, don’t match the pattern, from a, from a stats perspectives, out of like a rough way to describe the? Yeah, definitely find things that match, right.

Figuring out what the patents really say, figure out what the pattern is, then when we get a new data point, we just apply those patterns. So in a regression framework, we say, every time or in a classical regression, we said, for every time the person that previously visit our site, the elected of our sale went up 2%.

So we’ve got some baseline NASA has come that visit our site tintypes, which set the baseline at 20%.

And that’s our predicted probability, OK?

Yeah, like causation and correlation though, I mean, that’s the that’s the thing that’s where everybody sort of gets tripped up. I mean, we’ve talked a little bit about that in Shiv. I can’t remember if this was in your article?

Or not. It talks about the the The correlation between like the time of day and your cell phone charge and no That wasn’t You.

Know, My my my chart was the correlation between the cats movie coming out? The economy going to excuse my language, wherever it and that was correlation. As opposed to, you know, … being the issue and that that actually being the causation.

So, yeah, so that Yeah, that’s Yeah, exactly and then. so sort of same, same sort of story.

The, the the correlation between insurance, underwriting, and your cell phone battery charge when you fill out the application and we have something that I would love to test on my side which, by the way which fascinates me. We could spend like 20 minutes just talking about that. But setting that aside, and, Dan, like, maybe we ought to talk about this after the call or something, but, but then there’s this other one that I think very much is, like us, like, you know, in just a very real case study for perfect audience.

We, there’s a correlation between the time of day sign-up, right, And the quality of that account and the quality of that advertiser, and the quality of, you know, the campaigns that were running, based on when, when they signed up, during the, during a 24 hour period. So, and even if you’re cutting out international, and everything else, you’re just talking about domestic US, there’s a correlation between those two, So.

Yeah, I mean, I think just, there are so many things that, you know, you know, you know, an AI is going to be able to pick up. You know, that we would just never, in a million years, be able to draw that connection to.

And you know, just just fascinating To me, like I said, we could spend an hour on this, so anyway. Sorry, to intervene before we close, I was like, is that for causation?

Later that you have your cell phone, hydrogen or whatever.

You’re more higher risk, Or is it, you know, that’s that’s one that would get me, I guess. I don’t know.

Yep.

I would say that’s correlation?

This just happens to correlate.

It’s not well correlation That’s that’s right. Yeah, It was I think correlation is a little bit of a gray area, right, because I think things can be tangentially related, right?

And, Um, But the key is one doesn’t impact the other, right?

Like, thinks two variables can move along the same line and be like related to each other in some way, but it’s not that the battery being drained is causing, you know, the the other insurance thing to happen, right? Like, I think that’s the key thing, Is there’s no direct impact from one variable to the next that’s causing it to move in the same direction. Alright, You know, what? What are the ways that data scientists think about this?

Which I think is quite useful as to what we’d say, what its causes me, really, what are the what are the things we control? If we were to do something that, If you back to this example we see this historical relationship between ice cream sales and shark attacks the peak in summer. Alright, if we were to give everyone a discount for ice cream.

Would that change the number of architects, and that’s how that’s sort of like we can make, do some intervention, and that’s how we tease out the difference, And it’s super useful, because marketing, like the thing you’re doing is changing what people see. And so, is it correlation or causation?

Yeah, the right definition is, if we do more of this thing, are we going to urbina somewhere?

That’s an amazing example, and thank you, thank you for explaining that and much more eloquently than I did. So thank you, Yeah. Super humble.

Yeah. Well, just to that point, as we close out here, a guest for you guys. So, let’s say, I’m an ad network. I’m serving ads. I can tell through my modeling that someone’s about to make a purchase, show them an ad.

Did that is that correlation is not causation and I think that’s really does it help? If I see an ad right before and should I get credit as the network for you, to doctor Becker’s point the way you test that is shut that down right? Like, run it. Then don’t run it and see what happens, right? You’re going to tell 100 of these things, without the ad, and you saw 100 of them, with the ad.

No. I don’t care what they saw last, you’re going to sell the same amount. That’s, that’s not.

I’d say that’s not a causal relationship, but maybe if you’re in the ad, network, you, stomach work.

So might want to track the credit for it.

Right? You do want to write, because you’re trying to show optimal CPA until you contributors, I go back and forth, because we know, like, we’ve seen things. You brought a big prospecting campaign with your banners.

The SCIO goes up, great, because people are seeing it, and then they’re looking up the name.

Or you do see these things.

I think one of the challenges is always, what’s, what should we focus on and also, how do we best help use an advertiser, make more money?

that’s really great, not sure, but that’s, you know, I think to me, you don’t have to look at it holistically.

If you, if you start getting lost, right, going down, different rabbit holes and, you know, you gotta, you gotta start with. We’ve, we’ve done, I don’t know, 2 or 3 different webinars, you know, over the last several months about this topic. Where we just talk about, like, what are the, what are the established KPIs that are really going to move the needle for you and your business, right?

Focus on those, and then work your way backwards, you know, sort of if you think about it in terms of a funnel, right, what are the most important things?

I mean, that’s why we talk about retargeting all the time, and, know, we know, it’s, jalali, I mean, we talk about that being sort of the low hanging fruit of the, you know, of the, of the advertising world, right?

I mean, that is, know, you start there, and you start working your way back up, up the funnel, based on whatever.

I mean, you just apply an 8020 rule, right, and get the 3, 4, or five KPIs that really matter, that drive your business. Right, Focus on those and, and get those right, and then you can start, you know, moving out in sort of concentric circles.

But, yeah, no, I, like I said, we could probably spend a lot of time on this.

So, Jalali, we’ve got a ton of questions. It feels like to me. Maybe you and I can do a follow up video maybe on Monday or something, or to, I guess, Tuesday, where we just go through, we answer some of these questions offline?

Then post them, but, you know, is there anything else that you you wanted to cover, as we’re as we’re going through the slides here?

Yeah, we’re just about sorry, over time. So, if anybody’s interested in, and, again, it’s not that we’re not trying to do sales pitchy, but I will, I will basically closed, and we’ll follow up with links to these gentleman’s, LinkedIn, and websites, and so on. If anybody’s interested in actually participating in the lab.

So we’re, we’re just rolling this out, but this is where we actually work, with you.

There’s no cost to it, it’s just a function of your ad, spend. We work with you to get this optimization in place. So all, we’re looking for people with known conversion funnels. So like you have some channel is working, and you’re getting leads. And sales, we even give you an app, could kick it off and originally for people that want to kind of leverage this and work as part of a small cohort. So, you can use e-mail, Me also put myself in the chat and then just keep an eye out. Kathleen and the team will get this around, everybody, that recording all of this. So, just truly thank you, Dan, and Chip, for taking the time to get on your. I think it’s, I’ve learned a lot. Hopefully, you guys all have to.

I think it’s just the beginning of this. I think this is a huge shift in industry happening, or, I think part of it is just like, let’s all get on the same page. So, thank you all for coming, by the way.

Yep. Thanks for having me. Yeah, this was fantastic. I had a lot of fun. Yeah. Thinking, so. Awesome. Well, have a good rest of the week, everybody, and we’ll talk to you soon. Thanks.