What happened so far
This post is the fourth and deciding part in a series of posts to reveal the secret of AdWords quality score. Previous posts:
The first post was about the lack of consensus about what quality score actually achieves. It’s a little bit as if Google had an abstract formula for something they call quality and advertisers just had to live with it. The second post contradicted the notion that Google was all about quality. With the help of some examples I showed that Google is in fact more interested in making money, even if it comes at the cost of quality. So the idea of quality score being something that sacrifices profit in favor of quality seems more like clever marketing on Google’s part.
The third post pursued the idea of Google being primarily interested in maximizing its own profits. I took a closer look at the ad auction – the procedure that determines the order in which ads are shown on Google. The question was how the ad auction had to work in order for Google to earn as much money as possible. The only parameter Google can control is quality score. I found that quality score has to match click-through rate (CTR) in order for Google to maximize profits. So last post’s conclusion was:
Quality Score = Click-Through Rate
First of all it’s important to make one thing clear: There is no doubt that quality score has to equal click-through rate in order to maximize profits for Google. The proof isn’t just in the last post: before Google introduced quality score in 2005, CTR was used in precisely the same way. The ad rank was calculated as Bid x CTR, today the formula is Bid x Quality Score.
I know, that sounds absurd and contradictory: if the formula was already optimal and was subesquently changed, wouldn’t that mean that Google is pursuing another goal in addition to maximizing profits? Or did Google not change anything at all?
The nature of Click-Through Rate
It’s actually a little more complicated. A closer look at the trivial-looking CTR can shed some light on the issue.
When Google determines the ad order, there are two things to be considered: how much does Google make per click on an ad and how many clicks on that ad are there? In the last post I used two terms to describe these factors: bid and click-through rate.
But click-through rate is actually an inadequate metric here, because it refers to the past. Click-through rate is simply the number of clicks so far divided by the number of impressions at this point in time.
But whenever Google handles a search query and has to determine the order for the ad, then it’s not about the past: it’s about whether the ads will be clicked here and now. It’s about the immediate future. When the ad order is determined, it is just about this one impression for just the current search query. Will a click follow or not? Click-through rate has little to do with that – rather, this is about click-through probability.
What’s the difference?
To illustrate the difference between click-through rate and click-through probability we can look at a coin toss. Heads or tails, both have the same probability of 50%. These probabilities are set and will never change.
By counting the coin tosses that result in heads and dividing these by the total number of tosses, we can determine the “heads rate”. So if we get heads once out of two tosses, we get a heads rate of 50%. That would be consistent with the “heads probability”, which is also 50%.
However, two coin tosses might lead to heads twice. That would be a heads rate of 100%. Still, heads probability remains unchanged at 50%.
It’s the same with click-through rate and click-through probability: if an ad had two impressions and one click it would have a click-through rate of 50%. Still, that doesn’t mean it will have a click-through rate of 50 % for future impressions as well. If Google wants to make predictions, the real click-through probability is needed.
As hinted in the example, determining click-through probability isn’t as easy as one might think. Of course, there is a connection between click-through rate and click-through probability. However, just looking at click-through rate and concluding that click-through probability should be the same doesn’t work.
Let’s go back to the example of the coin toss where we had heads twice and a rate of 100%. If we assume that the probability for heads is 100% as well, we would be very wrong.
At first glance one might think that rate and probability would converge quickly. Naturally, with ten coin tosses it’s easier to make assumptions concerning probability than it is after just two tosses. But after ten coin tosses it is even more unlikely that exactly 50% will come out heads. Reverse that, and it follows that even with five out of ten being heads you cannot conclude that the underlying probability is 50%.
Before this gets too complicated, let me demonstrate the dilemma with a few numbers. The following table shows the number of coin tosses, the number of times they resulted in heads and the heads rate (always at 50%). The last two columns show the range in which the probability is likely to fall into. An example for how to read the table: If there are ten coin tosses which result in heads five times, the heads rate is at 50%. The probability for heads is likely to be somewhere between 19% and 81%.
(In case someone wants to check the math: these are Clopper-Pearson intervals with a confidence level of 95%.)
The numbers show: even if the rate is always at 50%, you need a lot of tosses in order to narrow down the probabilities. (By the way: even the ranges which are used in this example are only correct 95 % of the time. That means, 5% of the time the real probabilities lie outside of this range.)
Google finds itself faced with the same dilemma when trying to deduce click-through probability from click-through rate. The following table shows the same situation as above, but this time it’s about an ad with a click-through rate of 5% – a very common situation in AdWords.
As you can see, a lot of impressions are needed until reliable numbers are available. Furthermore, Google needs numbers to be as exact as possible, because an ad auction is often about multiple ads with click-through probabilities very close to each other.
This exercise may appear very theoretical – who cares about how accurately one can calculate click-through probabilities? Worst case, Google miscalculates and messes up the ad order a bit, losing a penny in revenue now and then. However, if you consider that Google is delivering millions upon millions of ads each day, these subtleties add up to a big difference.
Estimating Click-Through Probability
In order to rank ads optimally and achieve the highest possible profit from them, Google needs the best estimates possible for click-through probabilities. The tables demonstrated that for a good estimate much data is required. Until this data is collected, Google would have to fly blind – or at least would have to rely on bad estimates. So, during the time of data collection, suboptimal results would have to suffice – Google would have to settle for lower earnings.
Google faces an even bigger dilemma. The tables suggest that there will be reliable data at some point. Reality is oftentimes different, because there usually isn’t the one click-through rate to rely on. For example, let’s look at the keyword “running shoes”, which has a click-through rate of 5% after 10,000 impressions. According to the second table that narrows down real click-through probability fairly well.
Nonetheless, these 10,000 impressions come from a large number of different search queries like “running shoes” or “discount running shoes”, etc. Naturally, every search query has its own click-through probability. Maybe our ad has the headline “Discount Running Shoes”, so the query “discount running shoes” will probably have a higher click-through probability. So each query has to be looked upon separately. Thus the 10,000 impressions are split between the actual search queries.
But it doesn’t end there. Ad rotation makes things even worse. Every query works differently in combination with different ads. For example, the ad for discount running shoes will work better for the query “discount running shoes”rather than an ad with the headline “Best Running Shoes”. And of course there is the situation that a query or ad are simply new and there is no data available yet.
Bottom line: Google really needs enormous amounts of data to estimate click-through probabilities if those estimates are based on clicks and impressions. Many times, that data is simply not available. So what else could Google use in order to estimate click-through probabilities? Lucky for us, we don’t have to look very far. The answer lies in the well-known components of quality score, which can easily be looked up under the AdWords help pages.
There’s click-through rate itself – well, we know why that one is important. Other factors are centered upon relevancy. That makes sense: the more relevant an ad is, the higher the probability for a click. Aside from click-through rate and relevancy factors, the landing page is often mentioned. But the landing page is considered within the ad auction. In the second post I called that a contradiction to Google’s promise of quality. Now we know why the landing page is irrelevant for the ad auction: it has nothing to do with click-through probability, because the user sees it only after the click. The only thing that gives any indication about the landing page is the display URL – and its click-through rate is in fact listed as a separate factor.
It is reasonable to assume that there are many more signals Google utilizes to refine its estimate. How exactly all of this is used in estimating click-through probability is everything but trivial and cannot be determined from the outside looking in. However, we can assume that a quality score for newly added keywords or ads is calculated based on factors like relevancy and more or less related click-through rates (whole account, similar keywords, etc.). When more and more data is collected, those factors slowly fade into the background.
The Secret of Quality Score
In the first post I promised answers for three questions. Now we have these answers:
What is quality score good for?
To maximize profits for Google.
What is quality?
What is quality score?
One might think this is, again, click-through probability. However, as outlined above, click-through probability is an intangible variable which can only be estimated as well as possible. That estimate is quality score.
So, again in clear bold letters:
Quality score is Google’s estimate for click-through probability.
And there we have revealed the secret of quality score.
Is that really true?
As I mentioned in the first post: I didn’t break into the Googleplex to steal the secret. Neither has Google in any way confirmed that this post is correct. So I honestly have to call it a theory.
On the other hand one has to consider that this theory is simply following up on the idea of Google trying to make as much money as possible. So if this theory is wrong, this would mean one of two things: either Google doesn’t really want to maximize its earnings or they simply haven’t come up with the idea I just presented. Both seem unlikely to me.
But what about actual quality, you might ask. Wouldn’t this mean that Google corrupts its search results in order to make a quick buck? No, because there is no contradiction here. If quality is click-through probability, this basically says that a good ad is an ad that gets clicked often. This is a definition of quality that works for the users, too. As long as Google makes sure that shady advertisers cannot abuse the system – which is accomplished by high minimum bids and advertising policies that exclude such advertisers.
By using click-through probability as a measure of quality, Google makes as much money as possible and users get good search results – a win-win situation. Of course, Google only highlights the part where they deliver high-quality search results and downplays the fact that it makes them a ton of money. But that’s not a conspiracy, it’s just marketing.
So what now?
The big secret has been revealed, but that doesn’t mean we’re done. Because using this knowledge we can dig a little deeper into how quality score works. We can use it to correct myths and misunderstandings, as well as answer some open questions about quality score. We can even speculate about previously unknown factors. So, in the coming days and weeks, several posts based upon this one will follow.
Until then, thanks for reading.