Quality Score vs. Quality

Posted on Thursday, December 3rd, 2009 by Print This Post Print This Post

Categories - Featured, SEM

How an Inaptly Named Bulk-Discount Program Can Lead to Confusion and Bad Business Decisions

Names and labels aren’t everything, but as those of us who work in marketing know (by virtue of the fact that we have jobs), they sure can make a difference. Google’s inappropriately named “Quality Score” is no exception.

There’s been a lot of industry chatter recently regarding Quality Score – more analysis of its impact, more investigations into how it’s calculated, even occasional calls to optimize to Quality Score as if it’s a performance metric. The reason for this trend is, in my opinion, psychological. Just as we SEM ad-writer/analysts play with words in our ads to elicit different reactions from users viewing SERPs, Google has done a number on the SEM industry—even if unintentionally—by inserting the word “quality” into “Quality Score.”

Sure, there are relevance-related variables in the QS equation. Tying keywords to their ads and landing page tends to give them a good starting QS when launching a new SEM campaign. This is what Google tells us, and we all see it in practice.

But what Google also makes very clear is that once a campaign starts generating traffic, the most important factor by far in determining QS is click-through rate.

To step back a moment, the Quality Score that we see in the Google UI is really a bird’s-eye view. Quality Score is a moving target calculated one impression at a time. In other words, the combination of one keyword with one ad in one position at one moment in time, and the combination of the same keyword in the same ad group with a different ad in some other position at some other time, will generate different Quality Scores at the point-of-query. The QS number we see for a keyword in Adwords is an average of all those query-by-query QS calculations, NOT a flat ranking that Google assigns once.

Google could have chosen to show QS at the ad level instead of the keyword level (as YSM does with its Quality Index). If Google did it that way, the ad QS number would also be an average, effectively amounting to Google telling the advertiser, “If you take all the times that all the keywords in this ad group have triggered this particular ad, and average all those unique quality-score instances together, this would be your average Quality Score for this ad.”

Google does not show QS by ad, but the impact of Quality Score is arguably more clear when looking at performance data at the ad level. Take this ad group as an example (the #s are real, but ad copy is not shown to protect the innocent):

quality score

The first thing to note here is that the campaign in which this ad group lives is set to “Rotate Ads Evenly” rather than auto-optimize, meaning that ads are supposed to be displayed against an equal % of queries rather than giving more impressions to ads with higher CTRs. However, there is still a severe impression discrepancy (column B). How can that be?

Google’s answer is Quality Score. If the QS of a keyword-ad combo causes the ad’s position to fall off the first page of search results, and if the user does not click through to that second page of paid ads (and who does?), then that ad will not display. It has still received its equal opportunity to show up in the search results, but it won’t get the impression. So in the sample above, Ad #1 has a higher QS and shows up on the first page of search results more frequently, which leads to a higher impression count.

Without looking at the ad copy itself, what exactly is it about these ads that’s causing this Quality Score difference? See column D: CTR. The ad that dominates the CTR game gets a higher QS, gets shown on the first page more often, and gets more impressions as a result. Again, none of this is guesswork or any sort of conspiracy theory. This is Google’s explanation.

Now let’s move over to the conversion-rate metrics (columns H & I). What do we see here? Performance flips. The high-CTR (high-QS) ad is by far the worst converting, delivering by far the worst CPA.

Translation: From the perspective of the advertiser’s ROI business goals, the ad that the Google algorithm is giving the highest Quality Score is… wait for it… the worst-quality ad, driving the worst-quality traffic to the site on average.

So why would we keep that ad running? Because if we drop it out of the rotation, CTR for the whole ad group plummets, as do the keywords’ average Quality Scores, and we lose significant impression share.

Yes, you’re reading this correctly: We keep the worst-quality ad running in order to maintain a high Quality Score. I won’t go so far as to say categorically that in order to get good-quality traffic you have to have a bad Quality Score, but in many (if not most) cases that is in fact how it works out.

To paraphrase, what Google is effectively telling the advertiser about this ad group is this: “Ad #1 is going to lead to a lot more clicks, which is good for us. We get more money out of the SERP real estate that you occupy with Ad #1 than with Ads #2-3, so you can have that real estate at a cheaper rate with Ad #1.”

If Google called a duck a duck – or in this case called “Quality Score” a “Bulk Discount Opportunity Rating” or something like that – there would be no confusion. The bulk discount is a respected, time-tested sales tool that has its proper place. We could take these findings to our clients and say, “Google is charging us less for individual clicks on Ad #1 because of its high ‘Bulk-Discount Score.’ But these other two ads convert much better and do a better job of helping us meet your ROI goals. So which is more important to you right now? Volume or efficiency?”

But because Google has labeled this system with the word “quality” rather than “bulk discount,” the already delicate balancing act between volume and efficiency that we perform every day on behalf of our clients gets even more complicated. “Google says we have a bad Quality Score,” we sometimes hear, “so can we please do something to improve our Quality Score so we get better quality traffic?” As outlined above, that just ain’t how it works.

Fortunately, we can demonstrate that Google uses the word “quality” differently than we do by showing countless cases like the one above where optimizing to CPA necessarily leads to taking actions that hurt Quality Score. But that word “quality” makes it a lot harder to swallow.

The insight that Google does provide into Quality Score is extremely valuable, and can inform good SEM decision making—if Quality Score is understood for what it is. It’s an important research tool in that it serves as an indicator of what Google will favor. But for advertisers interested in measurable ROI, Quality Score is not a measure of quality, nor is it a metric to which SEM campaigns should be optimized.

And it needs a name change.

Tags | , ,

9 Responses to “Quality Score vs. Quality”

  1. Mike says:

    Position by ad would be helpful to include in your results. Ads 2 and 3 probably had a lower position than ad 1. If ad 1 was in a top position people might be clicking on it just because its near the top of the page hurting conversion rate. If you ran ad one at a lower position conversion rate might improve.

    Also you should test running on just ads 2 and 3 and with an increased bid and see what happens to CPAs and volume.

    I do agree though that Google should have ad level QSs and “quality” is in eyes of Google. It would be even easier to understand if was renamed something like “CTR Index”.

  2. Barbara says:

    Excellent informative post. And yes in this case, the name does feel like its everything since it is perceived (particularly by clients) to mean something that it doesn’t. “Quality” in all instances is something we strive for, so it was not coincidental that Google would use that over “Bulk Discount” and cause a frenzy to maintain a high quality score.

    Thanks for the thoughtful explanation.

  3. Nate Price says:

    Thanks for the comments. Mike, position definitely is part of what’s going on in the given example, and I probably should have included that. The reason ads #2 & 3 have fewer impressions is indeed because their average positions are much lower — to the point that they often drop off the first page. The reason that happens is the CTR-QS relationship.

    You make a good point too about possibly leaving out Ad #1 and simply bidding higher to try to get more impressions for the other two better-converting ads. We’ve done that in cases like this in the past, and the usual result is an increase in CPC and therefore CPA, even though the conversion rate may remain high. In other words, the CTR is inherently lower for ads 2 and 3, so it takes a higher CPC to overcome the QS deficit and get into higher positions, which in turn hurts CPA (and ROI). CTR may go up some as position rises, but not to the level of Ad #1. In the end, it depends on whether or not the resulting higher CPA is within an acceptable range for the client’s goals, and that will vary ad group by ad group. But it’s a good suggestion and it can be beneficial in some cases. Thanks!

  4. Richard says:

    This was a very informative post, particularly in highlighting the trade-offs advertisers sometimes face between CTR and conversion rates. I agree that high CTR, low converting ads can be useful in balancing out low CTR, high converting ads within an ad group. However, I would also suggest continuous creative testing to develop ads that intrinsically balance CTR and conversion rates. Thank you!

  5. Mike says:

    I think this is a great explanation in breaking down a complex and obtuse metric that just can’t be isolated without understanding it’s interplay with other metrics in a search account. Thanks!

  6. Quality score isn’t already indicative of good quality. We’ve had to have our sites go through a manual scoring because our quality score went down and cost went through the roof. Its such a qualitative metric. Great post!

  7. Bradd Libby says:

    Nathan, this post is a great description of why the highest Quality Score is not always desirable and why Google’s use of “quality” in the name is ambiguous and perhaps misleading.

    The only thing I have to add is that their use of “score” is likewise ambiguous and perhaps misleading. I can’t tell you how many times I’ve seen some blogger write a description to the effect that Quality Score is “how good Google thinks your ad is”, as if Google is a schoolteacher and QS your grade/mark/rank for writing good copy. Too few people seem to realize that Google does not select your Quality Score, searchers do. And, the use of the word “score” makes it sound like, as you said, QS is a performance metric to be optimized or maximized (two words which are frequently used as synonyms even in cases where they’re not) or, at least, to be made higher than your competitors’ values, when in fact it’s profit we’re looking to maximize, not CTR.

    Let’s face it “position-adjusted normalized clickthrough rate” just doesn’t roll off the tongue the way that “Quality Score” does. The first Mike’s comment is onto something: it should be called ‘CTR index’, or something similar. But, as long as Google keeps calling it the “Quality Score” people will probably be inclined to keep saying that you want it to be as high as possible.

  8. Seems to me you’ve tried to make a general rule out of a specific result. Had the first ad converted proportionally to the cost it would negate this effect as you’ve concluded, however it bodes well we need all do our math.

    Our approach here would be to dissect each ads success/weakness and continue to invent and iterate. Furthermore without awareness of keywords used, there well be much room for splicing up this adgroup for more efficient yield. But I’m no pro & would love clarification if indeed I am missing something here.

    Most importantly your observation of “abortive impressions” was startling – and you’d have won us over if you’d given it a catchy name. After all, isn’t immortality what all authors dream of ;)

Trackbacks/Pingbacks

  1. Quality Score: Better with Age? - Web Design Greece – WEBTEC

Leave a Reply

Follow Us on Twitter

Authors