Is Google Exploiting Neuromarketing in Reporting Quality Scores?

Posted on Monday, March 21st, 2011 by Print This Post Print This Post

Categories - Featured, SEM

I’ve got a mystery I’m trying to solve. You might call it ‘The Case of the Missing Quality Scores‘.

For each keyword where an advertiser places an ad, Google reports a Quality Score number between 1 and 10. All indications are that this number is largely based on the ad’s clickthrough rate compared to other ads that have appeared in the same positions in the search engine results.

It doesn’t take long to see that the most commonly reported Quality Score for many accounts is 7, with 10 being seen for the best-performing terms (often, brand terms) and numbers below 7 being considered ‘Poor’ or just ‘OK’. In an article published at Search Engine Land in October 2009, Efficient Frontier’s Dr. Siddharth Shah noted that Quality Scores of 8 and 9 are “very rare”.

But it hasn’t always been that way.

The graph below shows, for each Quality Score, the fraction of the account’s impressions whose ad had that Quality Score, for February 2011 and February 2009 for an account that has spent about $1 million per month on pay-per-click advertising. Because Google’s Quality Score calculations only use data from cases where the search query is identical to the keyword, the results shown here are only for exact-match terms.

In February 2009, more than half of the impressions occurred in ads whose Quality Score was 7, with higher and lower Quality Scores being less common depending on how far from ’7′ they are. By February 2011 though, the picture had changed radically. Now, Quality Scores of 3-7 are very common, with 8s and 9s being seen less often than 1s and 2s.

This change would be easy to explain if the performance of this account had crumbled over the past couple of years, but in fact the opposite is true. Impressions, clicks, conversions and net profit are at record highs. The account-wide Conversion Rate has doubled in the past two years. CPCs are up (as they should be). Yet the account-average Quality Score has fallen from about 7 to about 5.

Quality Scores of 8 and 9 are now very rare, but where have they gone?

The graph below shows the percentage of impressions (for the same account as the graphs above) in ads in several Quality Score categories for the past couple of years.

Quality Scores of 8 and 9 largely disappeared in the spring of 2009. However, the fraction of impressions in ads with QS of 1-6 has continued to rise, leveling off for this account (perhaps temporarily?) only in the past couple of months.

This change was not simply due to a large change in the traffic received by keywords with a certain Quality Score – the absolute number of ads with QSs of 8 or 9 became extremely low, while the number with low QSs increased.

And again, this change might be easy to dismiss if it was just one account. But it’s across-the-board. For those accounts that TSA has managed for the past two years, in early 2009 on average about 20% or more of the impressions in a given account were from ads whose Quality Score was 8 or 9. A few months later, you basically need scientific notation to report that number. For all of the top-spending accounts The Search Agency manages, you won’t now find one where the percentage of impressions in ads with a QS of 8 or 9 exceeds 1/10th of 1%.

(click image to enlarge)

Why is that the case? I’m thinking that perhaps Google is capitalizing on a basic aspect of human psychology to prompt Quality Score optimization efforts.

The premise is simple: Nobody likes to think they are below-average in something. This fact is the basis of both the Dunning-Kruger Effect (the idea that below-average performers don’t recognize that they are below average) and the Lake Wobegon Effect (the reason that most drivers consider themselves above-average at driving), as well as numerous, um, personal medical devices of dubious efficacy.

When you make changes to your AdWords account, in fact you are working for Google. As Alex Cohen of ClickEquations notes, Google might be able to do the dirty work of choosing keywords for you. But, they can just as easily let you choose the keywords, knowing that you’ll try to pick keywords that will make you the most money. So, when you choose a keyword to bid on, you are acting as Google’s meat-based query-to-ad matching algorithm.

When you change the text of an ad or the bid on a keyword, you are changing (a portion of) Google’s search engine results pages.

And when you modify your AdWords account, you are not paying attention to your Bing (or, formerly, Yahoo) account.

Therefore, it’s in Google’s best interest to get you to devote as much of your time and attention as possible to their platform. One great way to do that, it seems to me, it to simply tell everyone, regardless of their actual status, that they are “below average” and then wait for knee-jerk human psychology to react.

Let’s say it’s February 2009 and you log in into your AdWords account and sort the top-performing keywords by their number of impressions. You see that about 60% are 7 or 10 and about 20% 8 or 9. Only 20% are 6 or below. That is, 80% of your impressions are in words that are 7 or above. What do you do? Probably find something else to worry about.

OK, now say it’s 2 years later. You log in and find that 75% of your impressions are in ads whose Quality Score is 6 or less. To make things worse, every account-wide average Quality Score looks worse than the month before. Now what do you do? Find some new keywords! Change some ad text! Spend more time on AdWords! Optimize that account!

The biggest lessons for online marketers, I think, are:

If you do month-over-month comparisons of your impression-weighted Quality Score, realize that the percentage of impressions in keywords whose Quality Score is 1-6 still might not have stabilized 2 years after 8s and 9s virtually disappeared from Google’s reporting.

If you’re wondering why your Quality Scores are disappointingly low, realize that Google might simply not be reporting many Quality Scores as being above average.

And, realize that Google might simply be reporting ‘below average’ Quality Scores for everybody, specifically to get you to focus your attention on AdWords as much as possible.

Tags |

15 Responses to “Is Google Exploiting Neuromarketing in Reporting Quality Scores?”

  1. Michael says:

    Another explanation is that this is a normalized scale where the quality by which you are being compared has improved over time. Given a QS by which to judge success, I would hypothesize that there is a race to improve and that the entire ad market has improved. I have always suspected, and there’s a lot of data to support, that QS is relative to your competition.

    • Bradd Libby says:

      Michael,

      You’re right – This is certainly a possible explanation for the changes I’ve seen. But it seems to me that, if an improving ad market was the reason, we should see the presence of QS 8s and 9s drop for some accounts, rise for others, and even stay the same for others. But from what I can tell, 8s and 9s virtually vanished by the summer of 2009 while ads with Quality Scores of 1-6 have gained ground (and might still be gaining ground) since then.

      Where are the accounts that get a significant amount of traffic from QS 8/9 ads?

  2. Ted Ives says:

    Interesting thinking Bradd.

    I have seen numerous QS distribution graphs with a peak at 4 and a peak at 7, just like you show all over the web as well – WordStream for instance shows one here:
    http://www.wordstream.com/images/screenshots/quality-score-distribution-2.JPG

    The chief input to quality score is Click-Through-Rate (per the classic Hal Varian video), but there is no reason I can think of to expect CTR to follow such a radical bimodal distribution – unless Google were purposely warping some more natural looking distribution by overlaying an additional function that is bimodal.

    If Google is doing so to affect account managers’ psychology, one might expect to see Google applying some rotation to keyword’s quality scores (i.e. a 9 all of a sudden becoming a 5 – then maybe popping back) over the course of a couple of years – to make sure you paid attention to every keyword in an account, at some point (or at least all the head terms).

    An interesting test would be to pause every keyword in an account except for the 8s, 9s, and 10s and see what happens – would they stay at those numbers, or slowly change so the account has the familiar distribution? I would bet the latter.

    Good luck getting someone spending real money to try that over multiple quarters of course! (this explains why there are no prominent stories in the industry about anyone doing such a test).

    P.S. By the way, 50% of the people reading your posting enjoyed it less than the average reader, so try harder next time!

    • Bradd Libby says:

      Google used to report Quality Score as ‘Poor’, ‘OK’ and ‘Great’, with those ratings now apparently roughly conforming to the current score ranges of 1-6, 5-7, and 8-10 (respectively).

      So, that might explain jumps in word counts at the break-point QS’s.

  3. One factor that most seem to ignore (not saying that you are) is the aftereffect of Google allowing advertisers to bid on competing brand terms.

    With very few exceptions, the competing brand terms usually settle between 2/10 and 4/10 QS. I don’t know about your clients, but at least with ours the most low QS keywords are derivative terms of competing brands.

    I find it perplexing that you keep mentioning “AD quality score” despite the fact that QS is only reported on keyword level. :)

    But thanks for a very interesting post anyway.

    • Bradd Libby says:

      Toni,

      Thanks for the comment. To address your second point about ad-vs-keyword Quality Score first: please check out Google’s chief economist Hal Varian’s articles, especially the one in the International Journal of Industrial Organization where he describes position-adjusted quality. He consistently refers to ‘ad quality’, not ‘keyword quality’, so I try to do the same.

      AdWords has a feature that will show the higher CTR ads more frequently (for a given keyword), so Google must be tracking CTR at the ad level, not the keyword level. This means that they can (in principle) calculate Quality Score at that level too.

      So, even if Google only reports it at the keyword level, I try to frequently remind myself that the actual value of Quality Score used for any individual impression is probably ad-specific, not keyword-specific.

      Regarding your first point about competitors’ brand terms, I have no reason to doubt that you are right. The data I show above includes brand, non-brand and competitors’ brand terms, so it’s likely that some of the QS 2-4 impressions in the February 2011 data are from competitors’ brands. However, this observation doesn’t settle the issue in my mind of where the QS 8 and 9 terms went. They are extremely rare now, but they were much more common 2 years ago. Google has allowed us to add QS 2-4 terms in the form of competitor brand terms, but why are we no longer seeing QS’s of 8 or 9?

      • Well said. I agree and believe that Google tracks ad level quality (as well as e.g. account level quality) but indeed, separately from keyword level 1-10 QS.

        The very low number of 8′s and 9′s does raise questions and no doubt is part of Google’s monetization strategy. :)

        PS. I have noticed that some highly relevant but still not that common keywords (among advertisers) sometimes get 8′s and 9′s.

    • Bradd Libby says:

      Not only does Google very likely track ad-level quality, but they probably track it right down to the level of what they call the “UI treatment” (that is, the combination of operating system, browser, and so forth).

      I too have seen QS 8s and 9s, but they are not only very rare, but also short-lived. Efficient Frontier’s Dr. Sid Shah called them ‘transitional’ QS values, but to me this implies that a keyword with QS of 7 will get an 8 or 9 on its way up to QS 10 (or vice-versa). What I have primarily seen is words get a QS of 7 for many days in a row, then an 8 or 9 for a day or two, before returning to a 7. So, if you look at a large collection of words, a few will have a QS of 8 or 9 at any given time, but it’s a rotating selection. Each individual word’s presence in the group is more transitory than transitional.

  4. Pavlicko says:

    Brad,

    awesome post. Google playing ‘mind-games’ with its advertisers? Not that I don’t put that past them, but I think it simply comes down to $$$. Lower QS ads cost more to the advertiser. Plain and simple. The lower the QS, the more you’re paying.

    Keeping in mind that many companies try to run their own adwords campaigns without any prior experience managing or optimizing PPC campaigns, they (the big G) artificially inflate the CPC from the very beginning or over time by downgrading the QS. That newbie advertiser has no idea how to raise their QS so they just dump more money into the keyword bids, and as a subtle side effect, this also starts to tank that keyword’s performance CTR across the entire search network – which in turn helps to lower the keyword QS (while raising the bid rate) for all the other advertisers. How tricky.

    I see this as the reason 8s and 9s are disappearing – if we assume 10s are mostly BRAND-specific, the competition level isn’t there and those inept advertisers can’t tank the QS at a global level.

    Which is why it makes perfect sense for Google to give away $100 in advertising to pretty much any new domain registered nowadays – who cares how your ads perform if it’s not your money in the first place, right?

    • Bradd Libby says:

      Thank you for the kind words. In my gut I’m having trouble buying into some of what you’re saying. You say: “Lower QS ads cost more to the advertiser. Plain and simple. The lower the QS, the more you’re paying.”

      That might be true for any individual advertiser, but not when it’s across-the-board. Google multiplies each bid by that ad’s Quality Score. Quality Scores are believed to be based largely on your CTR relative to other ads that have appeared in the same ad slots. The basic reason Google calculates Quality Score is to determine how much more likely (or less likely) your ad is to get a click compared to other ads. So, if they cut my QS in half and your QS in half (and his QS and her QS too), then it’s the same as not changing anything at all.

      I would have an easier time believing your argument if, for each account that saw QS 8′s and 9′s dry up, there were others that saw far more QS 8′s and 9′s over time. If this is the case, I can’t find them among the accounts TSA manages. (Maybe TSA is just especially bad at managing AdWords accounts, but I’ve seen graphs from at least WordStream, Efficient Frontier and ClickEquations that look the same as mine. So, if we’re bad at it, then they are too.)

      And, I’ve have an easier time believing our argument if the accounts I described above were brand new in February 2009, but they were not. Two of three have been managed by TSA since at least 2006. If Google downgrades QS over time, why did these accounts still have 25% or so of their impressions in QS 8 and 9 words in January 2009 (and virtually none 6 months later)?

      I agree completely with your last point, though. It’s in Google’s best interest to get as many advertisers into their system as possible. $100 Google vouchers are to the last decade as AOL CDs were to the late 90′s. But the disappearance of QS 8s and 9s across the board is still a mystery to me.

  5. Terry Whalen says:

    Great article – I think there is definitely something to the notion that there is some neuro-marketing going on here. Quality scores oftentimes seem almost arbitrary – even on exact match keywords with lots of impression volume and high (Google.com) CTRs. Also, when I review duplicate keywords with same match type but in different ad groups, I often see that first page estimated bids do not correspond as they should to QS. In other words, kw 1 has a QS of 7 and kw 2 has a QS of 5 or 6. Kw 1 should have a lower estimated first page bid, since it’s QS is higher – but it often is the opposite case. My thought is that QS (and first page estimated bid) are useful and relevant, but they should be taken with a large grain of salt.

  6. Brian Bien says:

    “$100 Google vouchers are to the last decade as AOL CDs were to the late 90′s”

    Brilliant analogy! I love it. In my opinion, Google is going wild with inflationary tactics, printing excessive AdWords dollars… because they can.

    I would not put it past Google to play mind games with their advertisers. Google no doubt hires psych experts to maximize the billions in revenue they receive through AdWords / AdSense each year. Ever notice how the AdSense account focuses on how much you’ve earned, while the AdWords account buries how much you spend :) ?

    P.S. – found your site through a search for “quality score normalized”

  7. I will right away snatch your rss feed as I can not to find your email subscription hyperlink or e-newsletter service. Do you’ve any? Kindly permit me realize in order that I could subscribe. Thanks.

Trackbacks/Pingbacks

  1. Is Google Exploiting Neuromarketing in Reporting Quality Scores? « Big Engine Media
  2. http://www.watchwatchesonline.com/34.html

Leave a Reply

Follow Us on Twitter

Authors