I’ve got a mystery I’m trying to solve. You might call it ‘The Case of the Missing Quality Scores‘.
For each keyword where an advertiser places an ad, Google reports a Quality Score number between 1 and 10. All indications are that this number is largely based on the ad’s clickthrough rate compared to other ads that have appeared in the same positions in the search engine results.
It doesn’t take long to see that the most commonly reported Quality Score for many accounts is 7, with 10 being seen for the best-performing terms (often, brand terms) and numbers below 7 being considered ‘Poor’ or just ‘OK’. In an article published at Search Engine Land in October 2009, Efficient Frontier’s Dr. Siddharth Shah noted that Quality Scores of 8 and 9 are “very rare” .
But it hasn’t always been that way.
The graph below shows, for each Quality Score, the fraction of the account’s impressions whose ad had that Quality Score, for February 2011 and February 2009 for an account that has spent about $1 million per month on pay-per-click advertising. Because Google’s Quality Score calculations only use data from cases where the search query is identical to the keyword, the results shown here are only for exact-match terms.
In February 2009, more than half of the impressions occurred in ads whose Quality Score was 7, with higher and lower Quality Scores being less common depending on how far from ‘7’ they are. By February 2011 though, the picture had changed radically. Now, Quality Scores of 3-7 are very common, with 8s and 9s being seen less often than 1s and 2s.
This change would be easy to explain if the performance of this account had crumbled over the past couple of years, but in fact the opposite is true. Impressions, clicks, conversions and net profit are at record highs. The account-wide Conversion Rate has doubled in the past two years. CPCs are up  (as they should be). Yet the account-average Quality Score has fallen from about 7 to about 5.
Quality Scores of 8 and 9 are now very rare, but where have they gone?
The graph below shows the percentage of impressions (for the same account as the graphs above) in ads in several Quality Score categories for the past couple of years.
Quality Scores of 8 and 9 largely disappeared in the spring of 2009. However, the fraction of impressions in ads with QS of 1-6 has continued to rise, leveling off for this account (perhaps temporarily?) only in the past couple of months.
This change was not simply due to a large change in the traffic received by keywords with a certain Quality Score – the absolute number of ads with QSs of 8 or 9 became extremely low, while the number with low QSs increased.
And again, this change might be easy to dismiss if it was just one account. But it’s across-the-board. For those accounts that TSA has managed for the past two years, in early 2009 on average about 20% or more of the impressions in a given account were from ads whose Quality Score was 8 or 9. A few months later, you basically need scientific notation to report that number. For all of the top-spending accounts The Search Agency manages, you won’t now find one where the percentage of impressions in ads with a QS of 8 or 9 exceeds 1/10th of 1%.
(click image to enlarge)
Why is that the case? I’m thinking that perhaps Google is capitalizing on a basic aspect of human psychology to prompt Quality Score optimization efforts.
The premise is simple: Nobody likes to think they are below-average in something. This fact is the basis of both the Dunning-Kruger Effect  (the idea that below-average performers don’t recognize that they are below average) and the Lake Wobegon Effect  (the reason that most drivers consider themselves above-average at driving), as well as numerous, um, personal medical devices of dubious efficacy.
When you make changes to your AdWords account, in fact you are working for Google. As Alex Cohen of ClickEquations notes, Google might be able to do the dirty work of choosing keywords for you . But, they can just as easily let you choose the keywords, knowing that you’ll try to pick keywords that will make you the most money. So, when you choose a keyword to bid on, you are acting as Google’s meat-based  query-to-ad matching algorithm.
When you change the text of an ad or the bid on a keyword, you are changing (a portion of) Google’s search engine results pages.
And when you modify your AdWords account, you are not paying attention to your Bing (or, formerly, Yahoo) account.
Therefore, it’s in Google’s best interest to get you to devote as much of your time and attention as possible to their platform. One great way to do that, it seems to me, it to simply tell everyone, regardless of their actual status, that they are “below average” and then wait for knee-jerk human psychology to react.
Let’s say it’s February 2009 and you log in into your AdWords account and sort the top-performing keywords by their number of impressions. You see that about 60% are 7 or 10 and about 20% 8 or 9. Only 20% are 6 or below. That is, 80% of your impressions are in words that are 7 or above. What do you do? Probably find something else to worry about.
OK, now say it’s 2 years later. You log in and find that 75% of your impressions are in ads whose Quality Score is 6 or less. To make things worse, every account-wide average Quality Score looks worse than the month before. Now what do you do? Find some new keywords! Change some ad text! Spend more time on AdWords! Optimize that account!
The biggest lessons for online marketers, I think, are:
If you do month-over-month comparisons of your impression-weighted Quality Score, realize that the percentage of impressions in keywords whose Quality Score is 1-6 still might not have stabilized 2 years after 8s and 9s virtually disappeared from Google’s reporting.
If you’re wondering why your Quality Scores are disappointingly low, realize that Google might simply not be reporting many Quality Scores as being above average.
And, realize that Google might simply be reporting ‘below average’ Quality Scores for everybody, specifically to get you to focus your attention on AdWords as much as possible.