Attack of the Teenage Quality Scores

Posted on Monday, May 3rd, 2010 by Print This Post Print This Post

Categories - Featured, SEM

Unpredictable. Hyperactive. Irrational. And, potentially, very expensive. That’s how a keyword’s Quality Score (QS) can be like a teenager. Perhaps no single quantity in Google’s AdWords system is more mysterious or misunderstood than Quality Score, and what you don’t know about it can hurt your account.

Part of the mystery is because an ad doesn’t have just one Quality Score, there are actually multiple values Google calculates. One version specified before an ad auction occurs determines whether your ad is eligible for the auction at all. Slight variants are used for figuring an ad’s first-page bid and whether the ad can appear in the ‘promoted positions’ just above the organic results. A different Quality Score is calculated within an individual ad auction to determine your Cost-per-Click (CPC) based on the next-lowest-placed competitor’s bid. Another version is used only on the Content Network.

A second part of the mystery behind Quality Score is that the number that Google reports as ‘Quality Score’ is probably not the number that is actually used, for example, in a given auction to determine your CPC. Most likely, the in-auction QS is based largely on your ad’s historical clickthrough rate (CTR), if the ad has enough recent data, and is therefore stored as a number between 0 and 1. Quality Score is displayed as a number between 1 and 10, but there is probably not a linear transformation of the actual Quality Score to the reported Quality Score, since most QSs (about 40-50% of the ads in a typical account) are reported as ’7′ and there are few, if any words in an average account that have CTRs of 70%.

A third reason QS is so difficult to figure out is due to what I call Google’s “Quality Score Memory Hole“. At any time, Google will only report an ad’s current Quality Score, even if you run a report with the data broken down by the ‘Daily’ option.

To see this in action, create a report for a given keyword, with statistics reported by day. (If you know how to do this, you can skip to the next paragraph. If not, here are instructions: Go into the Google AdWords interface and select the ‘Reporting’ tab’s ‘Reports’ option. Create a New Report for ‘Placement / Keyword Performance’. For ‘View’, choose ‘Daily’ and for Data Range select ‘Last fourteen days’, or manually enter the past several weeks or so. You can choose ‘Manually select from a list’ to narrow the report down to one campaign and/or adgroup and the ‘Filter Your Results’ option to specify a single keyword and matchtype (preferably, ‘exact’). Then click the ‘Create Report’ button at the bottom of the screen.)

On April 27, I created a report by day for April for one keyword. The Search Agency previously recorded, every day at the same time each day, that day’s current Quality Score. These are shown below.

So, on April 1, Google reported that the Quality Score was 4. On April 23, they reported that it was 6. On April 26, they reported that it was 7. But, when on April 27 I requested the Quality Scores, by day, for the month of April, they reported that Quality Score each day was 7.

That is, on any given day, Google will only tell you the current Quality Score. If you wish to know how Quality Scores can change over time, you need to record the QS for every keyword which you are interested in every single day, then keep your own record. Let’s look at some of the findings from doing this.

Lesson 1: Quality Scores Can Be Very Unstable

Notice that for the keyword shown above, which has gotten over 400 clicks for a cost over $1000 in April thusfar, the Quality Score changed 5 times in 26 days, or more than once per week. In other words, the Quality Score you see might depend on which day you check. An analysis of hundreds of thousands of keywords across accounts in many different industries shows that upwards of 20% of clicks and 20% of the cost in an account might be in ‘unstable’ ads whose Quality Score changes more than once per week or by 3 or more QS points in a month. On average, about 15% of clicks and cost are in these ‘unstable’ ads.

In his new book Advanced Google AdWords, Brad Geddes recommends that “if you have a low quality score, you should optimize your quality score instead of raising bids.” (p. 202) But note that for this ad, The Search Agency did not try to “optimize” Quality Score to get it from 4 to 7. We didn’t “focus” on maximizing its CTR. In fact, we didn’t do anything at all. These changes in Quality Score are purely the natural fluctuations for this ad. You would not take financial advice from a teenager. You would not take career advice from a teenager. So, why take account management advice from a Quality Score that acts like a teenager?

The take-home lesson is: if you see a low QS, don’t panic. Do not immediately revise ad text nor make other drastic changes. Just check again the following day or the following week to see if QS remains low. It is only a problem when an ad has a chronically low QS and that ad is underperforming. (In fact, the ad above generated more profit per day and at a higher ROI when its QS was 4 than when it was 7.)

Lesson 2: A Higher Quality Score Does Not Necessarily Mean a Lower Cost-per-Click

One of the most persistent myths in search marketing is that CPC drops as Quality Score increases. Google even says explicitly on their AdWords help pages: “The higher a keyword’s Quality Score, the lower its cost-per-clicks (CPCs) and the better its ad position.” However, as we pointed out earlier, this is only true within a single ad auction and, furthermore, Google does not tell you your actual Quality Score.

Looking at the CPC for the keyword mentioned above by day for March and (most of) April, you can see that the average CPC when the Quality Score was 4 was just under $5 (actually, $4.81). When the Quality Score was 7, the average CPC was also $4.81, even though the bids and average position were stable during this time. When the Quality Score was 6, the average CPC was lower than when the QS was either 7 or 4.

In fact, examining many high-performing keywords, one finds that there is very little correlation between the CPC and the (reported) Quality Score. In some case, CPCs fall as QS rises. In some cases, they rise as QS rises. In most cases, like the one shown above, there is no obvious relation.

Lesson 3: Google’s Quality Score Algorithm Changes Periodically

Google’s Quality Score algorithm is believed to undergo periodic updates and a significant one might have happened on April 1, 2010 (and that’s no belated April Fool’s joke). A graph for the account of the keyword described above shows that the number of ads whose Quality Score changed between any two days averaged about 100 prior to April 1, 2010, then spiked to over 400 on April 1. The number of ads whose QS changed between any two days has been more volatile since that date than before it.

A similar pattern is seen in many accounts, indicating that this change is not specific to this one. This might have been a one-time recalibration done on April 1, or might represent the implementation of an ongoing change in the way Quality Score is calculated, but there’s no way to tell for sure at this point.

Lesson 4: Google Might be Manipulating the Numbers Even Further

Google claims that the only data which is used to determine Quality Score comes from searches where the search query is identical to the keyword’s text. This implies that, if an account bids on ‘leather shoes’, exact match and ‘leather shoes’, phrase match, then both keywords should have the same Quality Score.

However, in a comment on Search Engine Land and in communications via Twitter, Brad Geddes claimed that that this might not always the case. In particular, he’s found examples of high click-volume keywords where the Quality Scores differ across matchtypes. “Same ad group, one ad, same landing page,” he says, “[but] broad and exact match have widely different quality scores (biggest one I saw was 10 for exact and 4 for broad, another was 7 for broad and 2 for exact). So – there’s something else going on there with QS and match types than Google is letting on.”

Though some people can be led into false senses of complacency or panic due to high or low reported Quality Scores, before using QS to prompt any changes in how you manage your account, remember that many Quality Scores can act like teenagers. So, be sure to look not only at their current values, but also how they change over time.

Tags |

9 Responses to “Attack of the Teenage Quality Scores”

  1. brad says:

    Great post and totally agree.

    “It is only a problem when an ad has a chronically low QS and that ad is underperforming.” – fantastic statement.

    Google once told me that about every month or so they update their QS formula. It’s on those days that you might see some dramatic changes in your account without you doing anything at all.

  2. papercube says:

    How would this analysis relate to match type? As I understand QS is really pegged to exact match which in earnest is rarely the case used.

    • Bradd Libby says:

      Google claims that Quality Score is calculated only from data from searches where the query is identical to the keyword’s text.

      So, if you bid only on ‘leather shoes’/exact match, then presumably every query will be used for calculating QS. And, if you bid only on ‘leather shoes’/phrase match, then only the times when the query was ‘leather shoes’ will the data be used.

      This implies that if you both on both matchtypes, then they should always have the same QS. However, as I mentioned, Brad Geddes claims to have seen cases where a keyword that has gotten many clicks has different QS’s for different matchtypes.

      He also brought up an interesting question in personal communication: If you bid solely on ‘leather shoes’/phrase match, and add an exact match negative, how do Google calculate your QS? He didn’t know the answer and I can’t think of one either.

      For the analysis I did for this blog post, I only looked at exact match keywords.

  3. Hi Bradd,

    This is a great post. Very well written and convincing arguments.

    Your point about Google failing to report Quality Score historically (your ‘Quality Score memory hole’), as well as regular Quality Score fluctuations, are surely deliberate moves by Google to prevent smart PPC analysts (such as yourself) trying to ‘beat’ the algorithm.

    We see the same thing with PPC ads jumping positions when searching for the same keyword multiple times on Google’s ad preview page; we also see organic rankings changing daily. I don’t think there’s too much going on with the algorithm. I think it’s more Google trying to confuse us, make analysis difficult and keep their secrets secret.

    I personally think too much emphasis is placed on Quality Score. Since Quality Score is never the end goal for a business, there is little point ‘optimising’ for it. Better to focus on relevancy, conversion rate, ROI and user engagement, and Quality Score will naturally follow.

    Focusing too much on beating the system will leave you frustrated and out of touch with your primary objectives – to engage with your target audience and deliver a return on investment.

    Thanks for a fascinating article.

    Alan

  4. Tanya says:

    Very cool article, Brad. This was very enlightening for someone with a primarily SEO background, who is learning about SEM.

    I kept hearing “quality score” mentioned and understood that it was important, but your article puts it into perspective.

  5. Carl says:

    Great article, Bradd. It shows really clearly that while QS may or may not be a useful as a qualitative metric, it certainly is completely useless as a quantitative one!

  6. Terry Whalen says:

    Great article – Allan, totally agreed with your conclusion.

Trackbacks/Pingbacks

  1. Last Week in PPC News and Views: May 10, 2010 | Clix Marketing PPC Blog

Leave a Reply

Follow Us on Twitter

Authors