I bought a package of earplugs called ‘Hearos’ (Get it? “Heroes” … “hear”-os) to use when working with my chainsaw around the yard. Looking at the backside of the package I noticed that there’s this table of the noise reduction rating (‘NRR’, a.k.a. the ‘attenuation’) of the earplugs, in decibels, for 9 different frequencies of sound, measured in Hertz, along with the standard deviation of the attenuation values. There’s even the name of the American National Standards Institute (ANSI) specifications by which the measurements were made, in case I get really bored some weekend and want to try to reproduce their results.
So let’s put this into context: For, say, a million dollars a month in SEM spend, Google won’t tell me the standard deviation of the average position at which even one of my top-performing ads appeared. But for $1.99 some no-name company will tell me the standard deviation of the attenuation of its foam rubber earplugs to a precision of 0.1 decibels at a frequency of 3150 inverse-seconds.
What did you say, Google? Was that “Don’t be evil”? I’m sorry, I think I’m having trouble hearing you. You see, every undergraduate who’s ever memorized the equation for a Gaussian distribution knows that it’s not good enough to know only where the average (that is, the ‘mean’) of a group of values falls, you also need to know some measure of the width of that distribution. (I wrote about this topic in a previous post of mine: Average Position is a Really Perverse Metric . In short, by telling us the mean, but not the standard deviation, Google isn’t giving us enough information to make informed bidding decisions on our ads.) If I told you that your ad had an average position of 2, it would be much different if all of those impressions were in position 2 than if most were in position 1 and a few were in position 8. The averages in these two cases might be the same, but their deviations are much different!
I’m trying to think of what I might say to some hot-shot consultant I hired to do statistical analysis who presented me with a table of averages without their corresponding deviations, or what a math professor might say to a Ph.D. candidate who did the same, and frankly, nothing I can think of is polite enough to reprint on a business-friendly website like this one. If I ever get in the same room with the people who decided to show us only our ad’s average position, but not the standard deviation, I’d like to give them an earful about it. So, perhaps they need those earplugs more than me…