Some other guy stopped by and started eying the mangoes. So, I said “I’ll take the big basket for $2 a piece, please.” The farmer handed me the small basket. I said, “No, I want the big basket.” And the farmer replied, “Well now there’s someone else here who wants mangoes, so I’ll only sell you the small basket for $2 a piece. If you want the big basket, they’re now $5 each.”

So I gave him $20 for the 10 mangoes in the small basket and he gave me back $6 in change. ”What’s the change for?”, I asked. “Ten mangoes at $2 each should be $20.” “Well,” he said, “I count every dollar from you as if it’s worth $1.25.” So I held up the $6 and asked, “Then can I have a few more mangoes?”

“Don’t be ridiculous!” he replied. “What crazy kind of farm stand do you think this is?”

Click traffic is like food for a website and that crazy farm stand I go to, of course, is paid search. The search engines claim that their systems are auction-based. Some even say that the second-price mechanism by which ad slots are allocated promote ‘truthful’ bidding – that is, each advertiser, by the design of the system, has the incentive to bid honestly, without any regard to the bids of other participants.

If paid search is an auction, though, it’s a very strange one indeed. Here are some ways:

1. In paid search, both the seller and the auctioneer are the same entity. This is seen in some auctions, like those by governments for public debt or broadcast spectrum rights, but in most auctions (like eBay) the auctioneer and the seller(s) are different groups.

2. The rules are not fully disclosed. The search engines claim that fully disclosing the rules would make their systems more vulnerable to spammers and other low-grade advertisers. But if their systems encourage truthful bidding, then this should not be the case. Logically, they cannot claim both that (1) truthful bidding is an equilibrium outcome of their system and that (2) they need to keep some aspects of the system confidential in order to avoid low-grade advertisers. If truthful bidding is an equilibrium outcome, then it should be possible for them to fully disclose the rules.

3. Bidders can be excluded from any single auction (or category of auctions) on a whim. It’s strange and disturbing that the search engines often provide no greater explanation for why an ad is not showing for a particular keyword besides saying that the ad is ‘low quality’.

4. There are advertiser-specific minimum bids. One little-known aspect of AdWords, for example, is that for each keyword, for each advertiser, Google sets a minimum bid that advertiser must supply to be allowed into the auction. If no advertiser bids more than their minimum amount, no ads are shown. If one or more ads are shown, the lowest-placed ad must pay the minimum CPC chosen for that advertiser.

5. Bids are weighted. As with the farm stand above, in paid search the auctioneer treats money from some advertisers as being worth more than money from others, via the ‘Quality Score’ or ‘Quality Index’. (And again, the exact calculations behind these weighting factors are secret.) Dictionary.com says an auction is “a publicly held sale at which property or goods are sold to the highest bidder” (emphasis added) yet the search engines readily admit that the highest bidder is not guaranteed top placement in the results, or even placement in the results at all. (It’s surprising how often people get this wrong. A recent LA Times article said: “[Google] continues to advocate an advertising auction model that’s been successful in its core search business, whereby search terms are sold to the highest bidder.” This is simply untrue.)

6. You can bid against yourself. Say you bid on the phrase-match version of the word “shoes” and the phrase-match version of “leather”. A user enters a search for “leather shoes”. Which of your ads (if either) gets shown? Presumably, the search engine picks whichever will make them more money per impression, on average. If you raise the bid on the lower-CTR ad sufficiently, the search engine will make more money by starting to show that ad instead. So, a bid increase on “leather” can cause your traffic for “shoes” to drop, and vice versa. If you don’t spot the conflict, the engine has you bidding against yourself.

7. The auctioneer will bid for you. Google’s ‘Conversion Optimizer’ and its new ‘enhanced CPC‘ feature bid based on the performance of an advertiser’s ads. The advertiser puts code onto its website which gives Google conversion data (sales, revenue, and so forth). Of course, this makes it so that Google knows the full spectrum of data in the search process, from how many queries occur to how many sales result, and their value. Their feature is called the ‘Conversion Optimizer’, but what exactly they are ‘optimizing’ for is not made clear. (Notice that it is not called the ‘Conversion Maximizer‘.)

8. The auctioneer only reveals whatever data it wants to. When I went to an antiques auction near The Search Agency’s office recently, they had a viewing period before the auction started. Every bidder was allowed to browse the items and examine them (with the seller’s permission). As each auction ran, the participants bid by holding up paper signs. That is, every piece of information known to the auctioneer was known to all of the bidders. But in paid search, the engine only reveals the information it wants to. You cannot audit the records to find out who appeared where in any individual auction nor how much they bid, nor even to simply make certain that the engine has done its accounting correctly.

9. The auctioneer charges you to set or change your bid. To keep down the frequency of bid changes, Google permits advertisers to access their system a certain number of times per month via API for free. Additional requests for information incur a charge. (If AdWords promoted truthful bidding, it would not be necessary to surpress the frequency of bid changes.)

10. The ‘second-price’ mechanism is novel. The process the search engines use to allocate space is called the “generalized second-price” (or GSP) mechanism, but this term did not appear in auction theory literature until 2005. Many different kinds of auctions have been developed over the centuries (the Dutch auction, the rising-price English auction, and so on), yet the GSP mechanism apparently was not devised until just a few years ago. Perhaps the reason why the term ‘generalized second-price’ never before appeared in auction theory literature prior to this is that this system is just not an auction at all.

What then is paid search if not an auction? At Search Engine Watch, Alex Cohen called the system a ‘negotiation‘ and with that term I think he struck the nail on the head. With each keyword you choose and bid you submit, you are providing information to the search engine about your interest in buying certain traffic. The presence of other advertisers, the bids and the budgets, simply act as constraints with which the search engine must contend in order to maximize their own revenue. Just like at the farm stand, when dealing with paid search advertising, it’s not the other customers you need to worry about.

Many AdWords account managers spend, in my opinion, far too much time fiddling with bids. If an account has an overall cost-per-acquisition (CPA) target of $15 per conversion, a portfolio-like approach might suggest permitting one high-performing keyword to hit a $20 level while another is restricted only $10. Combined, their CPA might hit the overall target. To reach this noble goal, some people resort to what I have called the ‘Goldilocks Approach to Bid Management‘: if the CPA of an ad is above target, reduce the bid; if below target, raise the bid. It’s easy to show, though, that this approach might not be optimal.

Below are statistics from a keyword in an actual account that TSA manages. The keyword consists of two terms, the first of which is a verb describing intent to purchase (like ‘get’ or ‘buy’ or ‘find’) and the second a common generic noun (like ‘car’ or ‘shoes’ or ‘lamp’). The account bids on both the exact-match variant and the phrase-match variant of this word.

The tables below are based on actual performance data and on numbers provided by Google’s Bid Simulator (GBS), though the conversion rates have been rounded to help protect client data and some quantities estimated for demonstration purposes. In the month of June, the exact-match variant had an average position of 3.5 and the phrase-match variant an average position of 4.6.

Imagine that the spending rate in our account is such that we are going to finish the month with literally a few dollars left. We’d like to raise the bid on one keyword by just a penny to boost our spending that very small amount. We are currently bidding $0.78/click for the exact-match variant. It is getting about 5 times the traffic of the phrase-match variant, has a 3% conversion rate and is currently getting a CPA of $10.83. (The account-wide target CPA is $15.00, so this word is “CPA-positive”.) The phrase-match variant has a bid of $0.37/click, a conversion rate of 2.0% and is currently getting a CPA of $16.65 (meaning that variant is already above the account’s CPA target, so it is “CPA-negative”).

If we increase the bid of either of these variants by just one penny, the increase in ad cost will be enough to spend those extra couple of dollars we are looking to invest. The question is: which variant should get the bid increase?

By almost all measures, the exact-match variant is the Strong Horse. It has more traffic, a higher position, a higher conversion rate, a higher CTR, a lower CPC, is getting a lower CPA and is CPA-positive. (The only place where the exact-match is not the winner is that both variants have a Quality Score of 7. So, there it is a tie.) Increasing the bid for the exact-match variant from $0.78 to $0.79 per click will increase our cost by a few dollars, but also bring in conversions. If we take the ratio of the change in Cost to the change in conversions, we find that the marginal CPA (that is, the cost of an additional conversion) is $32.07.

The phrase-match variant has less traffic, a lower position, a lower conversion rate, a higher CPC, is getting a higher CPA and is CPA-negative. But increasing the bid for this variant brings in additional conversions at a marginal CPA of only $26.09. Even if you’re not the gambling type, betting on the Weak Horse is the clear winner. (One problem with relying on the Google Bid Simulator’s estimates to do this calculation in practice is that they are usually not finely grained enough to be able to calculate the marginal CPA directly, without doing some additional calculations first.)

The lesson is: When making bid changes, the traffic level, CPC, CTR, CPA, Quality Score, average position and conversion rate by themselves tell you nothing of any importance. If you have one additional dollar to spend, spend it on the word with the lowest marginal CPA – that is, the word that will get you the most conversions per dollar. That, inshallah, is the swiftest way to win the race.

Insofar as an account manager rules the keywords and ads of a pay-per-click (PPC) marketing campaign like a tormenting angel, so too must the bitterly pragmatic advice for a sovereign that Niccolo Machiavelli dispensed in The Prince and other works hold merit.

Though his name now is synonymous with malfeasance, many of Machiavelli’s dicta were fairly mundane: “Whosoever desires constant success must change his conduct with the times,” he wrote. And: “Entrepreneurs are simply those who understand that there is little difference between obstacle and opportunity, and are able to turn both to their advantage.” Who would argue with such sentiments?

Even his more-cynical observations, like “Of mankind we may say in general they are fickle, hypocritical, and greedy of gain” still find currency. In Small is the New Big, for example, Seth Godin said, “People are selfish, lazy, uninformed and impatient. Start with that and you’ll be pleasantly surprised by what you find.”

His philosophy can perhaps be summarized by what I call The Modified Golden Rule: “Do unto others as you fear they might do unto you.’ And it is apparent from even a cursory reading that Machiavelli’s most-renowned works hold much useful advice for the modern digital marketer.

In the opening scene of The Merchant of Venice, friends of Antonio (the title character) ask him why he seems distressed. Is he, perchance, worried about his ships at sea? No, Antonio demures: “My ventures are not in one bottom trusted, nor to one place; nor is my whole estate upon the fortune of this present year. Therefore my merchandise makes me not sad.”

Yet many digital marketers too often are needlessly made sad by their merchandise. Perhaps a word that yielded a few conversions last week gave none this week. Perhaps CPCs or Quality Scores have changed suddenly and without apparent reason.

In this respect, Google’s Bid Simulator (GBS) feature is intended to provide some guidance to help digital marketers maximize their profit. In ‘How to Calculate Profit-Maximizing ROI‘ and ‘Optimal Bidding, Part 1: Behind the Scenes of ‘Google AdWords Bidding Tutorial‘, I showed how the GBS’s estimates can be used to find the profit-maximizing bid for a given keyword. Let’s look at another example.

Consider a word that generates revenue of $5 per click, on average. (Say the ad gets $100 per conversion and the conversion rate is 5%, for example.) The Bid Simulator might provide estimates of the number of Clicks that ad can expect per week at several different possible bids. If so, then it might also give estimates of the weekly Cost expected for those clicks (from which the average cost-per-click can be calculated.) The numbers in the figure are fictitious, but realistic.

The expected weekly Revenue can be calculated from knowing that each click on average, for this particular example, brings in $5. The expected weekly Profit can then be found by subtracting the Cost (that is, Clicks times CPC) from the Revenue. (The numbers above are internally consistent, but might not appear so due to rounding.)

Form the table we can see that a bid of $4.60/click maximizes our profit. To bid lower gives up clicks at a profitable price, thus reducing our profit. To bid higher brings in clicks at too high a cost, which also reduces our profit. In fact, by modeling the relationships of ‘Clicks vs. bid’ and ‘CPC vs. bid’, we can calculate the expected weekly Profit at any individual bid, not just the limited options the GBS provides.

In the figure below, we can see the expected weekly Profit for any bid:

We see that a bid of $4.60 per click maximizes our profit. (This is why Hal Varian, Google’s chief economist, refers to the profit-maximizing bid as the ‘optimal’ bid.) To bid more reduces our expected weekly profit such that, at a little over $6.00 per click, we will make no profit at all (and to bid even higher will cause our expected profit to become negative).

But in a cutthroat bidding environment, it might sometimes be wise to settle for less profit for ourselves if it will hurt our competitors more. I call this concept ‘maximization of relative benefit‘.

Look closely again at the ‘Profit vs. bid’ graph. The profit-maximizing bid nets us, on average, $317.50 per week. But if we bid a little over $5 per click, we will still net a profit of around $300 per week. Our profit will be reduced by $17.50 (a bit over 5%), but the next-highest-positioned competitors will see their CPCs rise, in any auction in which we both participate, by $5.00/$4.60, or about 9%, regardless of their Quality Score.

Even in the worst case (that is, an ad which never converts for us and therefore brings us no revenue) our ad still occupies space, robbing your competitors of precious clicks. Thus, for one who values the pain caused to competitors more than the price paid to get that traffic, it is worthwhile to bid even for a non-performing term.

For ads that do generate business, the economics are even more compelling. Bidding slightly above the profit-maximizing level:

1. increases the CPC for the next-highest competitor (for a single ad auction)
2. reduces the number of clicks competitors are likely to accrue (for a collection of ad auctions), and
3. nets you more clicks (even though they are at an incremental price above which you should be willing to pay if you were a strict profit-maximizing bidder).

Regarding the second point, of course, there might not be a one-to-one relationship between the number of additional clicks you receive and the number your competitors lose, since your change in bid might, for example, make Google alter the number of ads they show and therefore the total number of clicks the paid listings receive. (I explained this effect in ‘Pay Attention to the Man Behind the Curtain.’)

In the graph below I have calculated the effect of small increases in one’s bid over one’s profit-maximizing level for the cases where each additional click we gain takes 0.00, 0.10 or 0.25 clicks from the next-highest-positioned competitor. (This analysis assumes that there is a next-highest-positioned competitor and that that competitor get the same amount in revenue per click, on average, as we do.)

Even when our higher bid takes no clicks from the next-highest competitor, it still drives up their CPC. For a bid only 5% above our profit-maximizing bid (in this case, $4.83), our expected weekly profit drops to $314.65, a loss of less than $3 per week to us. However, it costs our next-highest-positioned competitor about $11 per week (an extra loss of about $8 above our loss). If we increase our bid by 10%, we will sacrifice about $10 per week, but cost our competitor about $18. These amounts might not seem large, but if our competitor loses 0.25 clicks for each additional click we gain, then his loss should average about $30 per week, for a word that, if his economics are similar to ours, would otherwise generate over $300 in profit. That’s a nice chunk of his profit and perhaps sufficient to harm his overall marketing efforts if done on a wide enough array of words or on waves of carefully selected terms.

(This concept differs from so-called ‘vindictive bidding’ in that being vindictive involves bidding the highest amount possible to achieve our profit-maximizing CPC. But ‘relative benefit maximization’ involves aiming for a target CPC above our own profit-maximizing level simply because, for small bid increases, doing so hurts our competitors significantly more than it hurts us.)

In The Merchant of Venice when Antonio fails to pay his debt, Shylock demands a pound of flesh as compensation (and for this Shylock is regarded as one of Shakespeare’s cruelest villains). But we can see that for the example above, exceeding our profit-maximizing bid (up to a certain level) can hurt the next-highest advertiser’s profit worse than ours.

Even bidding just $0.05 (about 1%) over our ‘optimal’ bid cuts his profit noticeably while hardly affecting ours at all. So, while Shakespeare wished for his audience to hold Shylock in contempt, judging from the particular circumstances in this case by seeking only one pound of flesh, in my opinion, Shylock underbid.

Done properly, bidding to maximize one’s relative benefit can be a useful part of driving competitors out of a given set of keywords (or, with luck, out of a given market completely). Perhaps Machiavelli put it best: “If an injury has to be done to a man it should be so severe that his vengeance need not be feared.”

This relationship frequently pops up in situations dominated by a ‘power law distribution’, that is, when many entities have (or contribute) little individually, a moderate number have moderate impact, and very few entities are highly important.  This is what search marketers typically see in their ‘head’, ‘body’ and ‘long tail’ keywords.

If 20% of ad spend generates 80% of conversions, then it’s reasonable to ask if it’s possible to get 80% of that result (about 64% of conversions) from only 20% of that level of spend (about 4%). Similarly, perhaps it’s possible to get 80% of those results (about 50% of conversions) from only about 1% of the spend. Which equation yields 80% when the input is 20%, yields 64% when the input is 4%, and so forth? Mathematically, it is written Conversions = Spend^n and it’s straightforward to show that this equation describes Pareto’s relationship when ‘n’ is the base-p logarithm of (1-p), where p = 0.20:

In case the presence of a logarithm in an equation is giving you the cold shakes, remember that it’s just a decimal number. For the 80/20 case, it is about 0.13.

All this says is that, if Pareto is right for digital marketing, then the best-performing keywords that account for 50% of our spending should yield about 91% (0.5^0.13) of our conversions. To check this, I generated for various accounts a list of the keywords (treating each keyword’s matchtype as a separate entity) that have gotten at least 1 click in the 3-month period between March and May 2010. (I considered only words in Google’s Search network, not the Content network nor other search engines.) I sorted the list from those that got the most conversions per dollar to the least. I also calculated the total amount spent by all of the keywords and the total number of conversions that they produced. I was then able to calculate, for each entity, the cumulative fraction of money spent by that entity (and all the entities above it on the list) versus the cumulative fraction of conversions generated by that entity (and all the entities above it on the list).

First I considered a company that manages high-end rental real estate in major markets in the US. It turns out that, for this account, the ’80/20′ Rule is actually much closer to the ’70/30′ Rule (where p=0.30 and 1-p = 0.70):

For this account, 97.5% of all the spend was in words that generated at least 1 conversion. Considering only the conversion-generating keywords, we see that the ’70/30′ Rule (actually, to be more precise, the ’71/29′ Rule) is a very good representation of the relationship between Conversions and Spend. That is, the most efficient words generated about 1% of the total spending and 25% of the total number of conversions. About 10% of the spend generated about 50% of the conversions. And about 30% of the spend generated about 70% of the conversions.

An almost identical relationship was seen for a builder of new homes and for a seller of international calling cards in the US. Overall, 30% of the spending accounted for 70% of the conversions. However, among non-brand-related terms, the relationship for both of these accounts followed the 60/40 Rule, 40% of spending accounted for 60% of conversions.

The 60/40 Rule is commonly seen among non-brand terms, especially in very competitive industries with thin margins and little brand loyalty, such as was seen with an online retailer of movie tickets and a seller of gold bullion. Perhaps one measure of the strength of a company’s online brand is how much deviation brand-related terms introduce compared to when they are omitted from this sort of analysis.

The fraction of conversions generated by the best-performing non-brand keywords often follows the ’60/40′ Rule, equalling about the square root of the fraction of money spent on them. (This relationship has been noted before by Alan Rimm-Kaufman who in turn was quoting others.) Deviations from this rule might be due to budget limits artificially curtailing the performance of best-performing terms. If the performance is closer to the 80/20 rule, this sometimes signals an overdependence on brand-related terms.

As another guideline, you should generally see about 90+% of your spend in words that have generated at least 1 conversion in past 3 months. If more than 10% of your spend is in words that have not generated a conversion in the past 3 months, there might be a problem with the structure of your account, your bidding, or your data tracking. Whether you call it the ‘Pareto Rule’, the ’60/40 Rule’, or the ‘Square Root Rule’, though, the underlying principle is the same.

For more details, please check out my article at Search Engine Journal.

This year’s competition hosted 11 teams from universities around the world, including some who participated in 2009′s inaugural Ad Auctions tournament, like Wayne State University and the University of Zagreb (Croatia), and some new entrants, like Tel-Aviv University (Israel) and the Nanyang Technological University (Singapore).

Congratulations are in order for all of the tournament’s participants (a full list of whom can be found at the TAC/AA website run by the Swedish Institute of Computer Science), but especially for this year’s champion agent, TacTex, designed by Dr. David Pardoe of the University of Texas. Second place was won by Schlemazl, designed by a team from Brown University. Perhaps the most surprising finish was the third-place position taken by Mertacor, designed by Andreas Symeonidis of the Aristotle University of Thessaloniki (Greece). Though Mertacor didn’t even make the final round in the 2009 tournament, in a nail-biting finish this year it jumped into second place in the 82nd game (out of 88 total), before losing that position to Brown’s agent with just two games left before the end of the tournament.

“Our second-place finish [over Mertacor] had no statistical significance at all,” Brown team member Jordan Berg demurred immediately after the tournament on the bus ride to a reception at Microsoft’s New England Research & Development (NERD) Center’s building overlooking the Charles River. “A few games before the end, they were in second place. At the end we were in second place. If there were a few more games…” he said, pausing to shrug his shoulders, “…eh, who knows?”

Though the basic rules of the competition were largely unchanged from 2009′s inaugural Ad Auctions tournament, the designers at the University of Michigan, including Professor Mike Wellman, and Dr. Patrick Jordan (now of Yahoo! Labs, but who helped develop the game while a graduate student there) made some notable modifications. “The participants in the 2009 tournament created some very clever strategies,” Dr. Jordan explained, “exploiting information in ways we did not anticipate.”

One change, made at the suggestion of Professor Amy Greenwald of Brown University, was that the agents now receive only an approximate value of the average position of their competitors’ ads, rather than an exact value. “Amy’s reasoning was that this more closely models the real-world ad auctions problem,” explained Brown team member and Prof. Greenwald’s graduate student, Eric Sodomka, “you may be able to perform some searches to get a rough idea of your opponents’ average positions, but you don’t receive this information precisely.”

Another notable modification was the use of higher reserve scores. In 2009′s competition, the simulated search engine set a very low minimum (or ‘reserve’) bid which an advertiser was required to supply to be able to participate in a given ad auction. But part of Dr. Jordan’s Ph.D. thesis showed that the search engine could get a much higher revenue for itself by using moderate reserve scores. (In effect, with the higher reserve scores, the search engine took more money for itself that otherwise would have become profit for the advertisers.)

One result was that almost all of the agents saw their 60-day profit levels fall significantly from the 2009 tournament to 2010′s. For example TacTex, Dr. Pardoe’s champion agent, averaged about $79,000 profit in 2009 but saw its average profit cut to about $58,000 in 2010. Schlemazl, the second-place finisher, went from about $75,000 to $53,000. Even Mertacor, which was eliminated in the semi-finals in 2009 when it earned about $62,000 in profit, won third place with about $52,000 in 2010. (The biggest exception to this trend was the CrocodileAgent, designed primarily by Master’s degree student Irena Siranovic of the University of Zagreb, which averaged only $34,000 in profit in 2009 but about $48,000 in 2010, for a 6th-place finish.)

Perhaps what was most surprising about the Ad Agent competition, both in 2009 and 2010, was how close the finishers were to each other given their diversity of approaches to this problem. The CrocodileAgent team’s advisor, Professor Vedran Podobnik, explained that the newer version of their agent focuses much more on the conversion rate (the ratio of the number of conversions and clicks), the distribution capacity of the advertisers (an upper limit on the number of conversions an advertiser can fill before beginning to receive a penalty to the conversion rate), and on daily spending limits. Dr. Pardoe’s agent uses the information given about the average position of each advertiser’s ad to determine when each agent entered and exited the daily bidding, among other things. QuakTac, a 2009 entrant from the University of Pennsylvania, relied on ‘bid shading’, the practice of bidding a certain amount below one’s true value-per-click in order to help guarantee profit. And Brown’s agent used a mixture of the ‘multi-choice knapsack problem’ (MCKP) algorithm and rules-based approaches. “Patrick Jordan, Michael Wellman, and the rest of the Michigan team deserve much credit,” Eric Sodomka of Brown University said,” for making a game that has produced what seems to be a wide variety of viable strategies to the ad auctions problem.”

So I asked David Pardoe, who will soon finish the work for his Ph.D. degree, since over the past 8 years he’s won half of the bidding agent competitions he’s entered, including both of the TAC Ad Auction tournaments, “What are you going to do now, go to Disneyland?”

“Yeah,” he answered skeptically, slowly bobbing his head, “I’m job hunting. I’m looking for a job.”

Don’t misunderstand: The Search Agency uses a portfolio-based technique which is not derived from Modern Portfolio Theory (MPT) for the analysis of our own clients’ accounts. It’s partially from developing this technique that we were able to discover some of MPT’s dirty little secrets:

1. It’s actually just a production function.

Efficient Frontier shows diagrams on their website of ‘Conversions vs. Ad Cost’ and claims that the relationship between these quantities is called an ‘efficient frontier’. These diagrams are similar to the one shown below:

For different possible levels of spending, this chart shows how many conversions one can expect. There is a maximum amount of spending and conversions (marked by a terminal point) even if all the words in the account were pushed into first position. A green ‘X’ marks the actual performance of the account over the past week, for example.

EF calls this relationship an ‘efficient frontier’. Heck, they even named their company after it. But, unfortunately, it’s not an efficient frontier. It’s just a production function. (Check out the graphs in Wikipedia’s entry for ‘production function‘, to see what I mean. Plotted on the vertical axis is ‘units of output per time’, like Conversions. The horizontal axis is ‘units of input per time’, like Ad Spending.)

To graph an actual efficient frontier we must plot the average return from a basket of assets versus the covariance of the value of that basket. The purpose of an actual efficient frontier is to determine the optimal allocation of assets – for instance, what percentage of stock (or bond) A and stock (or bond) B we should own in order to maximize our expected return for a given level of risk.

I realize ‘production function’ doesn’t sound as sexy as “efficient frontier”, but it’s the correct term. So either Efficient Frontier doesn’t know the difference between a production function and an efficient frontier, or they are simply hoping that you don’t.

2. There’s also a “Deficient Frontier”

Efficient Frontier’s version of portfolio theory is concerned with identifying the maximum number of conversions an account can expect for various levels of ad spending. But to judge how effective your account management is, you also need to consider the minimum number of conversions you can expect.

In the 1985 comedy Brewster’s Millions, Richard Pryor inherits a fortune, but to get the money he must spend one-tenth as much in 30 days and have absolutely nothing to show for it. Of course, the minimum number of conversions a given amount of ad spending could theoretically generate is always zero. But as Richard Pryor found out, it is extremely difficult to spend lots of money and get no results at all.

So, if we wish to gauge how well an account performed, we must look at how well it did relative to the range of reasonable expected levels, not just compared to the maximum. Looking at the production function above, the actual performance might appear to be quite good. But when we consider the production function representing the ‘deficient frontier’ to the graph, we see that the actual performance wasn’t as remarkable as it first appeared:

By neglecting to disclose the Deficient Frontier, the performance of an account can be made to look better than it actually is.

3. The ‘efficient frontier’ is not your target.

After you have identified the ‘maximum conversions’ production function, the next thing to realize is that this line is not your target. EF claims that any bid set that moves you closer to the production function improves the performance of the account, but in my blog post ‘Transgressing the Boundaries‘ I showed that there is actually only one point on this line (which I call the “target point”) for which you should aim, based on budget limits or a CPA goal. To aim for just any point on the production function besides the target point will miss your goal.

4. The production function can move / be moved over time.

Though it is possible to improve the performance of an account by moving closer to the ‘maximum conversions’ production function, it is often easier to simply move the production function to higher values across the board. (Again, please read the section of the Wikipedia article on shifting a production function, if necessary.) Unfortunately, doing so generally requires labor-intensive, good old-fashioned account management (A/B testing, adding negative terms, pruning underperforming keywords, and such) and typically can’t be done by any currently automated means.

5. Using portfolio theory lets ‘deadwood’ float.

Proper account management should move the production function to higher values over time, but using portfolio theory to do this can undermine your efforts. Let’s say that we have an account containing thousands of keywords that overall are hitting their combined performance targets. The spending rate is within budget, the CPA is acceptable, and so forth. (Of course, some individual words might be above the target and some below the target, but overall their performance is acceptable.)

But let’s also say that just one of those words is terrible. Everyone agrees its tangential to the company’s business. It gets clicks and incurs cost, but never gets a conversion. Its presence might be penalizing the adgroup or account’s Quality Score. The rational thing to do is to delete this keyword from the account. But as long as the account’s combined performance is acceptable, then portfolio theory says that it is OK to leave it in. That is, even though the account’s performance could be improved slightly by removing the word, using portfolio theory will result in the word being left in, racking up ad spending for no good reason.

6. “Wall Street technology” almost destroyed the economy.

Speaking of spending for no good reason, despite the ongoing financial crisis and the unprecedented 1000-point intraday drop in the Dow Jones a few weeks ago, Efficient Frontier remains bafflingly proud of the fact that their technology is just like the high-risk methods that run Wall Street.

In their recent whitepaper ‘The Tale Behind the Tail‘ (registration required to obtain copy), EF says, ”Tail terms can be thought of as microcap stocks”. I won’t rehash my blog post, ‘Keywords Are Not Stocks‘, debunking this idea, but regardless of how you think of keywords, EF’s thinking is profoundly tone-deaf given the current economic situation.

7. Even Warren Buffet dislikes portfolio theory.

In case you still might be enamored by ‘Wall Street technology’, realize that Warren Buffet, the greatest investor of all time, is not: “Modern Portfolio Theory tells you how to be average.” he was quoted in The Warren Buffet Way as saying, “But I think almost anybody can figure out how to do average in the fifth grade.”

8. It’s a bad way to bid.

According to Efficient Frontier’s descriptions of their own technology, they guess ”billions of possible bids”, then select the set of bids with the best estimated performance. If a magician has you draw a card from a deck and then tells you that he can guess which card you chose using only a billion guesses, that would not be a very impressive trick. Marin Software’s bid management technology is able to directly calculate a bid from performance data without making ‘billions of guesses’. Google’s chief economist, Hal Varian, showed how to determine bids in his YouTube video ‘Google AdWords Bidding Tutorial‘ without making billions of guesses too. Any system that requires making billions of guesses clearly doesn’t know how to bid optimally.

But 2009′s competition, held in Pasadena, California, for the first time dealt with ad auctions of the sort which search marketers deal with on a daily basis. One of Prof. Wellman’s then graduate students, Patrick Jordan (now a researcher at Yahoo! Labs), devised a simplified form of Google’s ad auction, the core of which should be very familiar to anyone who has ever managed an AdWords account: Competing retailers bid on different keywords (in this case related to TV/audio equipment). Simulated users generate queries and then consider clicking on ads based on the order the ads appear in the results. If a user clicks on an ad, the search engine charges the advertiser, and the user might then purchase a product at the advertiser’s website, bringing revenue to that player.

Each simulated day, each advertiser receives a keyword-level performance report similar to what actual AdWords account managers receive: impressions, clicks, cost, average position, etc, as well as a keyword-level report of the number of sales that resulted and the revenue generated. As in the real world, there is a delay between the end of the previous day and when the data is reported for that day. The advertiser must use this information to set bids, select between more-targeted and more-generic ad creatives and even choose keyword- or account-level spending limits. All of these decisions must be done entirely by the computer code that each team writes, with absolutely no human intervention allowed during any given game. Each game continues for 60 simulated days, with the advertiser who accrues the most net profit winning. Oh, and to keep the game moving, each agent only has 10 seconds to make its decisions for the day. Dozens of games are played, in a semi-finals round and a finals round, to determine the overall winner of the tournament.

Last year’s TAC Ad Auctions game attracted entrants from universities around the world (a list of participants is at SICS’s website). We’re proud to say that third place was won by Professor Greenwald’s team at Brown University, which was assisted in the design of their agent, Schlemazl, by representatives from The Search Agency. Second place was taken by AstonTAC, developed by Aston University in Birmingham (England) and the winner was TacTex, designed by Pardoe, Chakraborty and Stone from the University of Texas at Austin. But only about a 5% difference in performance separated these agents from each other.

The results of the ad auction tournament are described in greater detail in Dr. Jordan’s Ph.D. thesis, but noteworthy are the additional games he ran after the competition was over using only the top three agents. He found that the final placement of these agents depends strongly on the reserve bids set by the search engine. In the actual competition, the search engine required only a very low bid in order for an advertiser to be allowed to participate in a given auction. However, this does not maximize the publisher’s revenue. As the search engine raised the minimum bid requirement, it made more money, and Aston’s agent became the most-favored equilibrium solution. As the minimum bid requirement was raised to a level slightly past the search engine’s optimal value, Brown’s agent became the most-favored.

Earlier this spring, I interviewed David Pardoe, who has written a paper describing his champion ad auction agent in detail and who has also twice won the supply chain management competition. It is fair to say that his program was aided greatly by a quirk of the game: To promote competitiveness, in the 2009 competition the agents were told about the average positions of the other advertisers. Dr. Pardoe was ingeniously able to use this information to determine the order which his competitors reached their budget limits and with that, gain a significant advantage over them.

When I asked how much overlap there was between the agent he built for the supply chain management tournaments and the one for the ad auction competition, he revealed that there was very little. He attributes his victory in the ad auctions competition to his experience from the supply chain management competition and to many rounds of developing the new agent. “Knowing how to experiment,” he said, “is more important than having a really cool algorithm.”

The 2010 competition will be held Monday and Tuesday, June 7-8 at the 11th ACM Conference on Electronic Commerce in Cambridge, MA, but with some rules changes. The average positions told to each agent will be based on a sample of the auctions, rather than all of the auctions. And the reserve bids will be raised, to make them closer to what an actual search engine might use. “We modified the precision of the average position estimate in the reports,” Dr. Jordan, the Game Master for the 2010 competition, said, “to more actually reflect the degree of uncertainty advertisers face in actual sponsored search auctions” and “we increased reserve scores to more actually portray the preferences of the publisher.”

Will the rule changes work against TacTex? Dr. Jordan says, “These [rule] changes are not meant to increase competitiveness in the colloquial sense: making it more likely that any given participant can win the tournament. Rather, our goal is to incentivize participants to develop competent strategies in a very complex, challenging environment, so that we can use insights from these strategies to understand behavior in real markets.”

Brown University team member Eric Sodomka put it more succinctly: “TacTex is definitely the favorite.” Though he’s currently focusing on finishing his Ph.D. dissertation, Dr. Pardoe put an end to the speculation that he might skip this year’s tournament. “I’ll be there”, he said.

So, please wish everyone luck and we’ll keep you posted after the competition as to how they do.

A follow-up to this article is available: ‘Congratulations to All in the Trading Agent Competition‘

Part of the mystery is because an ad doesn’t have just one Quality Score, there are actually multiple values Google calculates. One version specified before an ad auction occurs determines whether your ad is eligible for the auction at all. Slight variants are used for figuring an ad’s first-page bid and whether the ad can appear in the ‘promoted positions’ just above the organic results. A different Quality Score is calculated within an individual ad auction to determine your Cost-per-Click (CPC) based on the next-lowest-placed competitor’s bid. Another version is used only on the Content Network.

A second part of the mystery behind Quality Score is that the number that Google reports as ‘Quality Score’ is probably not the number that is actually used, for example, in a given auction to determine your CPC. Most likely, the in-auction QS is based largely on your ad’s historical clickthrough rate (CTR), if the ad has enough recent data, and is therefore stored as a number between 0 and 1. Quality Score is displayed as a number between 1 and 10, but there is probably not a linear transformation of the actual Quality Score to the reported Quality Score, since most QSs (about 40-50% of the ads in a typical account) are reported as ’7′ and there are few, if any words in an average account that have CTRs of 70%.

A third reason QS is so difficult to figure out is due to what I call Google’s “Quality Score Memory Hole“. At any time, Google will only report an ad’s current Quality Score, even if you run a report with the data broken down by the ‘Daily’ option.

To see this in action, create a report for a given keyword, with statistics reported by day. (If you know how to do this, you can skip to the next paragraph. If not, here are instructions: Go into the Google AdWords interface and select the ‘Reporting’ tab’s ‘Reports’ option. Create a New Report for ‘Placement / Keyword Performance’. For ‘View’, choose ‘Daily’ and for Data Range select ‘Last fourteen days’, or manually enter the past several weeks or so. You can choose ‘Manually select from a list’ to narrow the report down to one campaign and/or adgroup and the ‘Filter Your Results’ option to specify a single keyword and matchtype (preferably, ‘exact’). Then click the ‘Create Report’ button at the bottom of the screen.)

On April 27, I created a report by day for April for one keyword. The Search Agency previously recorded, every day at the same time each day, that day’s current Quality Score. These are shown below.

So, on April 1, Google reported that the Quality Score was 4. On April 23, they reported that it was 6. On April 26, they reported that it was 7. But, when on April 27 I requested the Quality Scores, by day, for the month of April, they reported that Quality Score each day was 7.

That is, on any given day, Google will only tell you the current Quality Score. If you wish to know how Quality Scores can change over time, you need to record the QS for every keyword which you are interested in every single day, then keep your own record. Let’s look at some of the findings from doing this.

Lesson 1: Quality Scores Can Be Very Unstable

Notice that for the keyword shown above, which has gotten over 400 clicks for a cost over $1000 in April thusfar, the Quality Score changed 5 times in 26 days, or more than once per week. In other words, the Quality Score you see might depend on which day you check. An analysis of hundreds of thousands of keywords across accounts in many different industries shows that upwards of 20% of clicks and 20% of the cost in an account might be in ‘unstable’ ads whose Quality Score changes more than once per week or by 3 or more QS points in a month. On average, about 15% of clicks and cost are in these ‘unstable’ ads.

In his new book Advanced Google AdWords, Brad Geddes recommends that “if you have a low quality score, you should optimize your quality score instead of raising bids.” (p. 202) But note that for this ad, The Search Agency did not try to “optimize” Quality Score to get it from 4 to 7. We didn’t “focus” on maximizing its CTR. In fact, we didn’t do anything at all. These changes in Quality Score are purely the natural fluctuations for this ad. You would not take financial advice from a teenager. You would not take career advice from a teenager. So, why take account management advice from a Quality Score that acts like a teenager?

The take-home lesson is: if you see a low QS, don’t panic. Do not immediately revise ad text nor make other drastic changes. Just check again the following day or the following week to see if QS remains low. It is only a problem when an ad has a chronically low QS and that ad is underperforming. (In fact, the ad above generated more profit per day and at a higher ROI when its QS was 4 than when it was 7.)

Lesson 2: A Higher Quality Score Does Not Necessarily Mean a Lower Cost-per-Click

One of the most persistent myths in search marketing is that CPC drops as Quality Score increases. Google even says explicitly on their AdWords help pages: “The higher a keyword’s Quality Score, the lower its cost-per-clicks (CPCs) and the better its ad position.” However, as we pointed out earlier, this is only true within a single ad auction and, furthermore, Google does not tell you your actual Quality Score.

Looking at the CPC for the keyword mentioned above by day for March and (most of) April, you can see that the average CPC when the Quality Score was 4 was just under $5 (actually, $4.81). When the Quality Score was 7, the average CPC was also $4.81, even though the bids and average position were stable during this time. When the Quality Score was 6, the average CPC was lower than when the QS was either 7 or 4.

In fact, examining many high-performing keywords, one finds that there is very little correlation between the CPC and the (reported) Quality Score. In some case, CPCs fall as QS rises. In some cases, they rise as QS rises. In most cases, like the one shown above, there is no obvious relation.

Lesson 3: Google’s Quality Score Algorithm Changes Periodically

Google’s Quality Score algorithm is believed to undergo periodic updates and a significant one might have happened on April 1, 2010 (and that’s no belated April Fool’s joke). A graph for the account of the keyword described above shows that the number of ads whose Quality Score changed between any two days averaged about 100 prior to April 1, 2010, then spiked to over 400 on April 1. The number of ads whose QS changed between any two days has been more volatile since that date than before it.

A similar pattern is seen in many accounts, indicating that this change is not specific to this one. This might have been a one-time recalibration done on April 1, or might represent the implementation of an ongoing change in the way Quality Score is calculated, but there’s no way to tell for sure at this point.

Lesson 4: Google Might be Manipulating the Numbers Even Further

Google claims that the only data which is used to determine Quality Score comes from searches where the search query is identical to the keyword’s text. This implies that, if an account bids on ‘leather shoes’, exact match and ‘leather shoes’, phrase match, then both keywords should have the same Quality Score.

However, in a comment on Search Engine Land and in communications via Twitter, Brad Geddes claimed that that this might not always the case. In particular, he’s found examples of high click-volume keywords where the Quality Scores differ across matchtypes. “Same ad group, one ad, same landing page,” he says, “[but] broad and exact match have widely different quality scores (biggest one I saw was 10 for exact and 4 for broad, another was 7 for broad and 2 for exact). So – there’s something else going on there with QS and match types than Google is letting on.”

Though some people can be led into false senses of complacency or panic due to high or low reported Quality Scores, before using QS to prompt any changes in how you manage your account, remember that many Quality Scores can act like teenagers. So, be sure to look not only at their current values, but also how they change over time.

But since the Cost-per-Acquisition (CPA) of an ad is measured in ‘dollars per conversion’ and the Conversion Rate (CR) is measured in ‘conversions per click’, we can multiply these factors together to determine our target CPC:

That is, if we should be willing to pay $10 per conversion and 20% of clicks result in conversions, then each click is worth $2.00 to us. We can see from the equation above that the only ways to reduce the target CPC to $0 are to have a target CPA of $0 (at which point an advertiser shouldn’t be willing to pay for even one click, which would be very bad for business) or to have a conversion rate of 0% (which would also be very bad for business), or both (which would be very, very bad for business).

Note in the equation above that Quality Score plays no role whatsoever in determining your target CPC. If there is an increase in one’s Quality Score, an account manager should not rejoice at the amount that the Cost-per-Click diminishes, but rather continue to pursue precisely the same target CPC, enjoying whatever increase is seen in the number of clicks.

One interesting thing to observe about the equation above is that any extent to which conversion rate optimization efforts are successful should result in higher target CPCs.  For example, if by designing new landing pages we manage to raise our conversion rate from 20% to 50%, then we should be willing to increase our target CPC proportionally from $2.00 to $5.00. That is, one measure of the success of landing page optimization efforts is the degree to which they increase the observed Cost-per-Click.

Looking at the equation above we can clearly see that AdWords account managers who wish to both increase their conversion rate and simultaneously decrease their average Cost-per-Click are pursuing counterposed goals.

A second interesting observation about the equation is that the judicious addition of negative terms to a keyword (in order to eliminate some low-value impressions) should concomitantly result in an increase in the ad’s conversion rate and, thus, increase the advertiser’s target CPC for that keyword.

Finally, it should be noted that the efforts by the search engines themselves to detect click fraud or to improve the relevance of a given ad to a given search query should also naturally result in increasing CPCs to whatever degree these changes enhance conversion rates. So, it should be expected that CPCs would have the tendency to inflate over the timescale of months or years as their algorithms improve.

For those who are unconvinced by theoretical arguments that successful online marketing efforts should tend to increase CPCs, we can also present hard-earned empirical evidence simply by looking at CPAs for various accounts.

For the month of March 2010, the graph below depicts the CPA for all of the non-brand-related keywords for a dating website which has a CPA target of about $100. (Naturally, brand-related terms tend to generate large numbers of conversions at dirt-cheap CPCs, so these terms have been eliminated from consideration for this analysis.) Collectively, these words cost just over $16,000 in March.

We can see from the graph that words whose CPC averaged $2.00 to $2.50 had a CPA less than $70 per conversion while words with an average CPC in the range of $0.00 to $0.50 collectively have a CPA of over $130 per conversion. It might be worthwhile, if the words with CPCs above $2 are limited by the account’s budget, for the manager to consider simply shutting off some words with CPCs below $0.50 in order to free up additional money for the high-CPC ads.

For ROI-based accounts, natural metrics to consider are the Profit per Impression and, of course, the Return on Investment. For one campaign in an ROI account that sells very expensive products, we again filtered out brand-related keywords. The remaining words in that campaign spent over $180,000 in March and the target ROI for this campaign was about 40%.

Again, the highest CPC keywords prove to be the most valuable, with ads whose CPCs averaged $10.00 to $12.00 bringing in an average profit of about $0.25 per impression and ads whose CPCs averaged $0.00 to $2.00 bringing in only about $0.01 in profit per impression. Not only did the lower-CPC keywords average less profit per impression, but they also had lower ROIs than higher-CPC words.

The point I’m making is that every ad occupies a small piece of visual real estate and real estate has value. Good real estate has a high value for a very simple reason – it’s worth it. That’s why you’ll find more Starbucks in downtown Manhattan than, say, the northern coast of Greenland. Low-CPC keywords tend to be low-CPC keywords precisely because they’re not worth much. (To be fair, though, in the course of this analysis I did find some quirky 3- and 4-term keywords that generated a lot of business at very low CPCs. It’s likely only a matter of time, though, before competitors discover them too.)

The take-home here is: Don’t put your ads in Greenland.

So, if your marketing efforts are going well, with improving conversion rates and good negative terms in place and someone asks you how your account is performing, be sure to hold your head high and say “My CPCs keep going up!”