Dirty Little Secrets of Portfolio Theory

Posted on Monday, June 7th, 2010 by Print This Post Print This Post

Categories - Featured, SEM

Several search marketing firms (like Efficient Frontier) claim to have based their account management technology on Modern Portfolio Theory, the method for allocating capital among classes of assets that won Harry Markowitz the Nobel Prize in Economics. But, there are problems with portfolio theory that could easily get online marketers into trouble if they take companies like EF at their word. Don't misunderstand: The Search Agency uses a portfolio-based technique which is not derived from Modern Portfolio Theory (MPT) for the analysis of our own clients' accounts. It's partially from developing this technique that we were able to discover some of MPT's dirty little secrets: 1. It's actually just a production function. Efficient Frontier shows diagrams on their website of 'Conversions vs. Ad Cost' and claims that the relationship between these quantities is called an 'efficient frontier'. These diagrams are similar to the one shown below: For different possible levels of spending, this chart shows how many conversions one can expect. There is a maximum amount of spending and conversions (marked by a terminal point) even if all the words in the account were pushed into first position. A green 'X' marks the actual performance of the account over the past week, for example. EF calls this relationship an 'efficient frontier'. Heck, they even named their company after it. But, unfortunately, it's not an efficient frontier. It's just a production function. (Check out the graphs in Wikipedia's entry for 'production function', to see what I mean. Plotted on the vertical axis is 'units of output per time', like Conversions. The horizontal axis is 'units of input per time', like Ad Spending.) To graph an actual efficient frontier we must plot the average return from a basket of assets versus the covariance of the value of that basket. The purpose of an actual efficient frontier is to determine the optimal allocation of assets - for instance, what percentage of stock (or bond) A and stock (or bond) B we should own in order to maximize our expected return for a given level of risk. I realize 'production function' doesn't sound as sexy as "efficient frontier", but it's the correct term. So either Efficient Frontier doesn't know the difference between a production function and an efficient frontier, or they are simply hoping that you don't. 2. There's also a "Deficient Frontier" Efficient Frontier's version of portfolio theory is concerned with identifying the maximum number of conversions an account can expect for various levels of ad spending. But to judge how effective your account management is, you also need to consider the minimum number of conversions you can expect. In the 1985 comedy Brewster's Millions, Richard Pryor inherits a fortune, but to get the money he must spend one-tenth as much in 30 days and have absolutely nothing to show for it. Of course, the minimum number of conversions a given amount of ad spending could theoretically generate is always zero. But as Richard Pryor found out, it is extremely difficult to spend lots of money and get no results at all. So, if we wish to gauge how well an account performed, we must look at how well it did relative to the range of reasonable expected levels, not just compared to the maximum. Looking at the production function above, the actual performance might appear to be quite good. But when we consider the production function representing the 'deficient frontier' to the graph, we see that the actual performance wasn't as remarkable as it first appeared: By neglecting to disclose the Deficient Frontier, the performance of an account can be made to look better than it actually is. 3. The 'efficient frontier' is not your target. After you have identified the 'maximum conversions' production function, the next thing to realize is that this line is not your target. EF claims that any bid set that moves you closer to the production function improves the performance of the account, but in my blog post 'Transgressing the Boundaries' I showed that there is actually only one point on this line (which I call the "target point") for which you should aim, based on budget limits or a CPA goal. To aim for just any point on the production function besides the target point will miss your goal. 4. The production function can move / be moved over time. Though it is possible to improve the performance of an account by moving closer to the 'maximum conversions' production function, it is often easier to simply move the production function to higher values across the board. (Again, please read the section of the Wikipedia article on shifting a production function, if necessary.) Unfortunately, doing so generally requires labor-intensive, good old-fashioned account management (A/B testing, adding negative terms, pruning underperforming keywords, and such) and typically can't be done by any currently automated means. 5. Using portfolio theory lets 'deadwood' float. Proper account management should move the production function to higher values over time, but using portfolio theory to do this can undermine your efforts. Let's say that we have an account containing thousands of keywords that overall are hitting their combined performance targets. The spending rate is within budget, the CPA is acceptable, and so forth. (Of course, some individual words might be above the target and some below the target, but overall their performance is acceptable.) But let's also say that just one of those words is terrible. Everyone agrees its tangential to the company's business. It gets clicks and incurs cost, but never gets a conversion. Its presence might be penalizing the adgroup or account's Quality Score. The rational thing to do is to delete this keyword from the account. But as long as the account's combined performance is acceptable, then portfolio theory says that it is OK to leave it in. That is, even though the account's performance could be improved slightly by removing the word, using portfolio theory will result in the word being left in, racking up ad spending for no good reason. 6. "Wall Street technology" almost destroyed the economy. Speaking of spending for no good reason, despite the ongoing financial crisis and the unprecedented 1000-point intraday drop in the Dow Jones a few weeks ago, Efficient Frontier remains bafflingly proud of the fact that their technology is just like the high-risk methods that run Wall Street. In their recent whitepaper 'The Tale Behind the Tail' (registration required to obtain copy), EF says, "Tail terms can be thought of as microcap stocks". I won't rehash my blog post, 'Keywords Are Not Stocks', debunking this idea, but regardless of how you think of keywords, EF's thinking is profoundly tone-deaf given the current economic situation. 7. Even Warren Buffet dislikes portfolio theory. In case you still might be enamored by 'Wall Street technology', realize that Warren Buffet, the greatest investor of all time, is not: "Modern Portfolio Theory tells you how to be average." he was quoted in The Warren Buffet Way as saying, "But I think almost anybody can figure out how to do average in the fifth grade." 8. It's a bad way to bid. According to Efficient Frontier's descriptions of their own technology, they guess "billions of possible bids", then select the set of bids with the best estimated performance. If a magician has you draw a card from a deck and then tells you that he can guess which card you chose using only a billion guesses, that would not be a very impressive trick. Marin Software's bid management technology is able to directly calculate a bid from performance data without making 'billions of guesses'. Google's chief economist, Hal Varian, showed how to determine bids in his YouTube video 'Google AdWords Bidding Tutorial' without making billions of guesses too. Any system that requires making billions of guesses clearly doesn't know how to bid optimally.

Tags | ,

8 Responses to “Dirty Little Secrets of Portfolio Theory”

  1. Pretty disappointing post filled with inaccuracies and misinformation, not to mention a conclusion that flies in the face of reality. The article is long on copy and pasted images, and short on intellectually honest analysis.
    The overarching concept of the efficient frontier was first introduced by Markowitz in 1952 in a Journal of Finance article entitled “Portfolio Selection.” A few years later, in 1956, Markowitz published an essay that showed how to calculate the optimal portfolio using a quadratic programming algorithm, and his later works further expanded on the topic. The high level idea of the efficient frontier is to calculate optimal portfolio outcomes. This is precisely what Efficient Frontier does through patented algorithms. There is no ‘guessing’ as you have falsely indicated. Efficient Frontier literally calculates and predicts outcomes for every possible combination of bids to identify the optimal yield scenario at any given point in time. This is in stark contrast to rules-based and manual bid management (which I’m sure you are familiar with) which attempts to have people decipher what bids will yield what outcomes in an opaque market (= impossible) or they make no attempt to guess at all. Efficient Frontier’s models are 90-99% accurate, time tested over 9 years on $900M/yr in SEM spend.

    “There is also a Deficient Frontier”. Most large SEM spenders wish to maximize performance and find the point of optimal return. That shouldn’t be a surprise and I believe most want to judge performance not versus the lower bound, but the upper bound of what’s possible. Isn’t that why we optimize – to get to the optimal point? That is precisely the value that algorithms and PhD level math can bring in the complex environment of today’s SEM (and biddable display) landscape. Is your point that advertisers shouldn’t aspire to maximum performance?

    “The efficient Frontier is not your target”. This is overly simplistic and incorrect. If you download Efficient Frontier’s white paper here (http://www.efrontier.com/research/whitepapers/algorithms-and-optimization), you will see on page 6 an explanation of how there are multiple points on the efficient frontier curve that may be optimal for a particular advertiser. Most sophisticated, large scale SEM marketers don’t look at a simple CPA, but track revenue, profit, margin and other metrics. Depending on the client’s goals, different operating points offer different trade-offs. This will depend on the client’s goals and priorities which can change often and be quite dynamic. Advertisers often look at trade-offs between revenue and profit, the maximization points of which are different points on the curve.

    “The production function can move / be moved over time.” Of course. You are stating the obvious. Any SEM marketer worth his/her salt recognizes the importance of both leading edge technology and the role humans play across all points in the Search value-chain (ie: copy, keywords, match-types, landing pages etc etc). However, history has shown that many times the biggest source of incremental value and moving the optimal performance point is better, smarter bidding which can drive 10-50% or more improvements. Over time, it’s also important to push the curve out via all tactics in the SEM toolbox. Saying there is value in optimizing copy isn’t a rational argument against having advanced algorithms versus and army of 24 year old bidders. History has proven that automation and math can drive superior value versus humans (who can’t scale) for certain tasks, chief among them bidding in opaque auction markets where modeling matters.

    “Using portfolio theory lets ‘deadwood’ float.” Your contention that Portfolio Theory precludes a smart Search Marketer from pruning and removing keywords is false and fairly ridiculous. Once again, a red herring argument. Any smart marketer would make use of the best math and technologies on the market, and utilize manual oversight/strategies to optimize performance.

    “Wall Street technology” almost destroyed the economy.” Yet another inane argument that proposes “throw the baby out with the bathwater” logic. So, I assume you would propose that we all stop driving cars because of the issues with the Toyota Prius? Math and algorithms are highly useful and present in our society. They are used for shipping logistics, credit card processing, inventory and retail shelving, call center queues and on and on….Not to mention that many hedge funds have successfully beaten the market using algorithmic trading, chief among them DE Shaw. Oh and those Wikipedia search you like to do…..algorithms!

    “Even Warren Buffet dislikes portfolio theory.” Do I need to respond to this one? Warren Buffet is a self proclaimed behavioral investor so no surprise here. He is against using complex formulas to valuate or trade stocks and prefers to base his investments on subjective judgments and his own cognitive formula. EF’s technology is based on applying portfolio theory and predictive models to the paid search market. I’m pretty sure that Mr. Buffet wasn’t condemning every possible application of portfolio concepts to other endeavors.

    “It’s a bad way to bid.” Again, you have falsely stated that EF’s technology “guesses”. The whole point of using predictive modeling is so you don’t need to have people guessing and using trial and error to bid in an opaque market. The goal of modeling is to predict accurately, which leads to more profitable decisions, and this is in fact of what all of EF’s patent are around: accurate modeling of future outcomes in opaque search markets. Decoding marketplaces. Their patent applications on the USPO.gov site are quite interesting. I like to base conclusions on facts and data. I can only conclude that their methodology is a great way to bid based on the clients they serve, spend they manage and the various case studies I’ve seen which consistently point to massive performance lift versus manual and simple rule-based tools (which don’t attempt to model or predict anything). My guess is that technology will play an even broader role as biddable display across the exchanges and social networks develops. No person or collection of people can model and valuate billions of impressions in real time. I always find is comical that those without any technology are always the ones that seek build irrational anti-technology arguments. The market votes, and it has.

    • Bradd Libby says:

      Clearly my post touched a nerve with someone who claims to be in search of ‘the truth’ but provides only a screenname and a fake email address.

      EF is really in an unenviable position here, since their company’s name alone makes them naturally disinclined to openly acknowledge the deficiencies of their methodology. (So, it’s not surprising that EF has not yet directly responded to my original post. Thus, I can only respond to their anonymous defender.)

      First, let’s clear up some minor terminology. You claim that EF does not “guess” bids, yet by your own admission you say that EF “literally calculates … every possible combination of bids”. A well-designed system would calculate only one set of bids, the optimal one. Whether you acknowledge that a “possible combination of bids” is, in fact, a “guess” does not matter. The fact remains that EF is using a method called ‘guess-and-check’, whether you personally are willing to admit that or not.

      Terminology that actually does play a role, though, is EF’s misappropriation of the term “efficient frontier”.
      You say: “The high level idea of the efficient frontier is to calculate optimal portfolio outcomes”. Unfortunately, ‘Conversions vs. Ad Spend’ is not mathematically equivalent to graphing a weighted-average return on a basket of assets versus the covariance of that return, which is the methodology that won Markowitz the Nobel Prize. If their methodology were identical to the Markowitz frontier (which it is not), then EF would be justified in calling their company by that name. Instead, they should be honest and simply call their company ‘Pareto Optimization, Incorporated’ or ‘Production Functionz R Us’ or some other more forthright name.

      Regarding the point that there is a “deficient frontier” you say: “most want to judge performance not versus the lower bound, but the upper bound of what’s possible”. But actually, yes, an intelligent person should wish to consider the full range of bounds. If you get an 80% on a test, it matters whether you got it on a chemistry exam where the lowest score was 40 and the highest was 85, or whether you got it on an IQ test where the lowest reliably measurable score is about 60 and the highest about 130. EF should acknowledge the range of possible performance.

      You agree with my point that the production function can move / be moved over time. It is often a preferable means to enhance performance compared to devoting additional attention to bidding.

      Rearding the point that the ‘efficient frontier’ is not your target, you claim that “there are multiple points … that may be optimal” but that “depending on the client’s goals, different operating points offer different trade-offs”, which is in line with my claim that, for a given set of advertiser constraints, there is one target point – It is logically impossible to achieve multiple target points, sine they each have different Cost and Conversion (or Profit) values. EF even shows this on their own homepage at: http://www.efrontier.com/innovation/predictive-modeling
      where they identify a single point of maximum ROI (and many, many guesses that are not the target point).

      Regarding the point that using portfolio theory lets ‘deadwood’ float, you say that “Your contention that Portfolio Theory precludes a smart Search Marketer from pruning and removing keywords is false and fairly ridiculous.” You are simply wrong on this point. Consider two assets, the first of which gets an average return of 15% and has a variance of 10%. The second gets an average return of 25% and has a variance of 5%. The first asset is dominated by the second: it gets a lower average return and has a greater ‘risk’ (i.e., variance). Markowitz’s theory claims that there is typically a risk-minimizing allocation of assets that holds a nonnegative stake in the first asset. (The non-intuitiveness of this claim, I think, is the core of the reason he was awarded the Nobel.) You, however, are claiming that a “smart Search Marketer” would remove the underperforming word. So, either EF is using a technique that behaves as Markowitz’s theory (in which case, they would not eliminate the lower-performing keyword) or their technique would not behave as Markowitz’s theory (in which case they should stop lying to people by calling themselves ‘Efficient Frontier’). Which is it?

      My arguments are not “anti-technology”, just opposed to the universal application of one methodology to all bidding situations. You resort to a rhetorical trick called the ‘false dichotomy’ by saying that EF “is in stark contrast to rules-based and manual bid management” and that algorithmic bidding is superior to an “army of 24 year old bidders”, as if the only choice is between EF’s approach and manual- (or rules-)based bidding. In fact, there are excellent alternatives to EF’s approach. Marin Software makes a tool which does not resort to your guess-and-check approach. ClickEquations has an option called ‘Maximize ROI’ which works very much like Marin’s tool, and it also supports rules-based, position-based, etc bidding.

      Part of my philosophy as a Ph.D. engineer is: the right tool for the right job. And the simplest tool is usually the best. Unfortunately, EF’s approach, with their ‘Wall Street algorithms’, is rarely the simplest one, the best one, or even a good one.

      I made the claim that it is profoundly bad marketing, in the current economic environment, for EF to liken keywords to stocks. You respond by defending the general concept of ‘math’ and ‘algorithms’ and how wonderful they are. However, Wikipedia’s algorithms are not the ones that have bled trillions from the global economy over the past year and half. EF claims that theirs, in fact, are. Therefore, I stand by my claim.

      Regarding Buffet you say: “I’m pretty sure that Mr. Buffet wasn’t condemning every possible application of portfolio concepts to other endeavors.” Frankly, the claim that your boat doesn’t sink as long as no one puts it in the water doesn’t impress me much. Buffet claims that Portfolio Theory is a great way to get average results, and everything I have seen that compares EF’s marketing platform to other automated platforms reinforces that conclusion: EF has a middling technology combined with the sort of suspender-snapping hubris that the Masters of the Universe used to have before reality hit them in the face.

      In closing, you say “I can only conclude that [EF’s] methodology is a great way to bid based on the clients they serve, spend they manage and the various case studies I’ve seen which consistently point to massive performance lift versus manual and simple rule-based tools (which don’t attempt to model or predict anything).” And also: “Efficient Frontier’s models are 90-99% accurate, time tested over 9 years on $900M/yr in SEM spend.”

      One-fifth of the planet lives under a communist government, but that doesn’t mean I’m going to be singing ‘L’Internationale’ any time soon. Simply because a large number of people have used something, doesn’t mean that it is good or that there’s no better way. By your logic, Marin Software (which claims that their tool manages $1B in spend), must be about 10% better than EF, which is right in line with your estimate that EF’s models are ~90% accurate. I’d wager ClickEquations’ “Maximize ROI” would give EF a run for their money, too.

      The Search Agency is algorithmically agnostic. When a Marin-like approach works best, we’d suggest that. When EF-like approach works best (which it rarely, if ever, does), we’d suggest that. When rolling up your sleeves and digging into the data is needed, we’d suggest that too. But Efficient Frontier is ideologically wedded to the production-function-based technique they have mislabeled the ‘efficient frontier’. They are not willing to admit its warts simply because they can’t.

      “I always find is comical that those without any technology are always the ones that seek build irrational anti-technology arguments.” I have a Ph.D. in engineering, so I can assure you I am no technophobe. Any of EF’s current clients who wish to contact TSA to see a demonstration of our online marketing platform, we’ll gladly show it to you. Several of EF’s high-profile (former) clients have already had us do so.

  2. Dr Siddharth Shah says:

    I have similar sentiments as stated in TruthAgent’s response, however, there are some additional points I would like to make. The Search Agency has made several mathematically incorrect assertions that do add up.

    It’s actually just a production function.” is absolutely incorrect. The assertion is that because the Efficient Frontier in SEM “looks” like a production function in economics that they are the same. Perhaps the writer is inexperienced with modeling and solving optimization problems but similar “looking” solutions come up in mathematics all the time. It does not mean that they are solving the same problem. For instance: Take the capacitor charging equation which looks like this: http://hyperphysics.phy-astr.gsu.edu/hbase/electric/capchg.html

    The Efficient Frontier and the charging equation graph look remarkably similar. Dont they ? So what do these two have in common ? Absolutely nothing.

    What The Search Agency is graphically showing as the efficient frontier in finance can be made to “look” a lot like the efficient frontier in SEM (just add an assumption that there exists a risk free asset with non zero return and a few other assumptions) but that is not the point. The analog between finance and SEM does not come from data representation. It comes from the problem formulation:

    (1) Financial portfolio theory’s optimization problem: Maximize return by choosing investment levels in a basket of possible INVESTMENT VEHICLE subject to the constraint that my risk (covariance) is at or below a selected threshold.

    (2) SEM portfolio optimization problem: Maximize return by choosing investment levels in a basket of possible KEYWORD choices subject to the constraint that my BUDGET (covariance) is at or below a selected threshold.

    (3) Production function optimization: Maximize production subject to the constraint that my inputs are constraint bound. Further (I am quoting the wikipedia reference here), “the relationship of output to inputs is non monetary, that is, a production function relates physical inputs to physical outputs and prices and costs are not considered”.

    It should be evident now that (1) and (2) are very similar. (3) is only similar to (2) in form, not function.

    Also note that between 1 and 2 , all we changed was INVESTMENT VEHICLE to KEYWORD and RISK to BUDGET. So yes they are analogs. While Search Agent critique on Efficient Frontier’s white paper calling a tail term as microcap stocks was impassioned and emotional , it is irrational and irrelevant. Tail terms do behave like microcap stocks. The mathematics bears it out. Further, blaming MPT for the wall street crash is like blaming the Ferrari for a car crash if a monkey drove it. Irrelevant.

    Your calling Efficient Frontier’s optimization a series of “guesses” reminded me of my professor’s advice on using accurate language when describing things mathematical. Efficient Frontier CALCULATES the best optimization choices with 90-99% accuracy for the user defined CONSTRAINT (budget/CPA/efficiency.. could be anything the advertiser wants) after making accurate ESTIMATES of keyword bid/cpc/clicks/performance tradeoffs. Usually 90-99% prediction accuracy is not called a guess.

    Also note that, one can MATHEMATICALLY PROVE that, with ACCURATE keyword models, Portfolio theory GUARANTEES THE BEST PERFORMANCE FOR ANY GOAL SUBJECT TO ANY BUSINESS CONSTRAINT. The best one can do by manual bidding is match portfolio theory bid choices to get the same optimal outcome. However, the chance of that happening are close to zero for all practical business situations. Our head to head tests of the two approaches bear that out.

    There are several other factual inaccuracies and misleading statements that have been addressed quite sufficiently by TruthAgent so I will not address them. In any case, if there are any questions do feel free to contact me.

    • bradd libby says:

      Dr. Shah,

      There seems to be legitimate disagreement as to whether or not EF’s graph of Conversions (or Profit) versus Ad Spend constitutes a production function. You point to Wikipedia’s claim that the inputs to a production function are non-monetary. Obviously in this system, where the inputs are monetary, a graph of input vs. output could be considered a production function. While I respectfully disagree, to be gentlemanly, I am willing to concede this point.

      What I am not willing to concede is my claim that EF’s graph of Conversions (or Profit) versus Ad Spend is not an Efficient Frontier.

      Markowitz provided equations – one for Average Return and one for Risk, which won him the Nobel Prize in Economics. The equation for the Risk is the covariance of the value of the portfolio. When graphed against each other, Return of a portfolio versus the Covariance (or, Risk) of that portfolio produces a curve called the Efficient Frontier.

      Dr. Shah, you say:
      “(2) SEM portfolio optimization problem:
      Maximize return by choosing investment levels
      in a basket of possible KEYWORD choices subject
      to the constraint that my BUDGET (covariance)
      is at or below a selected threshold.”

      You say budget is a constraint. (That is, it is *not* the covariance of the Return, as Markowitz’s theory describes.) For that matter, Ad Spend is not the covariance of the Return either.

      The x-axis on EF’s ‘Efficient Frontier’ graph is simply Ad Spend. In words, EF’s x-axis equation reads: “To find the Ad Spend of a portfolio of keywords, add up the spend of each individual keyword.” That is EF’s calculation of ‘Risk’. This makes no sense. If EF’s ‘Efficient Frontier’ graph was an actual Efficient Frontier, then the x-axis would be the covariance of the y-axis.

      EF’s ‘Efficient Frontier’ x-axis is not the covariance of Conversions (or Profit) and therefore I don’t believe that this an appropriate application of the term “Efficient Frontier”.  I think EF is unjustified in showing Markowitz’s equations for Risk on their ‘SEM Product Tour’ and I think they are unjustified in claiming (http://www.efrontier.com/innovation/portfolio-theory) that their methodology was “developed by a Nobel Prize winning economist”. This is not simply an issue of semantics.

      As a footnote to Dr. Shah and everyone else at EF: I wish you all the best. This is not a personal issue and I hope you don’t take anything I have written here personally. We seem to have a legitimate disagreement in how mathematically related Harry Markowitz’s MPT methodology is to your own. Perhaps this is simply a case of the good engineers and mathematicians at EF getting tripped up by an overzealous marketing department. (If so, it wouldn’t be the first time in history that this has happened.) Even if you are not able to do so publicly, I’m certain that you will be able to acknowledge in your hearts that what I have said rests on a logical foundation.

  3. Frederic says:

    This was a good post..1st visit to this blog. Appreciate you for sharing. I have to subscribe to this blog. I am very fascinated with the World economy. I really believe that the United States budget is quickly declining. I guess we will see. Thanks once again…..

  4. Cerach says:

    Hello, i came to this blog whilst searching for some thing else, but to my joy i discovered this internet site. Maintain doing what you might be performing, its fantastic.

  5. Online 21 is my favourite. Do you think it is better to play online ?

Trackbacks/Pingbacks

  1. Automatic Bid Management: Wer macht es wie

Leave a Reply

Follow Us on Twitter

Authors