NEWS AND INFORMATION - Details
| Articles | Industry Interviews | Conferences | Employment Classifieds |
 
 
PERFORMANCE BENCHMARKS AND SURVIVORSHIP BIAS FOR HEDGE FUNDS AND COMMODITY TRADING ADVISORS
 

By James Park, J.D., Ph.D., CEO of PARADIGM Capital Management, Inc., Stephen J. Brown, Stern School of Business, NYU and William N. Goetzmann, Director of the International Center for Finance at the Yale School of Management.

Hedge fund managers and CTAs derive their return from their skill and ability to process information and take the appropriate action in the global securities markets. Skill-based money managers are the precise opposite of indexed managers. In effect, rather than attempting to minimize deviations from a passive benchmark like the S&P 500, hedge fund managers and CTAs seek to provide positive "tracking error." Even when their performance is related to fluctuations of standard benchmarks, the relationship to indices may be highly non-linear. As such, traditional benchmarks for measuring performance will no longer do.

The important task is to develop appropriate benchmarks for the alternative investment industry. This is particularly relevant to institutions such as pensions, endowments and foundations. Any significant money coming from an institution would inevitably have to be broadly diversified. Broad diversification implies a risk/reward profile similar to the industry benchmark. Moreover, an accurate benchmark or index allows an investor to evaluate whether the investment is appropriate for their portfolio and provides an investor with realistic expectations. In this article we discuss problems associated with answering this question and some recent evidence on aggregate skill-based manager performance.

The last ten years have seen a dramatic increase in the quality and quantity of information available to investors about alternative vehicles such as hedge funds and CTAs. Private services and now web-based information systems increasingly allow investors to track changing net asset values of unregulated funds. While high quality information is now available about individual managers, the very nature of alternative investments makes the aggregate interpretation of this information challenging.

Two of the most important needs for investment benchmarks are portfolio choice and performance evaluation. The decision about whether to include an asset class into a diversified portfolio, for example, requires the use of a benchmark index that reflects the risk, return and correlation that can be expected from a diversified pool of such assets. Should such a portfolio be equal-weighted or value-weighted? Should it be all-inclusive or limited to the "bellwether" funds? Should it be composed of funds only open to new investment or the major funds in the industry? Should it reflect a range of styles or concentrate on the most popular? Should you use current funds only, or include funds that have gone out of business? Ideally, the index should be "investable", diversified, representative and truly reflect after-fee returns on a regular basis, with no survival bias. There is probably no one index of hedge funds that will satisfy all requirements needed for the asset allocation decision, but any choice must address at least some of these issues.

An important obstacle in the way of constructing a hedge fund or CTA benchmark by aggregating data from databases has been survivorship bias in the database of returns. Hedge funds and CTAs are unregulated by the SEC and thus do not need to report publicly performance regularly -- indeed for a long time doing so was construed by the SEC as a solicitation for funds and could cause legal problems for a manager. Unlike regulated vehicles such as mutual funds, investments in hedge funds are made primarily through a limited partnership interest or a managed account. A managed account is an account in the name of the investor with power of attorney given to the money manager with respect to trading. There are no mandatory reporting requirements to any public agency.

Data vendors such as MAR, TASS or Stark started only in the late 80's and 90's. The U.S. Offshore Funds Directory began in 1990 with annual performance figures for 1989. Initially, data vendors may not have been aware of the consequences of creating a database by back-filling or of the importance of the performance of non-surviving traders. To be fair, most vendors do not sell data specifically for the creation of an index, but rather to allow tracking of individual fund performance. However, survivorship is potentially an important issue. We have found that the attrition rate for hedge funds is about 15% per year and that the "half-life" of funds is about 2.5 years. 1

While survivorship is a clear potential problem, a natural question to ask is "Why not use all the information about past returns on funds of interest that you can?" Why maintain performance data for funds that no longer exist? In the past, fund databases have been created by "back-filling" using historical data. In addition, non-surviving performance records were sometimes systematically removed because of the view that clients were only interested in traders that were currently accepting investments. Back-filling and deletion of defunct funds both have serious implications for the construction of benchmarks for managers. If you only collect information from successful funds, and you delete information about funds that disappeared, you only get part of the aggregate performance story -- usually only the good part. Indices formed by using data subject to these conditions may result in aggregate annualized performance in excess of 20% and sometimes 30%! Our research together and separately over the past few years has focused in part on the implication of using data censored by things like survival and back-filling.

As a first step to dealing with these selection biases, we went to the annual volumes of The U.S. Offshore Funds Directory from 1990 to 1996 to study the aggregate performance of hedge fund managers. These volumes allowed us to follow managers forward through time and study the effects of removing defunct funds -- excluding losers over the 1989 through 1995 period, using annual data appears to bias the index return by 2 to 3% per annum.2 But that is just the beginning. In more recent work together, we have found strong evidence that fund disappearance is strongly correlated to poor past performance, consistent with positive survival-induced performance bias.3 Fund volatility tends to increase when relative performance deteriorates and

fund survival depends in complex ways not only on absolute performance but also on ranking relative to other funds. This, in turn means that higher frequency and carefully collected data is absolutely crucial to capture the effects of funds that may have dropped out in the course of the year.

The obvious solution is to collect and maintain the data of as many non-surviving traders as possible. Some firms have recently begun to do this. Both TASS and MAR to our knowledge have begun in recent years to include "dead funds." Other companies using the information have also begun to recognize the value of saving data on dead funds. In particular PARADIGM Capital Management Inc., an affiliate of one of the authors, since 1990 has collected and maintained monthly data on as many non-surviving hedge fund and CTA track records as possible by purchasing and monitoring most of the major data vendors. Analysis done by PARADIGM shows that including these non-survivors into the database reduces the aggregate annual performance of hedge funds from 17.4% to 14.8% and of CTAs from 14.6% to 12.4%. The standard deviation drops slightly and is most likely due to the fact that an increase in the number of track records causes an increase in the effects of diversification. Obviously, it is not possible to collect all the performance data of non-survivors. Using the best data available, this correction tells us that the survivorship bias of not including the non-survivors is at least 2.6% per year for hedge funds and 2.2% per year for CTAs.

Unfortunately, the inclusion of non-survivors into the database does not completely eliminate survivorship bias. Another form of survivorship bias is also present. This is the phenomenon that many practitioners have observed after having reviewed numerous performance records. The phenomenon is the unusually good performance during the early part of a money manager's career. This is a self-selection bias and it arises because only those traders who by luck or skill accumulate an outstanding first year track record will select themselves (survive) into the database by organizing themselves into a hedge fund or CTA. In other words, traders are entering the database with instant histories and predominantly only those traders with good track records will choose to start reporting and solicit money. For this reason, instant histories tend to be uniformly good and therefore, bias the aggregate performance of traders upwards.

A switching regression technique developed in Park (1995) is a useful method to measure self-selection bias and to left-truncate performance records to reduce its effect.4 A switching regression is a method for identifying changes in economic regimes. While most economists use it to identify such things as interest rate regimes or exchange rate regimes, it is a useful tool to identify a change in the realized return of the manager, depending upon the time period. Returns over 25% or 35% per year in the first few months of a track record, followed by lower returns suggest that the first few months are "back-filled." The switching regression isolates the unusual early return period. This correction technique results in additional reduction in the aggregate annual returns of 1.9% for hedge funds and 2.1% for CTAs. Removing the early part of the performance record also results in a slight increase in standard deviation of the benchmarks.

There is still yet one more type of survivorship bias. It may not be enough to diligently maintain the records of traders who stop reporting their performance results. Even with these records, it may be that the last few months leading to a termination of trading may never be reported. This was observed in August and September of 1998 when there was a huge increase to market volatility due to the Russian default and the Long Term Capital Management debacle. Many traders suffering from huge losses that month terminated trading and never bothered to report the last catastrophic month to the data vendors. This catastrophe bias has not been measured or corrected for in the database. At this time, it is only possible to be aware of its existence.

Several consultants and data vendors continue to publish hedge fund and CTA indexes that show an aggregate performance of 17-22% per year for the last eight years. Using PARADIGM data and methods to control (but not eliminate) both types of survivorship bias has reduced the aggregate annual returns over the last eight years from 17.4% per year to 12.8% per year with a 6.1% standard deviation for hedge funds and from 14.6% per year to 10.3% per year with a 10.4% standard deviation for CTAs. Contrary to the assertion by some that survivorship bias is either unimportant or approximately 1-3% per year, there is substantial evidence that survivorship bias accounts for at least 451 basis points and 424 basis points of the aggregate annual performance of hedge funds and CTAs, respectively. In fact, because all non-surviving traders can never be completely accounted for and the effect of catastrophe bias has not been measured or corrected for, we know that the impact of survivorship bias is certainly larger than 451 basis points and 424 basis points per year for hedge funds and CTAs, respectively.

These results may help explain why investors are disappointed with their hedge fund and CTA portfolios. Many firms have offered clients upwardly biased returns causing false expectations. By not accounting for statistical biases, portfolios of hedge funds and CTAs have consistently failed to achieve their promised expectations. Of course there are other issues that may account for the failure to meet expectations -- our focus in this article is solely on survival. Unrealistic return expectations have also led to unduly burdensome fee and expense structures. Moreover, the quest for high returns has often led to under diversified portfolios of 4-8 hedge funds and/or CTAs which eventually lead to unacceptable losses. The result is an overall perception that fund of funds do not add any value.

With hedge funds and CTAs returning 12.8% per year and 10.4% per year, respectively, the alternative investments industry is not a get rich quick strategy. However, the corrected hedge fund and CTA indexes show a reward/risk ratio of 2.11 and 0.99, respectively. This compares quite favorably with the S&P 500's reward/risk ratio of 1.54 over the same eight years and 0.55 over the last seventy-two years. Clearly, the risk/reward profile of hedge funds and CTAs would benefit a traditional stock/bond portfolio. This is particularly true when the non-correlation that hedge funds and CTAs have with stocks and bonds is taken into account. As information processors, hedge funds and CTAs are not tied to the S&P500 or any other long only benchmark and therefore, would continue to perform despite what happens to the stock or bond market.

New information sources on managers reflect promises and perils. One promise is the potential for developing reliable benchmarks and performance measures for managers of alternative investments. By their very nature, these managers are hard to pigeonhole. As a group, they trade all the new, exotic securities the global market has created and employ state-of-the-art access to information flows. The peril of new data sources on manager track records is that survivorship and various selection criteria introduce biases. We have found that these criteria make a huge difference in developing meaningful benchmarks for hedge fund managers and CTAs. Our hope is that future changes in the restrictions on public disclosure of historical performance will continue to improve our ability to measure and benchmark manager performance. u

1 Brown, Stephen J., William N. Goetzmann and Roger G. Ibbotson, 1999, "Offshore Hedge Funds: Survival and Performance: 1989-1995," The Journal of Business 72(1), pp. 91-118. Our evidence on this is further detailed in The Report of the President's Working Group on Financial Markets, "Hedge Fund, Leverage and the Lessons of Long Term Capital" and compared to other industry estimates, available at www.hedgefundnews.com.

2 Brown, Stephen J., William N. Goetzmann and James Park, 1999, "Conditions for Survival: Changing Risk and Performance of Hedge Fund Managers and CTA's," NBER Working Paper available at www.fen.com

3 Brown, Stephen J., William N. Goetzmann and James Park, 1999, "Conditions for Survival: Changing Risk and Performance of Hedge Fund Managers and CTA's," NBER Working Paper available at www.fen.com

4 See James Park, 1995, "Managed Futures as an Investment Asset," Doctoral dissertation, Columbia University.

 
Back to Top