1systematiq research| capital iq, a standard & poor’s business friday, october 23, 2015 a hard...

1SystematIQ Research| Capital IQ, A Standard & Poor’s Business

Friday, April 21, 2023

A Hard Knock Life: Why Analyst Accuracy Falls Short

QWAFAFEWMay 25, 2010

Carson Boneck CFA, David Pope CFA


AbstractIt is a commonly held belief among investors that there is persistence in security analyst's EPS forecast accuracy. Studies by Clement and Brown examine persistence directly and show that, at least in the pre Reg-FD period, there is a measurable amount of analyst accuracy persistence. We confirm this work and extend it to the Post-FD period, to the international markets, and then focus on the market profitability associated with persistence.

Perhaps to many practitioners' (non quants) surprise, we find that persistence is not all that it is believed to be. We show that while persistence is statistically measurable, it has very limited usefulness in predicting the future accurate analysts. Our work shows that using past accurate analysts forecasts leads to somewhat better results than the naive mean, but falls well short of alternative methods. Our work explores the metrics that make forecasts accurate rather than the individuals that make the forecasts themselves.


Agenda• Validate Clement / Brown Work

› Update for Post Reg FD

• Isolate importance of Past Skill

• Past skill as indicator of future accuracy

• Compare alternative metrics of forecasting:› Rock Stars, Time Weighted, Naïve Mean, BEST

• Q&A


How Good are the “Experts”, really?


We want to BELIEVE – “Lead Analyst” Syndrome• The reality is forecasting is hard

• Part of an Analyst’s role is to get noticed or get “shelf space”

• To be the most accurate, a forecast by definition has to be an outlier or bold forecast

• Analysts typically form their reputation by making one BIG call rather than being consistently right – Is Abby Joseph Cohen Accurate?

• Surprisingly clients we spoke with had never actually looked at Analysts’ accuracy themselves

• People like to believe in persistence of skill, or performance when it counts, often citing sports case studies› Hot Hand in Basketball - Gilovich, Vallone & Tversky (1985)› Clutch Hitting in Baseball – Cramer (1977)


Perfect Foresight is Very Profitable


Clement / Brown – Accuracy PersistenceRegress( abs(estimate error )- abs(avg estimate error)) / abs(avg estimate error) = estimate age demaned + Last Years Accuracy

Estimate Last Years Age Accuracy AdjAge Accuracy t t R^2

Pre RegFD 1992 0.00 -0.02 147.8 -18.0 2.3%1993 0.00 -0.03 176.8 -20.5 3.0%1994 0.00 -0.02 183.8 -16.1 3.2%1995 0.00 -0.03 210.0 -21.2 4.0%1996 0.00 -0.02 234.6 -14.4 4.6%1997 0.00 -0.03 207.5 -20.7 3.5%1998 0.00 -0.03 263.3 -22.2 5.2% Last Year of Larry Brown Study1999 0.00 -0.01 258.4 -10.6 5.0%2000 0.00 -0.01 291.3 -6.2 6.5%2001 0.01 -0.01 384.2 -10.3 11.5%2002 0.01 0.00 309.4 1.0 8.0%2003 0.00 -0.02 268.5 -13.4 6.1%

Post Reg FD 2004 0.00 -0.01 282.0 -11.0 5.8%2005 0.00 -0.02 283.9 -13.4 5.7%2006 0.00 -0.02 294.0 -17.4 5.9%2007 0.00 -0.01 283.0 -10.3 5.2%2008 0.00 -0.01 279.6 -9.3 5.3%2009 0.01 -0.02 189.7 -8.5 10.5%

expected sign + - + -


Clement / Brown – Accuracy Persistence (Top Decile)Top Decile of Last Years Analysts

Regress( abs(estimate error )- abs(avg estimate error)) / abs(avg estimate error) = estimate age demaned + Last Years Accuracy

Estimate Last Years Age Accuracy AdjAge Accuracy t t R^2

Pre RegFD 1992 0.00 -0.04 30.29 -11.52 2%1993 0.00 -0.02 35.136 -6.012 2%1994 0.00 -0.02 32.925 -6.991 2%1995 0.00 -0.03 32.79 -8.371 3%1996 0.00 0.01 44.911 2.47 4%1997 0.00 -0.02 34.10 -4.97 3%1998 0.00 -0.04 52.49 -12.12 6% Last Year of Larry Brown Study1999 0.00 -0.01 40.70 -3.93 4%2000 0.00 -0.01 63.60 -2.45 7%2001 0.01 0.01 83.00 1.82 15%2002 0.01 0.00 60.72 1.03 9%2003 0.00 -0.01 50.14 -2.54 6%

Post Reg FD 2004 0.00 -0.01 56.18 -4.70 6%2005 0.00 0.00 50.20 -1.45 5%2006 0.00 -0.01 57.50 -4.56 5%2007 0.00 -0.01 59.52 -5.15 5%2008 0.00 0.00 65.24 0.47 6%2009 0.01 0.02 42.00 2.86 12%

expected sign + - + -


Isolate Age – Regress on Accuracy

First Regress Out Age then Regress Resids on Last Years Skill

Top Decile Top Decile All All Bottom Decile Bottom DecileYear r-sq Avg DError Avg Dage Avg DError Avg Dage Avg DError Avg Dage

2001 0.01% -0.03 0.12 0.00 0.41 0.01 0.842002 0.00% -0.01 -0.20 0.00 0.44 0.00 -0.572003 0.03% -0.02 -1.10 -0.01 0.21 0.01 1.252004 0.02% -0.05 -1.38 0.00 0.37 0.02 0.212005 0.01% -0.04 -0.91 0.00 0.28 0.01 0.522006 0.01% -0.04 -0.36 0.00 0.36 0.01 0.362007 0.01% -0.04 -1.36 0.00 0.31 0.01 0.282008 0.01% -0.04 -1.53 0.00 0.17 0.02 0.982009 0.02% -0.05 0.10 0.00 0.18 0.05 0.91

Average 0.014% -0.04 -0.74 0.00 0.30 0.02 0.53


Isolate Accuracy – Regress on Age

First Regress Out Last Year kKill then Regress Resids on Age

Top Decile Top Decile All All Bottom Decile Bottom DecileYear r-sq Avg DError Avg Dage Avg DError Avg Dage Avg DError Avg Dage2001 11.5% -0.16 -41.19 0.00 0.41 0.27 31.062002 8.0% -0.12 -41.24 0.00 0.44 0.20 28.842003 6.1% -0.10 -40.80 -0.01 0.21 0.16 28.522004 5.8% -0.08 -39.20 0.00 0.37 0.15 27.522005 5.7% -0.10 -39.31 0.00 0.28 0.15 28.062006 5.9% -0.10 -41.17 0.00 0.36 0.16 29.432007 5.2% -0.09 -40.30 0.00 0.31 0.14 28.412008 5.2% -0.08 -40.23 0.00 0.17 0.14 29.292009 10.4% -0.14 -45.88 0.00 0.18 0.31 37.93

Average 7.098% -0.108 -41.035 -0.001 0.303 0.187 29.897


Skill Looks to Mean Revert (1990 – 2009)

BEST WORSTLast Year Accuracy Decile

1 2 3 4 5 6 7 8 9 10This years Accuracy Decile BEST 1 6% 9% 10% 10% 16% 14% 11% 12% 9% 4%

2 3% 8% 10% 14% 17% 13% 13% 10% 7% 5%3 1% 4% 10% 23% 25% 15% 10% 6% 4% 3%4 0% 2% 6% 24% 33% 19% 8% 4% 2% 2%5 0% 1% 4% 18% 36% 24% 10% 4% 2% 2%6 0% 1% 3% 13% 31% 29% 14% 6% 2% 1%7 0% 1% 3% 9% 24% 29% 19% 9% 4% 2%8 0% 1% 3% 8% 19% 25% 22% 13% 6% 4%9 0% 1% 2% 6% 15% 22% 23% 17% 9% 5%

WORST 10 0% 1% 2% 5% 10% 14% 20% 21% 16% 11%


Post FD – Story is the Same (2001-2009)

Post FD

Last Year Accuracy Decile1 2 3 4 5 6 7 8 9 10

This years Accuracy Decile BEST 1 7% 10% 10% 9% 17% 14% 11% 12% 9% 3%2 3% 9% 11% 15% 16% 13% 12% 9% 7% 5%3 1% 4% 10% 24% 25% 14% 10% 6% 3% 3%4 0% 2% 6% 24% 33% 18% 8% 4% 2% 2%5 0% 1% 4% 18% 35% 23% 10% 4% 2% 2%6 0% 1% 3% 14% 31% 27% 13% 6% 2% 1%7 0% 1% 3% 11% 25% 28% 17% 8% 4% 2%8 1% 2% 3% 9% 20% 25% 20% 12% 6% 4%9 1% 2% 3% 7% 16% 22% 21% 15% 9% 4%

WORST 10 1% 2% 2% 5% 11% 15% 20% 20% 15% 11%


In the (not so) Long-Run All Analysts Are Average


High T-Stat, Low R^2 – Explaining a lot of nothing• Quantitative Slight of Hand

› Neither Clement or Brown ever show R^2’s on past accuracy, only multiple regressions

• Analogy:

Think about medicine as an example. Maybe they can find out that coffee drinking has a statistically significant and even important effect on the probability of heart disease, but still not be able to predict whether or not you will suffer from heart disease, regardless of your coffee-drinking habits.

- Leamer


How Do You Define Accuracy• Two Definitions of Accuracy

› Demeaned Accuracy- Allows easy comparisons across stocks, time- [Abs(Estimate – Actual) – Abs(Average Error) ] / Abs( Avg Error)

› Accuracy relative to Actual- Probably the first thing most investors think about when considering accuracy- Abs(Estimate – Actual) / Abs(Actual)


Rockstars Mean Revert (Post FD)Avg Error RockStar = 0, TimeWeighted = -.16


“RockStar” Analysts are BOLD, but not Accurate


The Reality of Analysts’ (Lack of) Accuracy


Even “Rockstars” don’t beat Naïve Mean Consensus


What, then, makes a Forecast more Accurate?• We know Analysts are not Accurate; they have

more error than a naïve consensus

• Literature proves us a guide into other areas• Clement, Lee

Estimate Age Broker Size Tenure

Importance:1000

Importance:100

Importance:10


Alternative – Focus on Factors that Make Forecast Accurate

Regress abs(a nalyst error - averag e erro r) / ab s(averag e erro r) = each of the fa ctor s d em ea ned univaria te ly belo w

Estim a te Ad j B roker Adj Analyst Adj Estim ate Dum m y Ag e SizeAge t R ^2 Size t R ^2 Tenure t R ^2 Age Bro ker Size t t

Pre RegFD 1989 0.001 80.32 0.67% 0.000 -53.92 0.30% -0.014 -42.96 0.19% 0.001 -0.066 80.48 -29.161990 0.003 146.3 2.29% 0.000 -56.93 0.35% -0.006 -21.89 0.05% 0.003 -0.059 146.39 -30.611991 0.003 152.8 2.57% 0.000 -62.37 0.44% -0.005 -19.21 0.04% 0.003 -0.044 152.81 -22.381992 0.003 146.1 2.25% 0.000 -48.99 0.26% -0.004 -16.33 0.03% 0.003 -0.022 146.214 -9.9761993 0.003 178.4 3.05% 0.000 -48.64 0.23% -0.006 -27.58 0.08% 0.003 0.001 178.349 0.3041994 0.003 184.2 3.18% 0.000 -55.76 0.30% -0.003 -18.72 0.03% 0.003 -0.011 184.183 -5.7531995 0.004 210.1 3.89% 0.000 -55.19 0.28% -0.002 -12.27 0.01% 0.004 -0.012 210.112 -6.6081996 0.004 235.9 4.61% 0.000 -64.4 0.36% -0.002 -10.59 0.01% 0.004 -0.007 235.93 -3.771997 0.003 207.8 3.47% 0.000 -60.03 0.30% -0.001 -9.754 0.01% 0.003 0.000 207.802 -0.2391998 0.004 262 5.10% 0.000 -73.11 0.42% -0.001 -4.254 0.00% 0.004 -0.069 261.670 -40.1501999 0.004 260.1 4.97% 0.000 -60.1 0.28% 0.000 3.629 0.00% 0.004 -0.062 259.800 -37.0002000 0.005 294.2 6.46% 0.000 -66.91 0.36% 0.001 4.483 0.00% 0.005 -0.039 293.660 -24.110

Post R eg FD 2001 0.007 386.5 11.39% 0.000 -69.3 0.41% 0.001 6.243 0.00% 0.007 -0.039 385.700 -21.8002002 0.006 320.8 8.09% 0.000 -31.26 0.08% 0.001 4.342 0.00% 0.006 -0.023 320.900 -13.5202003 0.005 280.8 6.29% 0.000 -33.62 0.10% -0.001 -4.937 0.00% 0.005 -0.041 280.640 -22.4802004 0.005 286.9 5.83% 0.000 -47.88 0.17% 0.000 -0.965 0.00% 0.005 -0.021 286.720 -12.5102005 0.005 287.7 5.65% 0.000 -48.76 0.17% 0.001 7.873 0.00% 0.005 -0.020 287.610 -12.2402006 0.005 299.9 5.97% 0.000 -35.86 0.09% 0.000 -0.527 0.00% 0.005 -0.007 299.861 -4.4992007 0.004 288.4 5.28% 0.000 -43.72 0.13% 0.000 -2.175 0.00% 0.004 -0.016 288.160 -10.6902008 0.007 283.2 5.27% 0.000 -22.05 0.03% -0.002 -16.21 0.02% 0.004 -0.006 283.092 -4.2882009 0.004 187.9 10.38% 0.000 -9.796 0.03% -0.002 -12.54 0.05% 0.007 0.007 187.912 1.782

exp ected sign + - - + - + -


Similar Results InternationallyRegress abs(analyst error - average error) / abs(average error) = each of the factors demeaned univariately below

Estimate Adj Broker Adj Analyst Adj Estimate Dummy Age Size AdjAge t R^2 Size t R^2 Tenure t R^2 Age Broker Size t t R^2

Pre RegFD 1989 0.0 87.8 2% 0.00 -4.671 0.01% -0.01 -8.587 0.02% 0.00 0.00 88.36 1.19 2.3%1990 0.0 111.6 3% 0.00 -15.47 0.06% -0.02 -17.58 0.08% 0.00 -0.01 111.66 -4.70 3.1%1991 0.0 154.7 4% 0.00 -8.648 0.01% 0.00 -2.904 0.00% 0.00 -0.03 154.78 -12.40 3.6%1992 0.0 153.1 3% 0.00 -10.59 0.01% -0.01 -30.18 0.10% 0.00 -0.02 153.80 -12.19 2.6%1993 0.0 172.4 3% 0.00 -10.73 0.01% 0.00 1.706 0.00% 0.00 -0.01 171.98 -6.81 2.8%1994 0.0 133.5 1% 0.00 -6.459 0.00% 0.00 6.176 0.00% 0.00 -0.01 132.93 -5.86 1.4%1995 0.0 184.6 2% 0.00 -9.024 0.01% 0.00 2.321 0.00% 0.00 -0.02 183.60 -12.91 2.1%1996 0.0 207.9 2% 0.00 -12.31 0.01% 0.00 3.812 0.00% 0.00 -0.01 207.18 -7.65 2.3%1997 0.00 174.4 2% 0.00 -27.31 0.04% 0.00 22.19 0.02% 0.00 0.00 174.30 -16.29 1.5%1998 0.00 210.7 2% 0.00 -42.4 0.09% 0.00 6.05 0.00% 0.00 -0.03 209.48 -27.27 2.3%1999 0.00 171.9 1% 0.00 -29.98 0.05% 0.00 1.406 0.00% 0.00 -0.03 171.50 -20.10 1.5%2000 0.00 184.5 2% 0.00 -13.38 0.01% 0.00 4.334 0.00% 0.00 -0.02 183.38 -13.70 1.9%

Post Reg FD 2001 0.00 222.5 3% 0.00 -9.758 0.01% 0.00 15.5 0.01% 0.00 0.00 221.21 1.31 2.8%2002 0.00 220.5 3% 0.00 18.37 0.02% 0.00 9.282 0.00% 0.00 0.04 219.68 25.20 2.7%2003 0.00 200.4 2% 0.00 -4.609 0.00% 0.00 -5.02 0.00% 0.00 0.00 199.94 0.52 2.1%2004 0.00 215.1 2% 0.00 -11.97 0.01% 0.00 -2.011 0.00% 0.00 0.00 214.38 -0.43 2.2%2005 0.00 227.8 2% 0.00 -24.46 0.03% 0.00 -5.331 0.00% 0.00 -0.01 227.10 -6.40 2.5%2006 0.00 213.3 2% 0.00 -26.08 0.03% 0.00 -2.28 0.00% 0.00 -0.01 211.86 -7.42 2.0%2007 0.0 202.8 2% 0.00 -33.12 0.04% 0.00 -5.275 0.00% 0.00 -0.02 201.83 -16.16 1.6%2008 0.0 234.4 2% 0.00 -47.04 0.08% 0.00 -11.05 0.00% 0.00 -0.02 233.26 -20.97 2.0%2009 0.0 293.6 3% 0.00 -27.39 0.03% 0.00 -16.54 0.01% 0.00 -0.01 293.205 -7.956 3.72%

expected sign + - - + - + -


Beware….• Forecast Accuracy is not the same as market impact

› Take an average analyst at a top brokerage› Compare to a skilled unknown analyst at a brokerage we never heard of…..› Analyst typically establish their reputation on ONE big call› II Analysts (Institutional Investor) Poll


BibliographyBagnoli, Levine, Watts, 2003,”Analyst Estimation Revision Clusters and Corporate Events, Parts I and II”, Working Paper

Beaver, Cornell, et al, 2008,” The Impact of Analysts’ Forecast Errors and Forecast Revisions on Stock Prices”, Journal of Business Finance & Accounting, 35(5) & (6), 709–740

Brown, Lawrence, 2001, “How Important is Past Analyst Forecast Accuracy?”, Financial Analysts Journal, Vol 57, pp 44-49.

Chan, Jegadeesh, Lakonishok, 1996, “Momentum Strategies “, Journal of Finance

Clement, M. B., 1999. “Analyst forecast accuracy: Do ability, resources, and portfolio complexity matter?”, Journal of Accounting and Economics, 27 (3): 285-303.

Clement, Tse, 2003, “Do Investors Respond to Analysts' Forecast Revisions as If Forecast Accuracy Is All That Matters?”, The Accounting Review, Vol. 78 (1), 227-249

Doyle, Lundholm, Soliman, 2006, “The Extreme Future Stock Returns Following I/B/E/S Earnings Surprise”, Journal of Accounting Research, 44 (5)

Elgers, P., M. Lo, and R. Pfeiffer,J r, 2001, “Delayed security price adjustment to financial analysts' forecasts of annual earnings”, The Accounting Review, 76 (4): 613-632.

Gleason, C., and C. Lee, 2003, “Analyst forecast revisions and market price discovery”, The Accounting Review, 78 (1): 193-225.

Mikhail, Walther, Lewis, 1999, “Does Forecast Accuracy Matter to Security Analysts?”,The Accounting Review, Vol. 74( 2), pp. 185-200

Sinha, Brown, Das, 1997, “A Re-Examination of Financial Analysts’ Differential Earnings Forecast Accuracy”, Contemporary Accounting Research; 14, 1;


Interesting Stats – US: FY1

Y ear # StocksCo vered # Ana lysts # B roker Stock-Analyst Stoc k-B roker EstT enu re E stim a teAg e Ana lyst-Broker # T en ure topQ T enu re bott o m QT enure1989 4078 2705 184 7.6 112.4 4.3 62.3 14.7 1,074,981 3.7 7.7 0.231990 3967 2677 191 7.3 101.9 4.6 58.0 14.0 1,012,435 4.1 8.6 0.541991 3877 2362 190 7.9 98.8 5.3 55.5 12.4 976,423 4.9 9.4 0.591992 4157 2235 198 8.5 96.3 6.2 55.1 11.3 1,010,437 5.5 10.4 0.641993 4498 2414 222 8.6 93.8 6.7 55.9 10.9 1,103,558 5.6 11.8 0.341994 5085 2830 231 7.8 95.4 6.9 57.2 12.3 1,146,323 5.5 12.4 0.351995 5639 3099 236 7.5 99.1 7.0 56.0 13.1 1,215,823 5.5 12.5 0.381996 6357 3497 263 7.0 93.7 7.1 56.8 13.3 1,281,370 5.2 12.8 0.331997 6869 3900 308 6.7 85.3 6.9 56.5 12.7 1,366,492 4.9 12.8 0.331998 6898 4348 341 6.3 80.7 6.9 56.5 12.8 1,458,400 5.0 13.4 0.381999 6684 4591 342 6.0 80.2 6.9 56.8 13.4 1,453,759 5.1 14.2 0.412000 6226 4660 323 5.8 83.5 6.8 57.6 14.4 1,401,955 5.2 14.0 0.442001 5350 4641 314 5.3 78.2 6.8 53.6 14.8 1,277,551 5.2 14.6 0.372002 4767 4814 263 5.0 91.1 6.5 53.3 18.3 1,246,362 4.8 14.1 0.382003 4658 4879 342 4.9 70.2 6.5 51.8 14.3 1,248,955 4.7 13.7 0.392004 4928 4638 395 5.8 67.8 6.7 50.9 11.7 1,419,823 4.8 14.1 0.442005 5112 4586 405 6.3 70.8 6.7 51.7 11.3 1,491,761 5.2 14.1 0.482006 5325 4628 379 6.3 77.4 6.9 52.9 12.2 1,526,356 5.4 14.5 0.512007 5433 4709 353 6.6 87.7 7.0 52.7 13.3 1,609,796 5.4 14.3 0.472008 5106 4565 341 6.5 87.0 7.3 50.7 13.4 1,542,621 5.7 15.1 0.612009 4885 4207 360 6.7 78.7 7.8 51.1 11.7 1,501,152 6.2 16.7 0.77


Interesting Stats – Intl: FY1

Year #StocksCovered #Analysts #Broker StockPerAnalyst Stock-Broker EstTenure EstimateAge Analyst-Broker # Tenure topQTenure bottomQTenure1989 4724 1505 254 7.6 45.2 1.8 64.5 5.9 596,423 1.68 2.95 0.351990 4599 2023 295 6.3 43.2 2.4 63.3 6.9 663,098 1.73 3.68 0.251991 5519 2380 328 6.4 46.6 2.9 61.1 7.3 795,447 2.05 4.53 0.301992 6679 3173 397 6.7 53.2 3.5 61.1 8.0 1,118,865 2.24 5.10 0.291993 7331 4480 479 5.2 48.4 3.7 63.3 9.4 1,228,241 2.25 5.29 0.251994 8149 5352 537 5.4 53.4 3.2 65.4 10.0 1,490,118 2.26 5.56 0.281995 9211 7110 577 4.9 60.9 3.2 64.7 12.3 1,828,268 2.32 5.92 0.251996 9595 8234 599 4.9 67.5 3.1 66.3 13.7 2,103,468 2.40 5.93 0.291997 10081 9116 623 4.7 68.8 3.0 65.4 14.6 2,227,660 2.44 6.25 0.321998 9855 9941 652 4.2 63.4 3.0 66.6 15.2 2,191,471 2.61 6.48 0.341999 9499 10338 619 4.1 68.3 3.1 64.5 16.7 2,242,210 2.74 6.77 0.342000 9495 10683 571 3.7 68.7 3.3 67.9 18.7 2,038,502 2.88 7.14 0.332001 8898 10874 567 3.4 64.4 3.3 64.0 19.2 1,899,902 2.97 7.40 0.362002 8364 10430 535 3.5 68.7 3.3 63.0 19.5 1,912,244 3.07 7.55 0.412003 8530 10532 560 3.7 68.7 3.7 61.0 18.8 2,001,120 3.20 7.93 0.362004 9333 9678 590 4.2 68.5 4.1 61.0 16.4 2,141,238 3.54 8.56 0.412005 10243 9796 625 4.2 65.6 4.7 61.7 15.7 2,132,897 3.91 9.24 0.412006 11357 10799 660 4.1 66.8 4.9 61.2 16.4 2,293,725 3.97 9.83 0.342007 12610 12099 688 4.1 72.4 5.0 61.6 17.6 2,589,599 3.93 10.16 0.332008 13085 12492 709 4.3 75.7 5.2 59.1 17.6 2,791,745 4.15 10.63 0.452009 13328 12687 800 4.5 71.0 5.5 58.0 15.9 3,011,257 4.47 11.55 0.54

1systematiq research| capital iq, a standard & poor’s business friday, october 23, 2015 a hard...

Documents