improving the power of predicted surprise with trna news...

1

ImprovIng the power of predIcted SurprISe wIth trnA newS SentImentStarMine reSearch note

nathan Meixler

reUterS/Stephen hird

mAY 2014

thoMSon reUterS financial and riSk ManageMent

the StarMine Smartestimate and predicted Surprise have long been reliable indicators for anticipating future analyst revisions and predicting the direction of actual surprises at report time. in this research note we use the thomson reuters news analytics (trna) dataset to further that predictive ability. filtering on predicted Surprise, we analyzed the direction of future consensus revisions in reaction to significantly positive or negative news events. We found that the combination of trailing news sentiment and predicted Surprise gives us a powerful tool to accurately predict the direction of both future consensus revisions and actual earnings surprises. our main findings are as follows: • combining news sentiment with predicted Surprise significantly improves upon our ability to predict the

direction of consensus revisions from either news sentiment or predicted Surprise alone. • the combination of predicted Surprise and trailing news sentiment helps us accurately predict the direction

of quarterly earnings actual surprises. the combination is more accurate than predicted Surprise or trailing news sentiment alone.

• these findings are exploitable to create profitable trading strategies. in one such example, we implement a long/short strategy which achieves 14% annualized returns over a 10 year period.

2


IntroductIonthe StarMine Smartestimate has a long and proven track record of being a more accurate prediction of actual earnings than the oft-cited consensus. Similarly, the predicted Surprise, defined as the percent difference between the Smart estimate and i/B/e/S consensus estimate, has shown to be a reliable leading indicator of future analyst revisions and actual earnings surprises. past research indicates that when the magnitude of predicted Surprise is greater than 2%, the predicted Surprise forecasts the direction of fY1 actual surprise to nearly 70% accuracy (Stauth & Bonne 2009).

Much of the predictive power of the Smartestimate comes from two main differences between it and the i/B/e/S consensus. first, by detecting clusters of recent analyst revisions, the Smartestimate excludes stale estimates that occurred prior to a ‘revision cluster.’ Second, the Smartestimate incorporates the analyst’s track record such that historically more accurate analysts bear more weight in the final aggregation (Bonne et al. 2007).

for this research note we investigate if we can improve upon the accuracy of the Smartestimate by incorporating knowledge about the information to which analysts have reacted. We identify positive and negative news events using the thomson reuters news analytics (trna) data set and examine how the consensus reacts to that news when the magnitude of predicted Surprise is significantly high or low. We find that when conditioned on trailing news sentiment, the predicted Surprise is able to more accurately anticipate the direction of consensus revisions and actual surprises. finally we use this knowledge to construct a simple trading signal and present our results.

thomSon reuterS newS AnAlYtIcSthomson reuters news analytics (trna) is a uniquely powerful natural language processing technology capable of automatically analyzing news within milliseconds and scoring it across multiple dimensions such as sentiment, relevance, and novelty (trna revolutionizing news analysis Brochure).

the sentiment engine analyzes the text of any news article and individually scores the mentioned companies based on the sentiment of the related text. in this fashion, the sentiment engine outputs a 0-1 probability that the story is positive (sent_pos), a 0-1 probability that the story is ‘neutral’ (sent_neut), and a 0-1 probability that the story is negative (sent_neg) where sent_pos + sent_neut + sent_neg = 1 (trna output image format Users’ guide).

our study makes use of a ‘net sentiment’ value constructed as sent_pos – sent_neg where a net sentiment of -1 represents extreme negative news and a value of +1 represents extreme positive news. the trna data set also includes a ‘relevance’ data item scored on a zero to one scale with higher values indicating the news article is more relevant to a particular company and lower values indicating that the news item is only minimally relevant to that company. for our analysis, we chose to only look at highly relevant novel reuters news items1.

methodologYinstead of examining the intraday impact of individual news items, we constructed news ‘events’ by aggregating the net sentiment of an individual company to the daily level. this aggregation is an average of (sent_pos – sent_neg) for the relevant articles on each day. We then ranked all net sentiment values across the entire news universe on a 1-100 scale, with the highest values representing the most positive sentiment on that given day. for this research note, we refer to any event with a daily score ≥ 91 (top decile) simply as positive news. Similarly, we refer to any event with a score ≤ 10 (bottom decile) as negative news.

our study looked solely at fQ1 epS estimates for S&p 1500 index constituents between 2003 and the third calendar quarter of 2013. here, ‘fQ1’ refers to the fact that the estimates under consideration pertained to the most recent unreported quarter. We obtained consensus estimates and company actuals from the i/B/e/S data set (Methodology for estimates). excluded from analysis were any news events for companies that underwent a stock split in the same quarter; less than 2% of the news events were omitted for this reason.

to examine the reaction in the epS consensus (i.e. the mean of all analysts’ estimates for the quarter), we measured the percent of cases experiencing an upward revision, downward revision, or no revision after positive or negative news. for this study, we only included news events that occurred after the previous quarter’s report date and prior to the current quarter’s period end date. to avoid counting small fluctuations in the consensus, we set a revision threshold of ± $.01 (1 cent). We also measured the percent of cases revising when the predicted Surprise was significantly positive or negative, specifically, when the predicted Surprise was ≥ 2% or ≤ -2%. for the remainder of this research note, we simply refer to these regimes as positive predicted Surprise and negative predicted Surprise with the inherent notion of this ±2% threshold.

note that the events we analyzed are fairly rare. on average, less than 5% of S&p 1500 companies have any news events which meet our criteria on any given day. When we consider only those events which occurred in the presence of significantly positive or negative predicted Surprise we are left with only a smattering of stocks. on average, there are approximately 29 negative news events each month when predicted Surprise is also negative and 15 positive news events each month when predicted Surprise is also positive. although, these events are infrequent in any given month, our study was conducted over a 10 year time period leaving us with a hearty population of events to examine.

for our second analysis, we tested the predicted Surprise hit rate when conditioned on trailing news sentiment. that is, we counted the number of times a positive predicted Surprise preceded a positive actual surprise (actual > consensus) and the number of times a negative predicted Surprise preceded a negative actual surprise (actual < consensus). in this section, our news signal was aggregated from the trailing net sentiment of varying periods between 1 and 360 market days.

1 Specifically, we took news items where Relevance ≥ .9, Item_type != ‘Alert’, Attribution=’RTRS’ and lnkd_cnt5 ≤ 1.

3


in order to measure the hit rate of the predicted Surprise throughout the span of a quarter the values of predicted Surprise and actual surprise were sampled from the last trading day of every month. that is, the predicted Surprise is calculated as:

SmartEstimateMonth End – ConsensusMonth End

MAX(.1,ABS(ConsensusMonth End))

and the actual surprise is calculated as:

ActualReport Date – ConsensusMonth End

MAX(.1,ABS(ConsensusMonth End))

in the above two equations the absolute value of the consensus is capped at 10 cents to avoid undo influence to small denominators. included as ‘hits’ were cases where predicted Surprise ≥ 2% preceded actual surprises > 0 and cases where predicted Surprise ≤ -2% preceded actual surprises < 0. the hit rate was then aggregated from all month-end samples as:

Total Number of Hits

Total Number of Observations

reActIon to poSItIve newSfigure 1 shows the aggregate effect of 11,170 positive news events on the epS consensus given no other conditions. after 30 days the percent of cases revising up (17%) was almost identical to the percent of cases revising down (18%). When we conditioned on positive predicted Surprise we were left with 1,360 total events. in these cases, upward revisions were much more likely. We see upward revisions occur 40% of the time after 30 days and downward revisions only 13% of the time. Upward revisions are 3 times more likely in this scenario than downward revisions.

reActIon to negAtIve newSin contrast, figure 3 shows the impact of 12,659 negative news events on the consensus without incorporating any predicted Surprise information. in these examples 33% of cases had revised downward after 30 days compared to only 12% revising upward. in figure 4, we see the analyst’s reaction to the 2,802 negative news events when predicted Surprise was also negative. here downward revisions occurred 64% of the time and upward revisions only 7% of the time. in this situation, a downward consensus revision is 9 times more likely than an upward revision.

fIgure 2. Upward revisions are more likely after positive news when predicted Surprise is also positive.

fIgure 4. likelihood of downward revisions increases to 64% when predicted Surprise is also negative.

fIgure 1. Upward and downward revisions are equally likely after positive news.

fIgure 3. downward revisions are more likely after negative news.

Consensus Revisions After Positive NewsWhen Predicted Surprise is Positive

Upward Downward No Revision

1.00

0.80

0.60

0.40

0.20

0.00Frac

tion

of C

ases

Rev

isin

g

1 6 11 16 21 26

Market Days Since News Event

Consensus Revisions After Positive News


1.00

0.80

0.60

0.40

0.20

0.00Frac

tion

of C

ases

Rev

isin

g

1 6 11 16 21 26


Consensus Revisions after Negative News


1.00

0.80

0.60

0.40

0.20

0.00Frac

tion

of C

ases

Rev

isin

g

1 6 11 16 21 26


Consensus Revisions after Negative NewsWhen Predicted Surprise is Negative


1.00

0.80

0.60

0.40

0.20

0.00Frac

tion

of C

ases

Rev

isin

g

1 6 11 16 21 26


4


teStIng the combInAtIon of newS SentIment And predIcted SurprISeas we have seen, when the predicted Surprise is in the same direction as the news sentiment, the analysts are much more likely to revise in line with the news. this is not entirely unexpected, though, as we already know that predicted Surprise alone is a good indicator of future consensus revisions. to test whether the news sentiment is additive we examined the reaction to different types of news when the predicted Surprise is held positive or negative and compare these results to the unconditioned news case. here the unconditioned news case refers to the analysts’ reaction to any news when the predicted Surprise is held positive or negative.

figures 5 and 6 show clear stratification in analysts’ reactions to the various types of news. We see that when the predicted Surprise is positive we are most likely to see upward revisions in the presence of positive news, and are least likely to experience upward revisions in the presence of negative news. Similarly when the predicted Surprise is negative we are most likely to experience downward revisions in the presence of negative news and least likely to see downward revisions in the presence of positive news. though these results are not altogether surprising, the differing reactions to varying news sentiment provides evidence we are seeing a real effect and not just the revisions expected from the value of the predicted Surprise alone.

predIctIng ActuAl SurprISeas we observed in the previous section, consensus reaction to positive or negative news is easier to anticipate when the predicted Surprise is also positive or negative. that is, when the predicted Surprise and news sentiment agree in direction, we are most likely to see consensus revisions in the same direction. We also were curious to see if there is an increased ability to predict the direction of quarterly epS actual surprise when the trailing news sentiment and predicted Surprise agree.

to test this idea, we constructed six different news sentiment signals by varying the trailing aggregation periods from 1 to 360 market days. We then calculated a hit rate when filtering to the top/bottom deciles, quintiles, and halves of trailing news sentiment. for comparison, we also included the hit rates for cases where there was any news and for cases where there was no news. We break out performance for positive predicted Surprise, negative predicted Surprise and the combination of the two in the following tables. Unconditioned hit rates are provided in the table headings for comparison.

tAble 1. Variations of trailing news sentiment help improve hit rates of predicting actual surprise. for example, when 7 day trailing news sentiment is in the top quintile, a positive predicted Surprise precedes a positive actual surprise 74% of the time, as compared to an unconditioned hit rate of 67%.

News Sentiment Decile

New Sentiment Quintile

News Sentiment Half

ANY News Sentiment

NO News Sentiment

1Day

2Day

7Day

30Day

90Day

360Day

0.69 0.75 0.76 0.71 0.70 0.71

0.67 0.72 0.74 0.72 0.71 0.73

0.70 0.72 0.73 0.71 0.71 0.71

0.68 0.69 0.69 0.68 0.67 0.67

0.67 0.67 0.66 0.66 0.65 0.65

Trailing News Aggregation Period

Positive Predicted Surprise Only (Unconditioned HitRate = .67 )

fIgure 6. When predicted Surprise is positive, positive news leads to the most upward revisions.

fIgure 5. When predicted Surprise is negative, negative news leads to the most downward revisions.

Upward Consensus Revisions After News When Predicted Surprise is Positive

Positive News Negative News No Conditioning

1

0.75

0.60

0.45

0.30

0.15

0.00Frac

tion

of C

ases

Rev

isin

g

6 11 16 21 26


0.75

0.60

0.45

0.30

0.15

0.006 11 16 21 26


Downward Consensus Revisions After News When Predicted Surprise is Negative

Positive News Negative News No Conditioning

Frac

tion

of C

ases

Rev

isin

g

5


2 Quarters ending on January 31st 2013 and April 30th 2013.

tAble 2. Variations of trailing news sentiment help improve hit rates ofpredicting actual surprise. for example, when 7 day trailing news sentiment is in the bottom quintile, a negative predicted Surprise precedes a negative actual surprise 64% of the time, as compared to an unconditioned hit rate of 54%.



News Sentiment Half

ANY News Sentiment

NO News Sentiment

1Day

2Day

7Day

30Day

90Day

360Day

0.68 0.64 0.64 0.63 0.58 0.58

0.65 0.64 0.64 0.62 0.60 0.60

0.59 0.59 0.59 0.59 0.58 0.56

0.56 0.56 0.57 0.56 0.55 0.54

0.54 0.54 0.53 0.52 0.52 0.53


Negative Predicted Surprise Only (Unconditioned Hit Rate = .54)

tAble 3. hit rates of predicting actual surprise when predicted Surprise ispositive or negative. Many variations of trailing news aggregation period andquantile help improve the hit rate over the unconditioned 60%.

fIgure 7. Jcp experienced signifi cant downward revisions and negative actual surprises in the 2 quarters ending January 31st and april 30th 2013.



News Sentiment Half

ANY News Sentiment

NO News Sentiment

1Day

2Day

7Day

30Day

90Day

360Day

0.68 0.67 0.68 0.66 0.63 0.62

0.66 0.67 0.67 0.66 0.64 0.64

0.63 0.64 0.64 0.63 0.63 0.61

0.61 0.62 0.62 0.61 0.60 0.60

0.60 0.60 0.59 0.58 0.58 0.58


Positive and Negative Predicted Surprise (Unconditioned Hite Rate = .60)

in general, we see that predicted Surprise combined with trailing news sentiment does a better job of anticipating actual surprises than the predicted Surprise alone. for example, with a 7 day lookback and conditioning on top/bottom sentiment deciles, we see that our positive surprise hit rate improves from 67% to 76% and our negative surprise hit rate improves from 54% to 64%. note that the magnitude of improve-ment over the unconditioned case is similar for both the positive and negative predicted Surprise. We also see that the unconditioned hit rate (60%) is well under the previously mentioned 70% hit rate reported

in the Stauth study. this discrepancy is entirely caused by the fi scal period under examination. Whereas the Stauth study considered the hit rate for annual fY1 epS predicted Surprise, we analyzed hit rates for a quarterly fQ1 epS predicted Surprise. two other noteworthy fi ndings are that:1. conditioning on trna quintiles seems to perform almost as well as

conditioning on trna deciles and 2. there seems to be at least some value conditioning on trailing

news sentiment as far back as 360 market days.

cASe eXAmple: Jc penneY And the bAd newS beArSUnfortunately for Jc penney shareholders, the last two years have not been very kind. the company had been plagued by decreasing consumer traffi c, debt issues, and a carousel of executives unable to spark a robust turnaround. over this period, the stock price declined nearly 75% as the company had been generating more bad news than sales growth. here we take a closer look at Q4 fY2012 and Q1 fY20132. for both of these periods the predicted Surprise was signifi cantly negative, averaging -95% and -26% throughout the January and april quarters respectively. in the fi gure, we see the Smartestimate signifi cantly lower than the consensus and the consensus steadily revising down throughout the quarter. We also see that when earnings were reported, the misses were severely negative (-1000% and -55%).

1.58

0.70

-0.18

-1.06

-1.94

-2.82

Pri

ce C

hang

e (%

)

EPS ($)

Consensus, SmartEstimate, and Actuals JC Penney November 2012 – May 2013

Q4 FY 2012 Q1 FY 2013D J F M A M

MeanSmartEstimate®

S&P 500Reported Actuals

Price

1.58

0.70

-0.18

-1.06

-1.94

-2.82

Pri

ce C

hang

e (%

)

EPS ($)

Consensus, SmartEstimate, and Actuals JC Penney November 2012 – May 2013

Q4 FY 2012 Q1 FY 2013D J F M A M

MeanSmartEstimate®

S&P 500Reported Actuals

Price

6


What this plot doesn’t show is the abysmal amount of negative company news released over these same two quarters. throughout this time period Jcp’s daily news sentiment was in the bottom decile 7 times, in the bottom quintile 18 times and in the lower half of news sentiment on 35 occasions. in the chart below, we see a selection of this extreme negative sentiment in the news throughout these two quarters.

A Selection of negative news Jc penney november 2012 – may 2013

date headline net Sentiment

11/9/12 Jc penney options volume swells as shares drop -0.76

1/3/13 Jc penney down 1.6 percent in premarket trading -0.70

1/11/13 Jc penney falls in premarket after UBS downgrade -0.76

2/5/13 Jc penney bonds, cdS shrug off default notice -0.61

2/27/13 Jc penney 4th-quarter same-store sales fall 31.7 percent -0.76

2/28/13 Jc penney shares fall after sales plunge -0.76

3/5/13 Vornado sale of Jc penney shares adds to growing pressure on ceo -0.76

3/8/13 York’s dinan squares off against ackman over Jc penney -0.71

3/12/13 Jc penney shares shoot up on talk ceo may leave -0.76

3/20/13 Jc penney says turnaround could take longer -0.60

3/28/13 put play on Jc penney suggests more downside -0.66

4/9/13 no quick fixes expected at Jc penney from returning ceo -0.50

4/9/13 Jc penney exposes inefficiency valuing ceos -0.76

4/10/13 Jcp change in ceo will not solve near term earnings pressure -0.76

4/10/13 three more top executives leave Jc penney -0.75

4/11/13 Martha Stewart loses bid to dismiss Macy‘s contract claim -0.74

4/15/13 Macy’s appeals against ruling on Martha Stewart goods in Jc penney -0.76

5/7/13 Jc penney says 1st quarter same-store sales down 16.6 percent -0.76

tAble 4. a selection of negative news for Jc penney.

although the negative news produced subsequent consensus revisions, the analysts severely under-reacted to the quantity of negative information hitting the wire. incorporating the news sentiment would have given us a powerful confirmatory signal to limit any upside exposure to this stock.

trAdIng StrAtegY And performAnce cAlculAtIonSas we have seen, the combination of news sentiment and predicted Surprise gives us a powerful tool for determining the direction of future consensus revisions and the direction of the actual surprise. With the combination of news sentiment and predicted Surprise shown to be synergistic, we test two simple low frequency long-short strategies designed to capitalize on this effect. for our benchmark we create a long portfolio of stocks with high predicted Surprise and a short portfolio of stocks with low predicted Surprise. to build our trna conditioned portfolios we incorporate a trna Signal with a 30 day lookback. that is, we construct a long portfolio of stocks with a positive predicted Surprise and trailing 30 day trna net sentiment in the top decile/quintile with the idea that these stocks will have a tendency to experience positive upward revisions and positive actual surprises. We take short positions in stocks with a negative predicted Surprise and trailing 30 day sentiment in the bottom decile/quintile with the idea that these stocks will be more likely to experience negative consensus revisions and be more prone to earnings misses. We rebalance monthly on the 1st of the month. our results are presented in table 5.

Summary of backtest results for long-Short Strategies

Spread long portfolio Short portfolio

Sharpe ratio

annualized return

Sharpe ratio

annualized return

portfolio Size (avg)

Sharpe ratio

annualized return

portfolio Size (avg)

predicted Surprise only 0.40 2.5% 0.69 15.4% 159 0.56 12.0% 204

predicted Surprise + trna Quintile 0.40 6.2% 0.90 19.8% 14 0.43 8.7% 28

predicted Surprise + trna decile 0.66 14.6% 0.97 24.3% 7 0.28 3.9% 14

tAble 5. a Summary of Backtest results 2003 – Q3 2013

7


long only S&p 1500 Strategies

Sharpe ratio

Annualized return

equal Weighted S&p15000.71 14.2%

predicted Surprise + trna Weighted S&p 1500 0.86 17.3%

over the 11 year period our predicted Surprise + trna decile portfolio was able to generate an annualized return of 14.6%. this dramatically beats the 2.5% return of the benchmark predicted Surprise only portfolio. during this time period we see that the majority of our performance can be attributed to the long portfolios, with our short portfolio actually generating a positive annualized return.

Most readers will notice the small portfolio sizes for these strategies. as previously mentioned, stocks with such extreme values of news sentiment and predicted Surprise are rare. as such, we are trading an average of 21 stocks every month when conditioned on deciles of trailing news sentiment and 42 stocks every month when conditioned on quintiles of news sentiment. due to the relatively small portfolio sizes this signal may be more useful as a tilt/weighting factor rather than as a standalone signal.

in table 6 and figure 8, we present results from one such long-only strategy differentially weighting S&p 1500 stocks based on our news sentiment + predicted Surprise signal. Starting with an equal weighted S&p 1500, we remove any stock from our portfolio which had a negative predicted Surprise and trailing 30 day news sentiment in the bottom quintile. any stock which exhibited a positive predicted Surprise and trailing 30 day news sentiment in the top quintile we overweighed by a factor of 100. from this simple strategy, we are able to show outperformance over the equal weighted S&p 1500 in both annualized return and Sharpe ratio.

700

600

500

400

300

200

100

0

Cum

ulat

ive

Ret

urns

(%)

1/1/031/1/04

1/1/051/1/06

1/1/071/1/08

1/1/091/1/10

1/1/111/1/12

1/1/13

Equal Weighted S&P 1500News Sentiment + Predicted Surprise Weighted S&P 1500

Cumulative Returns S&P 1500 Using Two Different Weighting Schemes

fIgure 8. a simple news sentiment + predicted Surprise weighting scheme outperforms the equal weighted S&p 1500.

tAble 6. a simple news sentiment + predicted Surprise weighting scheme increases the Sharpe ratio and annualized returns over the equal weighted S&p 1500.

concluSIonS We began our research note by expounding the accuracy of the StarMine Smartestimate and predicted Surprise. Still, we questioned whether news Sentiment and the Smartestimate would naturally enhance each other. By examining the sentiment of recent news stories and conditioning on regimes of predicted Surprise, we found the value of these two data sets to be additive. although predicted Surprise has been a solid indicator in the past, we find that conditioning on trailing news sentiment leads us to more accurately predict the direction of consensus revisions and the directions of actual surprise. further, we were able to use this information to develop simple, yet profitable low frequency trading strategies.

these data are available through thomson reuters via ftp, data feeds, or in our eikon desktop solution. We are confident that these data can add value to current investment research process of our readers and encourage them to contact StarMine quantitative consulting for more information ([email protected]).


Visit thomsonreuters.com

For more information, please contact your Thomson Reuters representative or StarMine Quantitative Consulting at: [email protected]

Visit us at: http://alphanow.thomsonreuters.com/

© thomson reuters 2014. all rights reserved. 1006645 05/14

cItAtIonSBonne, g., et al. 2007, StarMine Analyst Revisions Model (ARM), StarMine Whitepaper.

Stauth, J. and Bonne g., 2009, SmartEstimates and the Predicted Surprise: Construction and Accuracy

Thomson Reuters News Analytics Revolutionizing News Analysis, 2010, Marketing Brochure. available at customers.reuters.com

Methodology For Estimates, A guide to Understanding Thomson Reuters Methodologies, Terms and Policies for I/B/E/S Estimated Databases, June 2013. available at customers.reuters.com

About thomSon reuterS fInAncIAl And rISKfinancial and risk management solutions from thomson reuters help investors, traders and compliance professionals overcome their three most pressing challenges: to drive profits; to connect and collaborate via open, secure networks; and to efficiently manage risk and regulation.

About thomSon reuterSthomson reuters is the world’s leading source of intelligent information for businesses and professionals. We combine industry expertise with innovative technology to deliver critical information to leading decision makers in the financial and risk, legal, tax and accounting, intellectual property and science and media markets, powered by the world’s most trusted news organization. With headquarters in new York and major operations in london and eagan, Minnesota, thomson reuters employs approximately 60,000 people and operates in over 100 countries. thomson reuters shares are listed on the toronto and new York Stock exchanges.

for more information about thomson reuters, please go to www.thomsonreuters.com.

improving the power of predicted surprise with trna news...

Documents