kenett on information nyu-poly 2013

Financial and Risk

Applications of InfoQ

Prof. Ron S. Kenett

KPA Ltd., Raanana, Israel

Universita degli Studi di Torino, Turin, Italy

NYU Poly, New York, USA

ron@kpa-group.com

Three case studies (1/3)

1. Predicting Changes in Quarterly Corporate Earnings Using Economic Indicators

http://galitshmueli.com/content/predicting-changes-quarterly-corporate-earnings-using-economic-indicators

This study looks at corporate earnings in relation to an existing theory of business forecasting developed by Joseph H. Ellis (former research analyst at Goldman Sachs).

2

Three case studies (2/3)

2. Predicting ZILLOW.com’s Zestimate accuracy

http://galitshmueli.com/content/predicting-zillowcom-s-zestimate-accuracy

Zillow.com is a free real estate service that calculates an estimated home valuation ("Zestimate") as a starting point for anyone to see for most homes in the U.S. The study looks at the accuracy of Zestimates.

3

Three case studies (3/3)

3. Predicting First Day Returns for Japanese IPOs

http://galitshmueli.com/content/predicting-first-day-returns-japanese-ipos.

An Initial Public Offering (IPO) is the first sale of stock by a company to the public. The study looks at the first-day returns on IPOs of Japanese companies.

4

InfoQ(f,X,g) = U( f(X|g) ) Depends on quality of g, X, f, U and relationship between them

The potential of a particular dataset to achieve a particular goal using a given empirical analysis method

5

g A specific analysis goal

X The available dataset

f An empirical analysis method

U A utility measure

Information Quality

Kenett, R.S. and Shmueli , G. (2013) On Information Quality, http://ssrn.com/abstract=1464444 Journal of the Royal Statistical Society, Series A (with discussion), 176(4).

Analysis goal

g Explain, predict, describe enumerative, analytic, exploratory, confirmatory

Goal Specification • “error of the third kind” - giving the right answer to the wrong

question – Kimball • “Far better an approximate answer to the right question, which

is often vague, than an exact answer to the wrong question, which can always be made precise” - Tukey

6

Analysis goal

g

7

Goal 1. Decide where to launch improvement initiatives

Goal 2. Highlight drivers of overall satisfaction

Goal 3. Detect positive or negative trends in customer satisfaction

Goal 4. Identify best practices by comparing products

Goal 5. Determine strengths and weaknesses

Goal 6. Set up improvement goals

Goal 7. Design a balanced scorecard with customer inputs

Goal 8. Communicate the results using graphics

Goal 9. Assess the reliability of the questionnaire

Goal 10. Improve the questionnaire for future use

Typical Goals of Customer Surveys

X Available data

Data Source • Primary, secondary • Observational, experiment • Single, multiple sources • Collection instrument, protocol

Data Type • Continuous, categorical, semantic • Structured, un-, semi-structured • Cross-sectional, time series, panel,

network, geographical

Data Quality • “Zeroth Problem - How do the data relate to the problem, and

what other data might be relevant?” - Mallows • Quality of Statistical Data (IMF, OECD) - usefulness of summary

statistics for a particular goal (7 dimensions)

Data Size and Dimension • # observations • # variables

8

f Data analysis

method

Analysis Quality • “poor models and poor analysis techniques, or even analyzing the

data in a totally incorrect way.” - Godfrey • Analyst expertise • Software availability • The focus of statistics education

Statistical models and methods • Parametric, semi-, non-parametric • Classic, Bayesian

Data mining algorithms Graphical methods Operations research methods

9

Utility measure

U

Utility Measure • Adequate metric from analysis standpoint (R2, holdout data) • Adequate metric from domain standpoint

• Predictive accuracy, lift • Goodness-of-fit • Statistical power, statistical significance • Strength-of-fit • Expected costs, gains • Bias reduction, bias-variance tradeoff

10

11

Goal of study:

1. Predict the final price of an Ebay auction at start of auction

2. Predict price during ongoing auction

3. Predict the auctions with the highest prices (ranking)

4. Identify factors that determine the final price of an eBay auction?

“Pennies from ebay: The determinants of price in

online auctions” Lucking-Reiley D., Bryan D.,

Prasad N. & Reeves D. Journal of Indust. Econ., 2007

An example….

X Available data

Analysis goal

g

12

461 eBay coin auctions (Indian Head pennies) Auction characteristics

Duration Open and close prices Number of bids and bidders Secret reserve price Weekday/weekend ending

Seller characteristics Seller rating

Item characteristics Year and grade of coin

X Available data

“Pennies from ebay: The determinants of price in

online auctions” Lucking-Reiley D., Bryan D.,

Prasad N. & Reeves D. Journal of Indust. Econ., 2007

13

Dimension Reduction

f Data analysis

method

An example….

14

Prediction error:

• Holdout data

• Metrics such as MAPE and RMSE

f Data analysis

method

Utility measure

U

An example….

Statistical Approaches for Increasing InfoQ

Study Design (Pre-Data)

• DOE

• Clinical trials

• Survey sampling

• Computer experiments

Post-Data-Collection

• Data cleaning and preprocessing

• Re-weighting, bias adjustment

• Meta analysis

Randomization, Stratification, Blinding, Placebo, Blocking, Replication, Sampling frame, Link data collection protocol with appropriate design

Recovering “real data” vs. “cleaning for the goal” Handling missing values, outlier detection, re-weighting, combining results

15

Assessing InfoQ “Quality of Statistical Data” (Eurostat, OECD, NCSES,…) • Relevance • Accuracy • Timeliness and punctuality • Accessibility • Interpretability • Coherence • Credibility

InfoQ dimensions 1. Data resolution 2. Data structure 3. Data integration 4. Temporal relevance 5. Chronology of data and goal 6. Generalizability 7. Operationalization 8. Communication

3 V’s of Big Data • Volume • Variety • Velocity

Marketing Research • Recency • Accuracy • Availability • Relevance 16

4 V’s of Big Data • Volume • Variety • Velocity • Veracity

#1 Data Resolution

17

#2 Data Structure

Data Types • Time series, cross-sectional, panel • Structured, semi-, non-structured • Geographic, spatial, network • Text, audio, video, semantic • Discrete, continuous

Data Characteristics Corrupted and missing values due to study design or data collection mechanism

18

19

www.riscoss.eu

Managing Risk and Costs in OSS Adoption

#2 Data Structure

20

#2 Data Structure

21

Who talks to whom?

IRC chat archives: http://dev.xwiki.org/xwiki/bin/view/IRC/WebHome

XWiki Community

#2 Data Structure

XWiki Community Use association rules To characterize the content of the clusters (tm, arules)

#2 Data Structure

XWiki Community

#2 Data Structure

#3 Data Integration

Linkage, privacy-preserving methods: Increase or decrease InfoQ?

24

#4 Temporal Relevance

Analysis Timeliness (solving the right problem too late)

Data Collection

Data Analysis

Study Deployment

t1 t2 t3 t4 t5 t6

Collection Timeliness (relevance to g)

g: Prospective vs. retrospective; longitudinal vs. snapshot Nature of X, complexity of f

forecast

25

#5 Chronology of Data & Goal

Data: Daily AQI in a city g1: Reverse-engineer AQI g2: Forecast AQI Retrospective/prospective Ex-post availability Endogeneity

26 http://www.airnow.gov/?action=aqibasics.aqi

#6 Generalizability

Statistical generalizability

Scientific generalizability

Definition of g Choice of X, f, U 27

#7 (Construct) Operationalization

χ construct X = θ(χ) operationalization (measurable)

• Causal explanation vs. prediction, description

• Theory vs. data • Data: Questionnaire,

physio measurement

28

#7 (Action) Operationalization

29 http://www.spcpress.com/pdf/DJW187.pdf

#7 Operationalization

30

National Education Goals Panel (NEGP) recommended that states answer four questions on their student reports: 1. How did my child do? 2. What types of skills or knowledge does his or her performance reflect? 3. How did my child perform in comparison to other students in the school, district, state, and, if available, the nation? 4. What can I do to help my child improve?

31

#7 Operationalization

http://sat.collegeboard.org/practice/sat-skills-insight/writing/band/200

32

#7 Operationalization

http://sat.collegeboard.org/practice/sat-skills-insight/writing/band/200

33

When asked what the 18% in line 1 meant, 53% of the policy makers responded incorrectly

1992 NAEP Executive

Summary Report

#8 Communication

43 16 2

34

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

35

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

36

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

37

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

38

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

39

#8 Communication

http://nces.ed.gov/nationsreportcard/itemmaps/index.asp

40 The Israeli version……

#8 Communication

http://rama.education.gov.il

'

N " N " N "

" 18,684 501 102 13,182 521 87 5,502 454 118

" 21,407 500 100 14,466 524 84 6,941 444 111

" 20,644 524 91 14,787 536 80 5,857 496 106

" 19,165 524 86 13,379 532 77 5,786 506 101

" 19,631 532 76 13,961 537 73 5,670 519 81

* " 20,222 528 77 13,957 541 70 6,265 498 82

41

http://www.madlan.co.il/education/schools

#8 Communication

The Israeli version……

Assessing InfoQ in Practice

Rating-based assessment

1-5 scale on each dimension:

InfoQ Score = [d1(Y1) d2(Y2) … d8(Y8)]1/8

Experience from two research methods courses

– Preparing a PhD research proposal (U Ljubljana, 50 students, goo.gl/f6bIA)

– Post-hoc evaluation of five completed studies (CMU, 16 students, goo.gl/erNPF)

42

# Dimension Note Value Index

1 Data resolution 5 1.0000

2 Data structure 4 0.7500

3 Data integration 5 1.0000

4 Temporal relevance 5 1.0000

5 Generalizability 3 0.5000

6 Chronology of data and goal 5 1.0000

7 Concept operationalization 2 0.2500

8 Communication 3 0.5000

InfoQ Score = 0.68

InfoQ=68%

InfoQ: Strengths and Challenges InfoQ approach streamlines questioning of data value • “Why should we invest in data?” – management

• Compare value of potential datasets, analyses

• Prioritize/rank projects

• Strengthen functional – analytical relationship

Multiple goals: • Goals can change during study: Reevaluate InfoQ

• Multiple goals: Prioritize. – clinical trials: effect of new drug, adverse effects

To Do: • Improve InfoQ assessment • Alternative InfoQ assessment approaches (pilot study, EDA, other) • Further dimensions (data privacy, human subject compliance and risk) • Effect of technological advances on InfoQ 43

Primary Data Secondary Data - Experimental - Experimental - Observational - Observational

Data

Quality

Information

Quality

Analysis

Quality

Knowledge

g A specific analysis goal

X The available dataset

f An empirical analysis method

U A utility measure

1.Data resolution

2.Data structure

3.Data integration

4.Temporal relevance

5.Chronology of data and goal

6.Generalizability

7.Operationalization

8.Communication

What

How

Goals InfoQ(f,X,g) = U(f(X|g))

Information Quality

Russom, P., Big Data Analytics, TDWI Best Practices Report, Q4 2011

Massive data sets

1. Data resolution

2. Data structure

3. Data integration

4. Temporal relevance

5. Chronology of data and goal

6. Generalizability

7. Operationalization

8. Communication

Big data Analytics

Three case studies (1/3)

1. Predicting Changes in Quarterly Corporate Earnings Using Economic Indicators

Stages in economic downturn: 1) the peak, 2) modest slowing, 3) intensifying worrying by investors (a lot of panic selling occurs in this stage), and 4) the advent of recession. Can we predict the economic slowdown in corporate earnings (S&P 500 EPS) well in advance?

Ellis claims (based on observations) there is a 0-9 month lag between wages and its effect on consumer spending. 0-6 months until changes in consumer spending affects changes in industrial production. Another 6-12 months between industrial production and capital spending. And finally, another 6-12 between capital spending and its effects on Corporate Profits.

46

Three case studies (1/3)

1. Predicting Changes in Quarterly Corporate Earnings Using Economic Indicators Ellis model:

47

Three case studies (1/3)

1. Predicting Changes in Quarterly Corporate Earnings Using Economic Indicators

The data: i) 180 quarters. 6 [Economic] x variables. Ii) Change in S&P EPS = y variable, iii) All variables transformed to year vs year % change, iv( All data used is publicly available via websites of US agencies: BEA, BLS, FED, and S&P.

The analysis: XLMiner on these different versions of datasets. Partitioned it. Ran predictor applications: ACF Plots, MLR, Regression Tree – full and pruned.

48

Auto Correlation Chart. Based on this, took Lag_1 as one of the predictors. Lag_1 = QEPS_YY(Q-1)

Three case studies (1/3)

1. Predicting Changes in Quarterly Corporate Earnings Using Economic Indicators

49

QEPS_YY%(t) = 0.0486 + 0.747*QEPS_YY%(t-1) -0.517*QRCAP_YY%(t-2)

# Dimension Note Value Index

1 Data resolution quarterly data 2 0.2500

2 Data structure no externalities 3 0.5000

3 Data integration 4 0.7500

4 Temporal relevance 5 1.0000

5 Generalizability 5 1.0000

6 Chronology of data and goal quarterly data 3 0.5000

7 Concept operationalization 5 1.0000

8 Communication 4 0.7500

InfoQ Score = 0.66

InfoQ=66%

Three case studies (2/3)

2. Predicting ZILLOW.com’s Zestimate accuracy

50

“Zillow.com” is a real estate service launched in 2006

It calculates a Zestimate-home valuation for most homes in the U.S

For MD and VA it gets only about 26% of predictions within the +/-5% range only.

1.Home Type (Single Family, Condo , etc) 2.No of Bed Rooms 3.No of Bath Rooms 4.Total Area –Sqft 5.Lot size –Sqft 6.No of Stories 7.Total Rooms 8.Distance from Metro 9.Primary School Rank 10.Middle School Rank 11.High School Rank 12.Age of house at Sale 13.Sale Season (Fall , Winter , etc) 14.Recession Period (Y/N) 15.Sales Volume

Three case studies (2/3)

2. Predicting ZILLOW.com’s Zestimate accuracy

51

• Data collected, cleansed and merged from 4 sources –Zillow , Redfin, School Digger and Google Maps

• 17 counties (29 Zip codes) in Northern VA

House sales data • Before Data Clean up: 3500+ • After Data Clean up: 1416 • Y –Is Zestimate correct (Y/N)

37.6%/62.43% • X –15 variables (5+ variables

where discarded from initial set )

Three case studies (2/3)

2. Predicting ZILLOW.com’s Zestimate accuracy

52

# Dimension Note Value Index

1 Data resolution by individual house 5 1.0000

2 Data structure no externalities 4 0.7500

3 Data integration 5 1.0000

4 Temporal relevance 5 1.0000

5 Generalizability only VA counties 3 0.5000

6 Chronology of data and goal 5 1.0000

7 Concept operationalization 4 0.7500

8 Communication 4 0.7500

InfoQ Score = 0.82InfoQ=82%

Three case studies (2/3)

2. Predicting ZILLOW.com’s Zestimate accuracy

53 http://www.madlan.co.il/education/schools

The Israeli version……

Three case studies (3/3)

3. Predicting First Day Returns for Japanese IPOs

Goal: To predict the First Day returns on Japanese IPOs (based on first day closing price), using public information available prior to the offer

The data: i) Japanese IPO data from 1997-2009*, ii) 1561 IPOs, iii) Industry(categorical) : 35 industries - 3 were spelling errors, corrected

Remove Air Trans (1), Fishery & Forestry (2) industries

–Removed first 128 entries (1997-1999) as they had no data for 2 columns : Underwriter’s fees & Allocation to BRLM

–New Columns

Minimum bid size

Secondary Offering %age

–Creation of Dummy Variables

BRLMs – 3, on the basis of Gross proceeds of IPO

Industry – 4, binned by average return

Market – whether the IPO was OTC or not 54

*Kaneko and Pettway’s Japanese IPO Database (KP-JIPO) http://www.fbc.keio.ac.jp/~kaneko/KP-JIPO/top.htm

Three case studies (3/3)

3. Predicting First Day Returns for Japanese IPOs

55

1) Age of company at time of IPO 2) Gross Proceeds (size of IPO) 3) Minimum Bid Amount 4) IS_OTC listing 5) Secondary offering as %age of total 5) Percentage shares allocated to Lead Manager 1 7) Underwriter’s Gross Spread (fees as %age of size of IPO) 8) Industry_Type (binned categorical variable – 4 categories) 9) Lead_Manager (binned categorical variable – 3 categories)

# Dimension Note Value Index

1 Data resolution 5 1.0000

2 Data structure 4 0.7500

3 Data integration no externalities 2 0.2500

4 Temporal relevance 5 1.0000

5 Generalizability no theory 3 0.5000

6 Chronology of data and goal should be ex ante 3 0.5000

7 Concept operationalization 5 1.0000

8 Communication 4 0.7500

InfoQ Score = 0.66

Prediction algorithms do not give a reasonable prediction of IPO returns from public information. (High RMSE: 90%)

InfoQ=66%

Thank you for your attention

56