tools for google data · 12.04.2013 · plot of [weight loss] and [best vacation spots] new year....
TRANSCRIPT
![Page 1: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/1.jpg)
1
Tools for Google Data
Hal VarianKauffman Blogger Conference
April 2013
![Page 2: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/2.jpg)
2
Google Trends
Google Correlate
Google Consumer Surveys
![Page 3: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/3.jpg)
Searches for [hangover]
Which day of the week are there the most searches for [hangover]?
1: Sunday
2: Monday
3: Tuesday
4: Wednesday
5: Thursday
6: Friday
7: Saturday
![Page 4: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/4.jpg)
Search index for [hangover]
![Page 5: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/5.jpg)
Hangover geo
![Page 6: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/6.jpg)
Hangover-vodka time series
![Page 7: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/7.jpg)
Searches for [civil war]
![Page 8: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/8.jpg)
Searches for [term paper]
![Page 9: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/9.jpg)
Gift for boyfriend v Gift for girlfriend
Forboyfriend
For girlfriend
![Page 10: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/10.jpg)
Gift for husband v Gift for wife
Forhusband
For wife
![Page 11: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/11.jpg)
11
Google Trends
Google Correlate
Google Consumer Surveys
![Page 12: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/12.jpg)
Searches correlated with [weight loss]
![Page 13: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/13.jpg)
Plot of [weight loss] and [best vacation spots]
New Year
![Page 14: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/14.jpg)
Correlated with [weight loss] 3 weeks later
![Page 15: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/15.jpg)
15
Unemployment
![Page 16: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/16.jpg)
Initial claims: good leading indicator for recessions
Grey bars indicate recessions
![Page 17: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/17.jpg)
Google Correlate with initial claims data
![Page 18: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/18.jpg)
Initial claims and [unemployment filing]
![Page 19: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/19.jpg)
Regression models
Baseline model yt = a yt-1 + c + et gives an in-sample MAE of 3.1%
Adding the “filing for unemployment” query yt = a yt-1 + b qt + c + et gives an in-sample MAE of 3.0%
Train using t weeks, forecast t+1 (rolling window forecast)MAE of baseline = 3.2%, MAE with query = 3.2%, 0% improvement
During recession MAE of baseline = 3.7%, MAE with query = 3.3%, 8.7% improvement
![Page 20: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/20.jpg)
Weekly pattern for [unemployment office]
![Page 21: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/21.jpg)
Gun sales
![Page 22: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/22.jpg)
FBI gun sales background check
![Page 23: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/23.jpg)
NICS time series
[stack on] has highest raw correlation[gun shops] is chosen by statistical model
![Page 24: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/24.jpg)
Trend
![Page 25: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/25.jpg)
Seasonal
![Page 26: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/26.jpg)
[gun shops]
![Page 27: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/27.jpg)
Searches on [gun shop]
![Page 28: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/28.jpg)
28
Google Trends
Google Correlate
Google Consumer Surveys
![Page 29: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/29.jpg)
29
How it works
![Page 30: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/30.jpg)
30
![Page 31: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/31.jpg)
31
![Page 32: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/32.jpg)
32
![Page 33: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/33.jpg)
33
b
![Page 34: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/34.jpg)
34
How this changes surveys
Anyone can do them
The cost is dramatically lower
Results come back in a few hours
Surveys can be replicated … or not
You can measure sensitivity wording
![Page 35: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/35.jpg)
35
Consumer Sentiment
![Page 36: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/36.jpg)
Consumer sentiment and retail sales
![Page 37: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/37.jpg)
Methodology
Time series = trend + seasonal + regression + error
Find the best predictors for regression
Fat regression problem (aka p > n)
Sometimes get good fit by chance alone
Want predictors that work out of sample
Does regression provide incremental value over trend+seasonal?
![Page 38: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/38.jpg)
BSTS: Bayesian Structural Time Series
Estimating time series: use Kalman filter techniquesExpress time series as trend + seasonal + noise (“basic structural model”)
Forecast univariate model using Kalman filter
Advantages: flexibility, adaptive, interpretable, handles non-stationarity well
Model selection using “spike and slab” Bayesian regressionSpike: prior probability that coefficient is included in regression
Slab: diffuse prior for coefficient, conditional on inclusion
Estimate a posterior probability that variable is in model
Combines well with Kalman techniques
Final forecast is weighted average of many models, with weights given by posterior probabilities (Bayesian model averaging)
Example of “ensemble estimation”
Agnostic with respect to “true model”
Tends to avoid overfitting by avoiding choice of “best” single model
![Page 39: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/39.jpg)
Probability of inclusion of predictor (n=98, k= 195)
White: procyclicalBlack: countercyclical
![Page 40: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/40.jpg)
Start with “trend”
![Page 41: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/41.jpg)
Start with “trend”
![Page 42: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/42.jpg)
Add “financial planning”
![Page 43: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/43.jpg)
Add “Investing”
![Page 44: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/44.jpg)
Add “Business News”
![Page 45: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/45.jpg)
Add “Search Engines”
![Page 46: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/46.jpg)
Add Energy and Utilities
![Page 47: Tools for Google Data · 12.04.2013 · Plot of [weight loss] and [best vacation spots] New Year. Correlated with [weight loss] 3 weeks later. 1 5 Unemployment. Initial claims: good](https://reader036.vdocument.in/reader036/viewer/2022081617/60368b4e1f90912cdd34c615/html5/thumbnails/47.jpg)
Challenges for the future
Private sector has high-frequency, real time data and a lot of it!
Visa, Mastercard, American Express
UPS and FedEx
Wal-Mart, Target, etc
Supermarket scanner data
Search engines
Government agencies
Long historical series, but usually low frequency
Carefully constructed but labor intensive, with delayed release and periodic revisions
How to combine the public and private data?
How to integrate massive amounts of private sector real-time information with traditional government statistics