how the weather company monetizes weather: the big data challenge (bdt208) | aws re:invent 2013
DESCRIPTION
(Presented by Basho) This session will discuss the transformation of the most widely distributed cable TV network in the United States, building on one of the world's most visited digital properties, to create a world class Big Data platform. Architects, CTOs, CIOs, IT Director, and development managers will learn how to run highly scalable analytics workloads on Amazon EC2 and Amazon EMR for complex, real-time analysis of large data sets. All while decreasing time to results and increasing business agility. Bryson Koehler, EVP & CIO of The Weather Company, will discuss architecture, technology choices, performance results and business benefits realized as part of their use of AWS services to host an exciting set of weather.com solutions and generate new revenue streams. Weather impacts over 30% of the global GDP daily and is the source of vast amounts of data collection. The Weather Company is the leader in weather forecasting and is bringing the world's most accurate forecasting capabilities alive in a full suite of data APIs built fully on Infrastructure as a Service platforms, including AWS and next generation products like Basho Riak, Hadoop, and Dasein. This session will discuss how the application of these technologies help keep people safe and helps businesses plan and become more profitable, thanks to the latest intersection of consumer behavior and weather forecasting and reporting.TRANSCRIPT
![Page 1: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/1.jpg)
How the Weather Company Monetizes Weather, the Original Big Data Problem Sathish Gaddipati, VP - Data Management, The Weather Company Raja Selvaraj, Manager - Data and System Engineering, The Weather Company
November 14, 2013
![Page 2: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/2.jpg)
WHERE THE WORLD GETS ITS WEATHER
#1 MOST DISTRIBUTED Cable
Network*
100M+ App Downloads* #2 All Time Most Downloaded iPad App**
47.2M Unduplicated Monthly Uniques***
61.6M Unduplicated Monthly Uniques
(weather.com + wunderground)*
72% visit 2x or more Daily**
TV
DIRECT CONSUMER
ONLINE & APPS
B 2 B 2 C
B 2 B
GOV’T PRODUCTS
*Nielsen, October 2012.
*comScore, Oct 2012 **Omniture, July 2012
*Various app stores, TWC only, **App Store Rewind, 2011 ***Omniture Site Catalyst, August 2012. TWC Mobile & wunderground
![Page 3: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/3.jpg)
Did You Know? TWC Weather Services for Businesses World’s Leading Weather Media Company • - Power 500+ broadcast channels world-wide
World’s Leading Aviation Weather Service • - Support 50,000+ flights daily with embedded
forecasters, systems and tools • - 85% of major US airlines. 30% of Top 100
global.
World’s Leading Energy Forecaster • - Over 300 power trading clients worldwide • - Hourly, daily, and seasonal trading
Fastest Growing Insurance Weather Service • - 7 of the top 10 P&C insurance customers • - Connecting Marketing, Underwriting, & Claims
with systems & tools
![Page 4: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/4.jpg)
WEATHER IS CHANGING • 2012 was the hottest in recorded
history for the US • Last time January was colder than
normal was 1983 • In July 2012 61% of the country was in
drought conditions • 9.2MM acres charred in 2012; third
highest in 13 years
Source: Time.com; Climate Central
![Page 5: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/5.jpg)
Weather changes… So must IT • In the next 12 months we will:
• Increase our data points by 1,400% • Increase our data velocity 300% • Increase our number of deployments
by 2,000% • Increase our number of TV feeds
from 1 to 7 • Increase the amount of video content
by 100% • Increase the amount of social
content by 500%
5
• 13 data centers to 4 • 75% of workloads to IaaS by
building a cloud abstraction toolkit
• Move to agile development teams
• New TV Transmission System (retire 2, build 1)
• 575 APIs and 20 data platforms to 1 platform
• 2500+ new localization servers • New HR, Email, Finance,
Inventory, Expense, etc. • New Ad Stack • New Video Asset
Management Huge
pro
ject
s req
uire
d to
supp
ort t
hese
bu
sines
s nee
ds
![Page 6: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/6.jpg)
The Weather Company’s Core
Science
Data Technology
![Page 7: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/7.jpg)
WDS
Insurance
Retail
Weather Data Services - Use Cases
Tornado and flood forecasts Weather warnings Historical weather trends
Wind Energy
Wind speed forecast Historical wind speeds
Precipitation forecast Temperature forecast Extreme weather forecast
Max. Premium Rate Min. Claims
PPC Maintenance
Inventory management Distribution
![Page 8: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/8.jpg)
WDS
Media
Ad Targeting
Hourly forecasts Daily forecast Current conditions
Temperature forecast Historical trends
Real-time conditions Forecasts Customer location
Weather data Weather content
Commodity bidding Demand forecasting
Impression serving Improved Ad. Exchange
Energy Exchange
Weather Data Service - Use Cases
![Page 9: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/9.jpg)
WDS
Mobile Apps
Hospitality
Hourly forecasts Daily forecast Current conditions
Weather forecast Historical forecast Watches and warnings
Current conditions Forecasts Airline delays/ cancellations
Weather data Weather content
Local weather National weather
Room rates Revenue optimization
Gov
Weather Data Service - Use Cases
![Page 10: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/10.jpg)
WDS
Hourly forecasts Daily forecast Current conditions Historical trends
Historical data Consumer spend vs. weather conditions
Air turbulence and wind speeds Weather forecasts Current conditions
Weather data Weather content
Business impact Consumer behavior
Optimal Routes Flight Schedules
O&O Products
Weather Analytics
Airlines
Weather Data Service - Use Cases
![Page 11: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/11.jpg)
11
ACCURACY THROUGH SUPERIOR DATA
Global Lightning Detection Network
Proprietary Radar
Algorithms
Industry Best Forecast Modeling
State-of-the-Science Forecast
Technologies PhD 220+ Full-time
Meteorologists
Robust weather data APIs
Largest collection of worldwide forecast data
Proprietary weather data analytics
APIs Network of 30,000+ Weather Sensors
Number 1 in forecasting accuracy per ForecastWatch in the US and Europe for
the last 6 months… 5 points better than NWS Forecast
![Page 12: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/12.jpg)
We defined an architecture that could support the needs of all of our divisions. • API interfaces • IaaS abstraction • Shardable data • Global scale
![Page 13: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/13.jpg)
1. Reduce time to deploy and market new data sets
2. Reduce operating cost of data services
3. Centralize data services across the company
4. Provide visibility of data access
5. Provides data governance process and framework
6. Serves world’s best weather forecast across all products
7. Low latency, highly scalable APIs
8. Secured access to data
9. Centralized and scalable architecture
10. Consistent “rich” content across platforms
Top 10 Objectives of Weather Data Services Platform
![Page 14: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/14.jpg)
Weather Data Services Platform
![Page 15: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/15.jpg)
1. Distribute thousands of gridded binary files to multiple locations across globe within 5 minutes
2. Serve more than billion data services API requests/day
3. Metering and authentication of API calls with low latency
4. Process multiple TBs of data every day
5. Ensure business continuity
6. Leverage data caching
7. Store petabytes of historical data
8. Meshing weather data with consumer behavior and derive analytics
9. Build flexible data ingestion platform to manage 100s of data feeds from external parties
10.Maintain above systems within OPEX budget
Top 10 Computing Challenges
![Page 16: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/16.jpg)
Data Store
Manageability
Availability
Performance
Replication
Capacity
DATA STORES – WHAT WE WANTED
![Page 17: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/17.jpg)
Historical weather data
Raster/Image data
Analytics platform
Severe weather data
DATA STORES – WHAT WE USE
![Page 18: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/18.jpg)
Public S
ubnet
AZ1
AZ2
Pub
lic
Sub
net
Priv
ate
Sub
net
AZ2
Private S
ubnet
AZ1
VPC VPC
Amazon – Region 1 Amazon – Region 2
RIAK REPLICATION @AWS
![Page 19: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/19.jpg)
NEXT GENERATION – WEATHER FORECASTING SYSTEMS
![Page 20: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/20.jpg)
Amazon Redshift
WeatherFX
CreativeFX
AudienceFX
App N
… TWC
Private Data
Center EDW
Partner EDW
Platform
Amazon Redshift
Amazon S3
TWC’s Redshift Cluster now contains: 600+ million records of weather forecast data since March 2005 700+ million records of current conditions observation data since April 2004 11+ billion records of Int. Civil Aviation Org. (ICAO) observation data Since 1970 TBs of storm event data from from 1950 - 2012 Weather sentiment survey data
Weather’s Big Data Analytics Platform for Enterprises and Government
~1 Billion records Bulk loaded in 36
minutes!
![Page 21: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/21.jpg)
Contact Us Sathish Gaddipati – [email protected] Raja Selvaraj – [email protected]
![Page 22: How The Weather Company Monetizes Weather: The Big Data Challenge (BDT208) | AWS re:Invent 2013](https://reader031.vdocument.in/reader031/viewer/2022020105/5484490e5806b590588b45fd/html5/thumbnails/22.jpg)
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
BDT208