noaa’s big data project: status updatenoaa’s data are increasingly popular and valuable. budgets...
TRANSCRIPT
![Page 1: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/1.jpg)
IOOS Advisory Committee Fall MeetingOctober 27, 2017 1
NOAA’s Big Data Project: Status Update
Shane GlassProject Manager, NOAA Big Data Project
![Page 2: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/2.jpg)
Due to increasein satellite andmodel data
Archive Storage Volume Increasing Over Time
Volu
me
(Pet
abyt
es)
2016 Total: 28.6 Petabytes
Why Open Data Partnerships?
Courtesy of NCEI/NESDIS2
![Page 3: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/3.jpg)
Why Open Data Partnerships?Data Dissemination Volume Increasing Over Time
Courtesy of NCEI/NESDIS3
![Page 4: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/4.jpg)
● NOAA’s data are increasingly popular and valuable.○ Budgets for capacity and security: Static○ Costs for data access support: Increasing
● Create new economic opportunities for the Nation ○ Democratize data access
Big Data Project Objective: Define enterprise business and
technical models for data-centric Public-Private Partnerships
Why Open Data Partnerships?
![Page 5: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/5.jpg)
BDP Specifics
Leverage value of NOAA’s data to increase their utilization 5
Service augmentation,
not replacement
Fair and LevelAccess
Data remains free and open
No Net Cost to Taxpayers
Leverages expertise of NOAA and
industry
Collaborators monetize services, not data
![Page 6: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/6.jpg)
Add Capacity
Augment
Amplify
BDP Data Access StrategyLeveraging Industrial Partners
Add Capabilities
Bring Processing to the Data
![Page 7: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/7.jpg)
One-way transfer out of federal systems. Only trusted users inside security boundary
Distributing a single copy of data can support all users.
BDP Distribution Scheme
7
A potential paradigm shift for providing data access
![Page 8: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/8.jpg)
Tangible BDP Benefits
NOAA DataEasier Access
NOAA Systems
Reduced Loads/Budget
Industry
New/Innovative Opportunities
BDP
![Page 9: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/9.jpg)
Increased 2.3XData Usage
NCEI Server Load
Increased Data UtilizationAWS and NEXRAD L2
Decreased 60%
![Page 10: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/10.jpg)
GCP and Historical Observation Data
NOAA NCEI: https://www.ncdc.noaa.gov/cdo-web/
Increased Data Utilization
![Page 11: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/11.jpg)
● Three datasets uploaded in 2017: ○ Global Summary of the Day (GSOD)○ Global Historical Climatology Network (GHCN)
● Early metrics are extremely positive○ 800,000 data requests between Jan and Apr 2017○ 1.2 PBs of data delivered○ Completely organic utilization
● More datasets available through Google Earth Engine
Increased Data UtilizationGCP and Historical Observation Data
![Page 12: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/12.jpg)
● International Comprehensive Ocean-Atmosphere Dataset (ICOADS)
Global marine data from ships and buoys…
...as far back as 1662
Video Credit: Kurt Schwer, UNHhttps://youtu.be/b_oaah7kdPE
GCP and Historical Observation Data
Increased Data Utilization
![Page 13: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/13.jpg)
Ongoing and Upcoming Efforts
National Water Model: ● 23-year reanalysis● Real-time forecast● Data transfer
underway
Provisional GOES-16:● Now: L1b ABI Products● Soon: L2 products (GLM)
National Water Center: http://water.noaa.gov/tools/nwm-image-viewer
NOAA NESDIS: https://www.nesdis.noaa.gov/content/flashy-first-images-arrive-noaa%E2%80%99s-goes-16-lightning-mapper
![Page 14: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/14.jpg)
14
● Target a problem instead of moving individual datasets○ Deliver multiple datasets and code to the platform to facilitate use
● Compute in “sandboxes” alongside NOAA (and others) data
● NOAA has approved the use of GitHub (NOAAGov)○ Guidance from NOAA CIO released in September
A New Approachto Enable Ocean Data
![Page 15: NOAA’s Big Data Project: Status UpdateNOAA’s data are increasingly popular and valuable. Budgets . for capacity and security: Static Costs. for data access support: Increasing](https://reader033.vdocument.in/reader033/viewer/2022052100/60399b65b69741796164a5f5/html5/thumbnails/15.jpg)
Way Ahead for the BDP
15
● Seek extension of CRADAs by one year (April 2019)
● Finalize Lessons Learned Document○ Captures information gathered in first 2.5 years
● Define requirements for any future agreement○ Actively engaging with NOAA offices to gather requirements○ Examine level of service and commitment requirements ○ Define the “Data Broker” role more clearly
● Determine the optimal partnership models options for the time period beyond the CRADAs