earthdata cloud analytics projectceos.org/document_management/working_groups/wgiss...2018/04/11 ·...
TRANSCRIPT
![Page 1: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/1.jpg)
Earthdata Cloud Analytics Project
Chris Lynnes* and Rahul Ramachandran*NASA
*U.S. Civil Servant
![Page 2: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/2.jpg)
2
Earth Observing System Data and Information System (EOSDIS)
Applications
capture and clean
data downlink
Education
process
archive
subset
distributeResearch
Users
EOSDIS
![Page 3: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/3.jpg)
3
Over time, EOSDIS archive volumes increase exponentially
projected
![Page 4: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/4.jpg)
4
Distribution increases similarly to cumulative volume
![Page 5: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/5.jpg)
5
How do we support user analysis of very large data volumes?
![Page 6: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/6.jpg)
6
Solution: Data-proximal Analysis
Data Archive
Data Analysis
![Page 7: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/7.jpg)
Goals
1. Enable big compute next to big data
2. Encourage user adoption of cloud for analytics
3. Maximum analytics capability at minimum costa. Use capabilities within NASA more effectively and efficiently
b. Leverage analytics capabilities of external partners
![Page 8: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/8.jpg)
Key Features1. Satisfy a diverse user community
2. Support analysis in the cloud without egressing data
3. Facilitate multi-dataset comparison and fusion
4. Support batch, interactive and streaming modes
5. Support distributed filesystems and databases
6. Support cost constraints and cost-sharing
![Page 9: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/9.jpg)
Earthdata Cloud Analytics Guiding Principles
1. Infusion- and innovation-friendly framework and building blocks
2. No monolithic systems
3. Open code and services
4. Interoperability and reuse
5. No unnecessary duplication (“undifferentiated heavy lifting”)
![Page 10: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/10.jpg)
Architectural Concept
Earth Science Data Analytics the Cloud-Native Way:
Everything is a Service
This approach produces key important benefits for
the user community and EOSDIS
![Page 11: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/11.jpg)
Abstract Analytics Workflow
ExtractTransform
Loaddata Analyze Visualize
![Page 12: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/12.jpg)
AODS1 as-a-service
Preprocessing as-a-serviceCumulus
Data Archive
Analysis as-a-service
Visualization as-a-service
Earthdata Cloud Analytics Reference Architecture
Extract Transform Load
1 Analytics Optimized Data Store
![Page 13: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/13.jpg)
Preprocessing as-a-serviceCumulus
AODS1 as-a-service
Analysis as-a-service
Visualization as-a-service
Interactive Mode: Analytics-Optimized Storage
Cumulus Data
Archive
1 Analytics Optimized Data Store
![Page 14: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/14.jpg)
Preprocessing as-a-serviceCumulus
Analysis as-a-service
Visualization as-a-service
Batch Mode
AODS1 as-a-service
Cumulus Data
Archive
1 Analytics Optimized Data Store
![Page 15: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/15.jpg)
Preprocessing as-a-serviceCumulus
Event Analytics as-a-service
Visualization as-a-service
Streaming Mode
Cumulus Data
Archive
![Page 16: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/16.jpg)
Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base
End-User-Specific Analysis
Preprocessing as-a-serviceCumulus
AODS1 as-a-service
Analysis as-a-service
Visualization as-a-serviceCumulus
Data Archive
1 Analytics Optimized Data Store
![Page 17: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/17.jpg)
AODS1 as-a-service
Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base
End-User-Specific Analysis
Preprocessing as-a-serviceCumulus
Analysis as-a-service
Visualization as-a-service
End-User Cloud-Native
Analysis
Cumulus Data
Archive
1 Analytics Optimized Data Store
![Page 18: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/18.jpg)
Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base
End-User-Specific Analysis
Preprocessing as-a-serviceCumulus
Analysis as-a-service
Visualization as-a-service
End-User Cloud-Native
Analysis
End-User Interpretation
Cumulus Data
Archive
AODS1 as-a-service
1 Analytics Optimized Data Store
![Page 19: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/19.jpg)
Open Pipeline Provides Outputs at Different Stages Appropriate for a Diverse User Base
End-User-Specific Analysis
Preprocessing as-a-serviceCumulus
Analysis as-a-service
Visualization as-a-service
End-User Cloud-Native
Analysis
Data Exploration
End-User Interpretation
Cumulus Data
Archive
End-User Interpretation
AODS1 as-a-service
1 Analytics Optimized Data Store
![Page 20: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/20.jpg)
Open Pipeline Enables Integration with Other Data, Scripts, and Workflows
End-User-Specific Analysis
Preprocessing as-a-serviceCumulus
Analysis as-a-service
Visualization as-a-service
End-User Cloud-Native
Analysis
Data Exploration
End-User Interpretation
Cumulus Data
Archive
End-User Interpretation
Data Exploration
AODS1 as-a-service
1 Analytics Optimized Data Store
![Page 21: Earthdata Cloud Analytics Projectceos.org/document_management/Working_Groups/WGISS...2018/04/11 · Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed8922b6714ca7f47682c10/html5/thumbnails/21.jpg)
Open Pipeline Enables Integration with Exploitation Platforms
Preprocessing as-a-serviceCumulus
AODS1 as-a-service
Analysis as-a-service
Visualization as-a-service
Exploitation Platforms
Cumulus Data
Archive
1 Analytics Optimized Data Store