![Page 1: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/1.jpg)
NASA Data Center Management Using the MMT (Metadata Management Tool)
Katie Baynes, EOSDIS System Architect, [email protected]
![Page 2: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/2.jpg)
2
Talk Outline- EOSDIS Overview (5 min)- Systems Overview (5 min)- Metadata in Context (5 min)- Metadata Management Tool (20 min)
- Overview/Objectives- Demo
- Ongoing Curation Activities/Tools (10 min)- Opportunities for Growth (5 min)
- Bulk Updates- Provider Specific Validations- Landing Pages
- Questions (5 min)
![Page 3: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/3.jpg)
EOSDIS Overview
![Page 4: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/4.jpg)
![Page 5: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/5.jpg)
5
Putting EOSDIS in Context
EOSDIS
Applications
Capture and Clean
Education
Process
Archive
Transform*
Distribute
Research
*Subset, reformat, reproject
![Page 6: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/6.jpg)
6
ASF DAAC SAR Products, Sea Ice, Polar Processes
PO.DAAC Ocean CirculationAir-Sea Interactions
NSIDC DAAC Cryosphere, Polar Processes
LPDAAC Land Processes and Features
GHRC Hydrological Cycle and Severe Weather
ORNLBiogeochemical Dynamics, EOS Land Validation
ASDCRadiation Budget, Clouds, Aerosols, Tropo Composition
LAADS/MODAPS Atmosphere
OB.DAAC Ocean Biology and Biogeochemistry
SEDAC Human Interactions in Global Change
CDDISCrustal Dynamics Solid Earth
GES DISCAtmos Composition & Dynamics, Global Modeling, Hydrology, Radiance
NCAR, U. of Co.MOPITT
JPLMLS, TES, SNPP
Sounder
U. of Wisc.SNPP
Atmosphere
GHRCAMSR-U,
LIS
GSFCSNPP,
MODIS, OMI, OBPG
SIPS DAAC
Distributed Active Archive Centers (DAACs), collocated with centers of science discipline expertise,
archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPS)
![Page 7: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/7.jpg)
7
![Page 8: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/8.jpg)
8
Common Metadata RepositoryLightning fast, always available
- 95% queries complete in <1s - 99.98% uptime (last 365d)
Big Data Ready- 34K collections- 367 million files indexed- Prepared to scale 1B+ records
Standards-focused- ISO-19115 metadata- OpenSearch/OGC CSW- REST based APIs
Community-focused- Developer’s portal- Active Developer’s forum- Ecosystem of supported tools
Internationally Recognized- Provides the backbone of the
Community of Earth Observing Satellites International Directory Network (CEOS IDN)
![Page 9: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/9.jpg)
9
Data Centric End Usershttps://search.earthdata.nasa.gov
Imagery Centric End Usershttps://worldview.earthdata.nasa.gov
![Page 10: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/10.jpg)
Metadata In Context
![Page 11: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/11.jpg)
11
What’s so important about metadata?
EOSDIS
process
archive
transform
distribute
![Page 12: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/12.jpg)
Converging Problems- Combining two legacy metadata catalogs
- GCMD (30k collections, not all EOSDIS)- ECHO (6500 collections, 260+ million granules, all EOSDIS )- 12 EOSDIS DAACs to reconcile (that’s a whole different
presentation)
- New Missions requiring ISO 19115 compliant metadata
- Bottom Line: Several new metadata formats required support while continuing to support legacy requirements
12
![Page 13: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/13.jpg)
Developing a Model
![Page 14: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/14.jpg)
14
Crosswalking Each Element
![Page 15: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/15.jpg)
15
ECHO10 NASA ISO
DIF
Unified Metadata
Model
![Page 16: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/16.jpg)
![Page 17: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/17.jpg)
17
Ongoing Update of the UMM“The EOSDIS community may contribute requests for updates, changes, clarifications, and deprecations of elements
in a UMM Profile, which are then tracked as part of NASA’s Metadata Quality effort. Requests for modifications are documented and an impact analysis is conducted. Recommendations along with their supporting documentation are then put before a UMM review board consisting of EOSDIS community members and the ESDIS Standards Office (ESO). After a review and commenting period, the UMM Profile is updated to reflect approved changes and a new version is released.”
https://go.nasa.gov/2qd0Af3
community inputs
review, impact analysis,
recommendations
community review and feedback
implementation, release,
documentation updates
![Page 18: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/18.jpg)
Metadata Management Tool
![Page 19: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/19.jpg)
Metadata Management Tool Original Objectives
❏ Intended to provide easy, intuitive
editing of metadata without xml editor
❏ Templates for new products❏ Export in various formats❏ Closely tied to UMM and controlled
vocabularies from GCMD❏ Allows access management of
metadata catalog entries❏ Scoring of metadata against specified
criteria❏ Open Source development
❏ https://github.com/nasa/mmt
![Page 20: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/20.jpg)
Demo
![Page 21: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/21.jpg)
21
Curationactivities
![Page 22: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/22.jpg)
22
So what’s the problem?
• Links die• Information gets out of date • Usage is inconsistent• Keywords change• Fields get added• And on and on and on...
![Page 23: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/23.jpg)
23
ISO TOO!
![Page 24: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/24.jpg)
24
Metadata Review (Now)
![Page 25: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/25.jpg)
25
New Tool
![Page 26: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/26.jpg)
26
Coming Soon: Metadata Review Tool
![Page 27: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/27.jpg)
27
Summary• EOSDIS has a large amount of metadata, from both NASA
and non-NASA sources• Metadata dialects are mapped via the unified metadata
model (UMM) which is updated annually via a EOSDIS wide review process
• The metadata management tool is intended to provide a simplified interface allowing for metadata editing and access control
• Curation activities for EOSDIS collections is ongoing• A Metadata review tool and dashboard is being developed
to aid in those curation activities and will eventually be integrated into the MMT.
![Page 28: Management Using the MMT NASA Data Center (Metadata … · 2017-06-12 · 8 Common Metadata Repository Lightning fast, always available - 95% queries complete in](https://reader034.vdocument.in/reader034/viewer/2022042803/5f4d77f32c401834cb2d9c98/html5/thumbnails/28.jpg)
NASA EOSDIS Metadata Future
- Metadata Bulk Updates- Provider Specific Validations- Ongoing governance of UMM- Ongoing curation- Evolution of UMM Profiles to meet
evolving needs (services, visualizations, etc)