eu 2nd year review – 04-05 jan. 2003 – wp9 wp9 earth observation applications demonstration...
TRANSCRIPT
EU 2nd Year Review – 04-05 Jan. 2003 – WP9
WP9 Earth Observation Applications
Demonstration
Pedro Goncalves : [email protected]
Christine Leroy: [email protected]
http://documents.cern.ch/AGE/current/fullAgenda.php?ida=a021814
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 2
The EO WebMap Portal The scientist interacts with the GOME information system using the WebMap
portal
This allows the user to choose a geographic area and time interval to be investigated
The WebMap portal retrieves and displays the available GOME products on the map
It does this by communicating with several background services :
Web Map Server – provides map images
Web Service Broker – provides GRID web services
EO Grid Engine – integrates the Data Grid services
Products which were recently requested will be already available
Other products will be processed “on the fly” using the GRID resources
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 3
EO Grid Engine
Web PortalApplication
Web Service Interface
Earth Observation GRID Engine
Data Grid ServicesFabric & Resources
JDL Composition
JobExecution
Data Transfer &Replication
MetadataManagement
ArchiveManagement
Inte
rfac
eA
pplica
tion
GR
IDW
eb
Map
pin
gTech
nolo
gy
Web
Serv
ices
Tech
nolo
gy
GR
ID +
oth
er
Tech
nolo
gy
GRID Surfer(Desktop application)
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 4
Global Ozone (GOME) Data Processing and Validation
The DataGrid testbed provides a collaborative processing environment for 3 geographically distributed EO sites (Holland, France, Italy)
Level-1 data (raw satellite measurements) are analysed to retrieve the physical quantities : Ozone Profiles (Level-2 data)
Ozone Profiles contain measurements of OZONE within a vertical column of atmosphere at a given lat/lon location above the Earth’s surface
Two different Grid processing techniques are being investigated OPERA (Holland) - Tightly coupled - based on MPI NOPREGO (Italy) - Loosely coupled - based on Neural
Networks
Selected Profiles are validated by comparing them with ground-based LIDAR measurements in a coincident area and time
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 5
ESA(IT) – KNMI(NL)Processing of raw GOMEdata to ozone profiles.
2 alternative algorithms~28000 profiles/day IPSL(FR)
Validate some of theGOME ozone profiles (~106/y)Coincident in space and time
with Ground-Based measurements
Visualization & Analyze
EDG EO challenge: Processing / validation of 1y
of GOME data
LIDAR data (7 stations, 2.5MB per month)
DataGridenvironment
Level 2
(example of 1 day total O3)
Level 1
Raw satellite data from the GOME instrument(~75 GB - ~5000 orbits/y)
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 6
EO Use Case File Numbers
Dataset Number of files SizeLevel 1 4,724 15 Mb
Level 2NNO (ESA)
9,448,000 10 kb
Level 2Opera (KNMI)
9,448,000 12 kb
ValidationLidar (IPSL)
12 2.5 Mb
Total: 18,900,736 files 267 Gb
(just one part of a 5-year dataset)
1 Year of GOME data :
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 7
Overview
Demo Part 1 Ozone Profile Processing
Demo Part 2 Ozone Profile Validation
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 8
Part 1 - GOME Ozone Profile Processing
Select geographic area and time to be investigated
Retrieve list of available Level-1 data files from the Product Catalogue
Transfer files to the GRID Perform Level-1 Processing, retrieve Ozone Profiles
Visualize results
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 9
Web Portal EO ProductCatalogue
EDGStorage Element
EDGUser Interface
EDGResource
BrokerEDGComputing
Element
EO Replica Catalogue
Processing Sequence
EOGrid Engine
EO ProductArchive
1. Search Level-1 catalogue
2. Retrieve Level-2 products
3. Level-2 Products already registered in RC?
8. Submit jobs to process Level-1 data
7. Register Level-1 data
11. Register level-2 data
9. Process Level-1 data
10. Transfer Level-2 data to SE
12. Return new Level-2 products
Yes? 4. Return available Level-2 productsNo? 5. Perform GRID processing on-the-fly 6. Transfer Level-1
data from Archive to the Grid
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 10
Validation Processing Sequence
Level 2 Catalogue
Lidar data catalogue
Queries and data information retrieval from the Lidar metadata catalogue
GRID
ComputingElement
Storage Elements with Lidar data
Queries and data information retrieval from the Gome Level 2 orbit or pixel metadata catalogues
When completed comparison between lidar and satellite ozone profiles
Satellite data validation Lidar site
Level 2 Catalogue
(5/02/2003)
GRID Portal
Storage Elements with Gome L2 data
Submission of the Job in the GRID
1
2
3
4
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 11
Part 2 - GOME Ozone Profile Validation
Goals of the DataGrid applicationto point out : the capability to validate satellite data with all ground based data available in an easy way: Comparison of ozone profiles provided by satellite with lidar data in differents locations and times (see the web portal) Statistical comparison and analysis in order to improve algorithms.
OZONE LAYER50 km
10 km
ERS/GOME satellite
Lidar at the Haute Provence Observatory
(5/02/2003)
EU 2nd Year Review – 04-05 Feb. 2003 – WP9 – n° 12
Summary The Web Portal Integrates several components of the EO Infrastructure
The EO GRID Engine provides a standard way for Web Clients to request GRID operations
The Web Portal and the EO Grid Engine allow the complexity of the underlying components to be hidden from the user
This Framework can be adapted to provide new tools for the satellite end-users