a data access framework for esmf model outputs roland schweitzer steve hankin jonathan callahan...
TRANSCRIPT
A Data Access Framework A Data Access Framework for ESMF Model Outputsfor ESMF Model Outputs
Roland SchweitzerRoland Schweitzer Steve HankinSteve HankinJonathan CallahanJonathan Callahan
Kevin O’BrienKevin O’BrienAnsley MankeAnsley Manke
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 22
Some MotivationSome Motivation
Model Components produce … Model Components produce … Model Runs which produce … Model Runs which produce … Data and Metadata whichData and Metadata which should/could produce …should/could produce … A data portal which allows user to:A data portal which allows user to:
Search, browse, compare, validate and Search, browse, compare, validate and analyze results …analyze results …
which in turn helps produce insightwhich in turn helps produce insight
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 33
Some More MotivationSome More Motivation
ESMF and the ESCurator manage ESMF and the ESCurator manage components, runs and outputcomponents, runs and output
LAS is a candidate to provide a data LAS is a candidate to provide a data portalportal
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 44
The Live Access Server (LAS)A highly configurable Web server designed to provide
flexible access to geo-referenced scientific data
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 55
LASLAS LAS is an information broker that:LAS is an information broker that:
• ingests descriptions of dataingests descriptions of data data setsdata sets variablesvariables (optional reshuffling of above into new categories) (optional reshuffling of above into new categories) domains: 1d-4ddomains: 1d-4d special constraints (ensemble index)special constraints (ensemble index) rendering hints (smoothing, palettes)rendering hints (smoothing, palettes)
• presents the descriptions to the user in a presents the descriptions to the user in a coherent fashioncoherent fashion
• asks other software to perform tasks for the asks other software to perform tasks for the useruser
• and gives the results to the user and gives the results to the user
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 66
Architecture ComponentsArchitecture Components
User Interface Server (UIS)
ProductServer (PS)
Backend Server (BES)
FerretData
(OPeNDAP)Server(FDS)
Backend Server (BES)
Data
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 77
LAS OffersLAS Offers
Independence from underlying dataIndependence from underlying data Uniform interface for browse Uniform interface for browse
(visualize), compare, validate and (visualize), compare, validate and analyze results for many data typesanalyze results for many data types
Easy way to make connections to Easy way to make connections to metadata and documentation metadata and documentation
Customizable user interfaceCustomizable user interface
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 88
Example (Multiple Hierarchies)Example (Multiple Hierarchies)
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 99
Example (Collapse Hierarchies)Example (Collapse Hierarchies)
Experimental Development Server
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1010
Example (Custom Headers)Example (Custom Headers)
The NVODS Server (http://www.ferret.noaa.gov/nvods)
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1111
Example (Adopt a Drifter)Example (Adopt a Drifter)
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1212
Return to the MotivationReturn to the Motivation
Model Components produce … Model Components produce … Model Runs which produce … Model Runs which produce … Data and Metadata whichData and Metadata which should/could produce …should/could produce … A data portal which allows user to:A data portal which allows user to:
Search, browse, compare, validate and Search, browse, compare, validate and analyze results …analyze results …
which in turn helps produce insightwhich in turn helps produce insight
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1313
Data BrowsingData Browsing
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1414
Analysis and ComparisonAnalysis and Comparison
Compute Compute differencesdifferences
Can define new Can define new variables that are variables that are the result of some the result of some analysis operation analysis operation (done in delayed (done in delayed mode so only data mode so only data that is needed is that is needed is computed) computed)
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1515
Analysis and ComparisonAnalysis and Comparison
Can use FDS to define regriddings (or Can use FDS to define regriddings (or other analysis) to allow more complex other analysis) to allow more complex comparisonscomparisons
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1616
ValidationValidation
Use FDS to create samplings of in Use FDS to create samplings of in situ observations which can be situ observations which can be compared to model datacompared to model data
(This is a planned for a future (This is a planned for a future release.)release.)
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1717
Detailed AnalysisDetailed Analysis
Can jump to local tool (Matlab, IDL, Can jump to local tool (Matlab, IDL, Ferret, etc.) by requesting scripts for Ferret, etc.) by requesting scripts for that application that application
Scripts encode access to remote LAS Scripts encode access to remote LAS data via FDSdata via FDS
Can take advantage of FDS analysis Can take advantage of FDS analysis to compute some data reductions to compute some data reductions and transfer only the result to the and transfer only the result to the local applicationlocal application
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1818
Additional Flexibility beyond Additional Flexibility beyond ConfigurationConfiguration
Use LAS framework with different Use LAS framework with different backend engine (new Product Server backend engine (new Product Server is designed to make this as easy as is designed to make this as easy as possible)possible)
Web-services interfacesWeb-services interfaces• To Product Server from other clientsTo Product Server from other clients• Will allow implementation of “well-Will allow implementation of “well-
known” service like WMSknown” service like WMS
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 1919
Architecture ComponentsArchitecture Components
User Interface Server (UIS)
ProductServer (PS)
Backend Server (BES)
FerretData
(OPeNDAP)Server(FDS)
Backend Server (BES)
Data
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2020
Access Control and the GRIDAccess Control and the GRID
The new architecture makes participating as a The new architecture makes participating as a GRID partner easierGRID partner easier
Web services-based authentication schemes can Web services-based authentication schemes can be used by the Product Server and Backend be used by the Product Server and Backend Server to get authentication information (if not Server to get authentication information (if not supplied by a browser certificate).supplied by a browser certificate).
Implemented as another “filter” in a already-Implemented as another “filter” in a already-existing mechanism for filtering on IP-address, existing mechanism for filtering on IP-address, number of accesses, etc.number of accesses, etc.
Configuration metadata can be augmented to Configuration metadata can be augmented to identify allowed access “roles” by dataset and/or identify allowed access “roles” by dataset and/or variable (and maybe by output product).variable (and maybe by output product).
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2121
Return to the MotivationReturn to the Motivation
Model Components produce … Model Components produce … Model Runs which produce … Model Runs which produce … Data and Metadata whichData and Metadata which should/could produce …should/could produce … A data portal which allows user to:A data portal which allows user to:
Search, browse, compare, validate and Search, browse, compare, validate and analyze results …analyze results …
which in turn helps produce insightwhich in turn helps produce insight
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2222
More InformationMore Information
www.ferret.noaa.govwww.ferret.noaa.gov Subscribe to the LAS User’s mailing Subscribe to the LAS User’s mailing
list list
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2323
The EndThe End
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2424
browser
UI (java/script)
gif,netCDF,etc.
servers
UIS
PS
LAS XML configuration metadata
browserinterface
BES
HTTP Request(later SOAP)
scripts &/orcommands
products
HTTP Response
data
LAS Architecture with OPeNDAPLAS Architecture with OPeNDAP
LegacyOPeNDAP
OPeNDAP Server
browser
OPeNDAP Server
BESBES
LegacyOPeNDAP
UIS = User Interface Server
PS = Product Server
BES = Backend Server
FDS = Ferret Data Server
SOAP
addXMLFDS
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2525
Architecture ComponentsArchitecture Components
User Interface Server (UIS) – keeps User Interface Server (UIS) – keeps browser up-to-date with relevant browser up-to-date with relevant interface widgetsinterface widgets
Product Server (PS) – takes users Product Server (PS) – takes users requests and returns resultsrequests and returns results
Backend Server (BES) – runs legacy Backend Server (BES) – runs legacy application to fulfill requestapplication to fulfill request
Ferret Data Server (FDS) – General Ferret Data Server (FDS) – General OPeNDAP server for LAS dataOPeNDAP server for LAS data
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2626
The UI Server ArchitectureThe UI Server Architecture
User Interface ServerUser Interface Server• Installation process translates LAS Installation process translates LAS
configuration into User Interface Widgets need configuration into User Interface Widgets need to present datasets, variables, geo-spatial and to present datasets, variables, geo-spatial and time extents in HTMLtime extents in HTML
• Allows dataset collection to be easily organized Allows dataset collection to be easily organized into any hierarchies the installer choosesinto any hierarchies the installer chooses
• Communicates requests to the Product Server Communicates requests to the Product Server and returns responses (images, files, data and returns responses (images, files, data tables, etc.) to the Web Browsertables, etc.) to the Web Browser
• Many other customization featuresMany other customization features
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2727
The Product ServerThe Product Server Product ServerProduct Server
• Has knowledge of the LAS configurationHas knowledge of the LAS configuration• Makes requests from one or more Back-end ServersMakes requests from one or more Back-end Servers• Fully implemented Web-services interface will be Fully implemented Web-services interface will be
available to client writersavailable to client writers• Can interact with a series of Backend-servers to fulfill a Can interact with a series of Backend-servers to fulfill a
single request from the user client single request from the user client Back-end serverBack-end server
• Can produce product itself Can produce product itself oror
• Interact with a single legacy application (like CDAT or Interact with a single legacy application (like CDAT or Ferret) to produce productFerret) to produce product
Requests are communicated via Request and Requests are communicated via Request and Response XML that is exchanged via SOAP.Response XML that is exchanged via SOAP.
20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 2828
FDS (Ferret Data Server)FDS (Ferret Data Server)
FDS is an OPeNDAP serverFDS is an OPeNDAP server Automatically installed with LASAutomatically installed with LAS Therefore every gridded LAS dataset Therefore every gridded LAS dataset
is served via OPeNDAPis served via OPeNDAP Has extended analysis capabilities Has extended analysis capabilities
beyond basic OPeNDAP serversbeyond basic OPeNDAP servers