draft april 28, 2005 esip aq cluster, [email protected] current air quality information...

DRAFT April 28, 2005 ESIP AQ Cluster, [email protected]

Current Air Quality Information ‘Ecosystem’ (Draft for Feedback)

AQ information includes emissions, ambient & satellite data and model outputs

The distributed data are produced and provided by agencies, mostly through portals

Providers have different access protocols, formats, and information usage conditions

This lack of interoperability causes the under-utilization of the rich data resources


Future Integrated AQ information System (Draft for Feedback)

DataMart

VIEWS

NEISGEI

AIRNow

AQMod

DAACs

ASOS

NEI

Emission

IDEA

GASP

Missions

WeaMod

Forecast

GloMod

FireInv

Data Federation Distributed, Virtual, Uniform

AQ Forecasting

AQ Compliance

Status and Trends

Network Assess.

Data Processing Filtering, Aggregation, Fusion

Info Products Reports, Websites

Data are maintained by custodians and exposed through ‘portals’ Mediators uniformly ‘wrap’ data and provide processing servicesAnalysts program the services to create application-specific productsResponsibility is shared among data providers and mediator/ integratorsESIPFed can provide the infrastructure and tools for the AQ info system

Mediators


Air Quality Information Providers

• AQ information includes emissions, ambient (surface) and satellite data and model outputs • The information is provided by multiple Agencies, have different form and is• AQ data usage requires considerable processing and integrating

EmissionAmbientSatelliteModel

Form | ContentNOAA

GASP

NASA

DAACs

NASA

IDEA

NASA

Missions

EPA

NEI

EPA

NEISGEINOAA

FireInv

State/Local

Emission

NOAA

ASOSRPO

VIEWS

EPA

AIRNow

EPA-AQS

DataMart

NOAA

WeaMod

EPA

AQModel

NASA

GloModel

NOAA

Forecast


DataFed WS Output Data Types

• UrlGranuleType• TimePointType• TimeDimensionType• MapVectorType• MapTrajectoryType• MapTimePointType• MapPointType• MapLocationTableType• MapImageLatLonType• MapGridType• ImageType• HtmlType• DotNetTableType• DataSetType


Interagency Working Group for Earth Obs. (IWGEO) Global Earth Observing System of Systems (GEOSS)

Decision Support Tools

Assessments

Decision Support Systems

Decision Support Tools

Assessments


Decision Support

Assessments


ManagementDecisions

PolicyDecisions

Societal BenefitsHigh PerformanceComputing,Communication,& Visualization

Standards &Interoperability

Predictions

Observations

Monitoring & Measurements

remotely-sensedin situ

Earth Science Models• Oceans• Ice• Land• Atmosphere• Solid Earth• Biosphere

Monitoring & Measurements

remotely-sensedin situ

Earth Observation Systems

• Remotely-sensed• In situ

Earth Science Models• Oceans• Ice• Land• Atmosphere• Solid Earth• Biosphere

Earth System Models• Oceans• Ice• Land• Atmosphere• Solid Earth• Biosphere

DATA

On-going feedback to optimize value and reduce gaps

T. Karl, NOAA, NCDC

http://www.esipfed.org/business/library/meetings/13th_fed_meeting/docs/TomKarl.ppt

Integrated Observing Systems

OBSERVING SYSTEM TIMELINE

21st CenturyAtmospheric Observations

Data Systems

Technology Development

Innovations

Breakthrough

Efficiencies Cost

Mass Productions

Space Observations

Ocean Observations

Innovations

BreakthroughEfficiencies

Cost

Mass Productions

66

T. Karl, NOAA, NCDC

http://www.esipfed.org/business/library/meetings/13th_fed_meeting/docs/TomKarl.ppt


Tools for Users

• Pare down large file sizes of high resolution data and products.

• (re-) Group different data sets to create needed products – such as initialization files for model development, analysis, and intercomparison.

• Subset the data: – in parameter space

– in physical space

– in temporal space


Collaborations: How do we get there?Collaborations: How do we get there?

• Data transport is being actively pursued: OPeNDAP, SOAP, ...

• Earth System Partners need to be able to find and use various data sets, wherever they may be, whatever format...

• THREDDS can provide dynamic access and generate catalogs

• GCMD is a major resource for metadata management for the entire GeoSciences community- this activity must evolve!

• Ontology projects such as SWEET in conjunction with THREDDS and GCMD can provide individual data sources, data variables and metadata management for the community.

G. Rutledge: Emerging Tools for Distributed Data Access and Collaborations

•Data systems based on the integration of independently developed system elements offer many more opportunities than more traditional centrally developed ones.•P. Cornillon

http://www.esipfed.org/business/library/meetings/13th_fed_meeting/docs/Rutledge_Keynote_04_final.ppt

http://www.esipfed.org/business/library/meetings/13th_fed_meeting/docs/Rutledge_Keynote_04_final.ppt


Tools & Data

Information Systems

ProfessionalProductivity

InformationManagement

SharedServices

. . . Becoming More Intelligent And Distributed. . . Becoming More Intelligent And Distributed

Web Services Networks

GIS is Evolving to a Web Services EnvironmentGIS is Evolving to a Web Services Environment


GIS Networks Will Allow Us to Connect andIntegrate Distributed GIS Resources

. . . Making Virtual Collaborations Possible. . . Making Virtual Collaborations Possible

MapsMapsModelsModels

GeoDataGeoDataSetsSets

Peer-to-Peer GISPeer-to-Peer GIS

MetadataMetadata

Data ModelsData Models


PervasivePervasiveComputingComputing

Terabyte/Second CommunicationsTerabyte/Second Communications

• Faster HardwareFaster Hardware• Distributed ComputingDistributed Computing• Mobile/WirelessMobile/Wireless• Services Oriented Services Oriented

ArchitectureArchitecture• Large Data RepositoriesLarge Data Repositories• GIS SoftwareGIS Software

Capacity In 10 YearsCapacity In 10 Years• 100x Computing100x Computing• 1000x Storage1000x Storage• 5000x Networks5000x Networks

Enabling Technology


InternetWeb Services

GIS Portals Support Data Dissemination

Select Select Format Format

Data Data ConversionConversion

Zoom to Zoom to ExtentExtent

. . . Clip/Zip/Ship. . . Clip/Zip/Ship

TIGER

DXF

VPF

S57

GML

XMC

MIF

Geomedia

SDTS

DLGDWG

DGNCAD

. . . Supporting Interoperability. . . Supporting Interoperability

IMS ServerIMS Server

Support Many Support Many FormatsFormats

Many Standard Formats And Many Standard Formats And


Interoperability Is Important

ConversionConversion

Direct Read Direct Read (API)(API)

DBMSDBMS IntegrationIntegration

. . . Focus Is On Simple and. . . Focus Is On Simple and Practical Approaches That WorkPractical Approaches That Work

Web Web ServicesServices

GISGISServerServer

There Are Many Standards . . .There Are Many Standards . . .

XML/SOAPXML/SOAP


Direct Read & UseDirect Read & UseDynamic Dynamic Read/Conversion/UseRead/Conversion/Use

Custom Format Custom Format ConvertersConverters

MIF GML

M.S.

MIF

Standards And Direct Proprietary InterfacesStandards And Direct Proprietary Interfaces

Interoperability Technology Is A Fundamental Part Of GIS Products

. . . Supporting Complex Data Transformation. . . Supporting Complex Data Transformation


For GIS Networks to Work Either Everyone For GIS Networks to Work Either Everyone Uses Same Software, Data Formats, and Data Models . . . Uses Same Software, Data Formats, and Data Models . . .

They Use Interoperability ProceduresThey Use Interoperability Procedures

. . . Geoprocessing Models Can Transform. . . Geoprocessing Models Can TransformData AutomaticallyData Automatically

. . . OR. . . OR

• Format Conversion• Schema Reorganization (ETL)• Scale Projection Changes• Generalization• Merge

GeoprocessingGeoprocessingModelsModels

Interoperability Is Important

. . . Enhancing Collaboration. . . Enhancing Collaboration


Geoprocessing On Servers

GISGIS

. . . Distributed Workflow & Process . . . Distributed Workflow & Process ModelsModels

Distributing Spatial Analysis And ModelingDistributing Spatial Analysis And Modeling

NowNow FutureFuture

GISGIS

BrowserBrowserDesktopDesktop

Data SetsData Sets


Managing Multi Dimensional Geographic Data Sets And Simulation Modeling

• Data ModelingData Modeling• Tools for ManipulationTools for Manipulation

– QueryQuery– Change AnalysisChange Analysis– Iterative ProcessingIterative Processing– VisualizationVisualization

–AnimationAnimation–ChartingCharting

With Particular Focus on TimeWith Particular Focus on Time

FutureFuture

T1T1

. . . Iterative/Recursive Modeling. . . Iterative/Recursive Modeling

Simulation / Time LoopingSimulation / Time Looping

New Folder\ELNINO_Final.aviaa


GIS Will Maintain Distributed Geographic Knowledge

Relationships Will be via “Messaging”(Sending/Receiving Web Services Messages)

Geodatabases Will be Distributed and FederatedGeodatabases Will be Distributed and Federated


Serving Globes Over the Web

. . . Serving 3D Virtual Geography. . . Serving 3D Virtual Geography

Globe WebGlobe WebServerServer


Personal GIS Server

. . . Users Will . . . Users Will ShareShare And Serve Their KnowledgeAnd Serve Their Knowledge

Supporting• Map Services• Metadata Catalog (Searching

& Harvesting)• Download

– Data – Models– Data Models

• Easy to Use• Simple to Install

GeodatabaseGeodatabase

Web ServiceWeb ServiceGIS GIS

DesktopDesktop PersonalPersonalServerServer

MetadataMetadata

ModelsModels

MapsMaps

GeodataGeodataSetsSets

DataDataModelsModels

Will Allow Peer to Peer CollaborationWill Allow Peer to Peer Collaboration


TT TT

TT

TTTT

UpdateUpdateMessagesMessages

NationalNational

StateState

LocalLocal

• ReplicatedReplicated• Periodically UpdatedPeriodically Updated• History/Archiving History/Archiving

Geodatabases Will Support Geodatabases Will Support Distributed Data ManagementDistributed Data Management

TT = Transactions= Transactions

22

Infusion Confusion Solutions:Putting Technology to Work

Earth Science Data System Working Groupon Technology Infusion

Karen Moe, NASA/ESTORob Raskin, NASA/JPL


• One of four groups established by the REASoN CAN– Standards & Interfaces– Metrics Planning &

Reporting– Reuse Frameworks– Technology Infusion

• Outgrowth of SEEDS– Strategic Evolution of ESE

Data Systems– Explored ways to support

NASA ES strategy• More PI production

processing• Measurement-oriented

systemsREASoN = Research, Applications, and Education Solutions NetworkCAN = Cooperative Agreement NoticeESDSWG = Earth Science Data System Working Groups

What is the Technology Infusion Working Group?

SEEDS

REASoN CAN

ESE Strategic

Plan

ProjectsProjects

Projects

• • • Data Life Cycle

ESDSWG• Standards• Metrics• Reuse• Infusion

New in 2005


Effective Technology

Infusion

Enterprise Context• Constrained budgets• Broad data service provider community

Pragmatic Infusion Approaches

• Information sharing• Demonstration

testbeds

Emerging Technologies

• Technology investments• Web and grid computing

• Linux clusters

Organizational Goals• Lower system costs• Increase community

participation• Increase flexibility &

responsivenessInternal

Opportunities

Drivers

External

Why is Technology Infusion Important?Drivers and Opportunities


Why is Technology Infusion Important?

Meeting ESE Goals Requires Tech Infusion

• Science and application needs– Faster & better models– Near-real-time data – Easier data fusion

• Science data system needs– Enable open distributed

architecture for PI processing

– Fill capability gaps in current systems

– Support evolution

New Research

New Applications

New System Capabilities

System Capability

Vision

Technology Infusion

Technology Identificatio

n / Developmen

t

Science & App Needs


Capability Needs

Technology Projections

Technology Roadmaps

Technology DevelopmentTechnology

Infusion

Operational Systems Identified

Gaps

Solicitation Formulation

Peer Review & Competitive Selection

Capability Vision

Technology Infusion is Part of a Larger System Evolution Process

• Think globally, act locally– How can we improve technology infusion across the

community?– How can you successfully infuse technology in your own

projects?


What Capabilities are Needed?

Capability BenefitAssisted information discovery

Identify needed data quickly and easily

Seamless information access Enable access to any data from anywhere

Assisted knowledge building Provide research and operations assistance

Interactive analysis environments

Reduce research algorithm implementation from months to hours

Super-scalable analysis portals

Provide computing power and data storage on demand

Interoperable information services

Increase synergy within the ESE community through service chaining

Integrated modeling frameworks

Enable linked and ensemble models for improved predictive capability

Responsive information logistics

Ensure research priorities are met and enable new uses of ESE data

Verifiable information quality Provide confidence in products and enable community data providers


THREEDS - Topics

• Traditional Unidata Approach– Mainly meteorological data– Subscription system pushes data to user sites– Unidata Program Center provides data analysis tools for

use on data at user sites

• THREDDS Enhancements– Broader menu of Earth system data– Local client access from remote servers– Less arcane, more general and accessible tools– Integration of data and analysis tools into educational

modules and digital libraries

THREEDS

The THREDDS (Thematic Realtime Environmental Distributed Data Services) project is developing middleware to bridge the gap between data providers and data users. The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data.

The mission of THREDDS is for students, educators and researchers to publish, contribute, find, and interact with data relating to the Earth system in a convenient, effective, and integrated fashion. Just as the World Wide Web and digital-library technologies have simplified the process of publishing and accessing multimedia documents, THREDDS is building infrastructure needed for publishing and accessing scientific data in a similarly convenient fashion.

http://mmug.calfish.org/Unidata-THREDDSforMarineMetadata.ppt


THREDDS THematic Real-time Environmental Distributed Data Services

Connecting people, documents and data

PeoplePeople

DocumentsDocuments DataData


Summary

• Universities have used Unidata tools to acquire, analyze, and display real-time atmospheric data for nearly 20 years

• THREDDS – along with related client/server access and display technologies-- makes an even broader menu of Earth system data to a more diverse community of users

• THREDDS technologies enable the creation of compound educational modules and scientific publications with embedded pointers to datasets and tools.