esri uc 2014 | technical workshop | analyzing multidimensional scientific data in arcgis nawajish...
TRANSCRIPT
Esri UC 2014 | Technical Workshop |
Analyzing Multidimensional Scientific Data in ArcGISNawajish Noman
Kevin Butler
Esri UC 2014 | Technical Workshop |
• ArcGIS and Scientific Data ArcGIS and Scientific Data
• Ingest and aggregationIngest and aggregation
• Visualization and AnalysisVisualization and Analysis
• Service, Ready-to-Use Maps, Web ApplicationsService, Ready-to-Use Maps, Web Applications
• Extending Analytical Capabilities using PythonExtending Analytical Capabilities using Python
• OPeNDAP and Future DirectionOPeNDAP and Future Direction
Outline
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Scientific Data
• Stored in netCDF, GRIB, and HDF formats
• Multidimensional
• Ocean data
Sea temperature, salinity, ocean current
• Weather data
Temperature, humidity, wind
• Land
Soil moisture, NDVI, land cover
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Scientific Data in ArcGIS - Vision
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• NetCDF data is accessed as• Raster• Feature• Table
• Direct read• Exports GIS data to netCDF
Reading netCDF data in ArcGIS
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Directly reads netCDF file using o Make NetCDF Raster Layero Make NetCDF Feature Layero Make NetCDF Table View
• Directly reads HDF and GRIB data as raster
Ingesting Scientific data in ArcGIS
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
CClimate and limate and FForecast (orecast (CFCF) Convention) Conventionhttp://cf-pcmdi.llnl.gov/http://cf-pcmdi.llnl.gov/
Initially developed forInitially developed for•Climate and forecast dataClimate and forecast data•Atmosphere, surface and ocean model-generated dataAtmosphere, surface and ocean model-generated data•Also for observational datasetsAlso for observational datasets
•CFCF is now the most widely used conventions for geospatial is now the most widely used conventions for geospatial netCDF data. netCDF data. It has the best coordinate system handling.It has the best coordinate system handling.
•Current version 1.6Current version 1.6
•You can use Compliance checker utility to check a netCDF file. You can use Compliance checker utility to check a netCDF file. http://cf-pcmdi.llnl.gov/conformance/compliance-checker/ http://cf-pcmdi.llnl.gov/conformance/compliance-checker/
CF Convention
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Geographic Coordinate Systems (GCS)Geographic Coordinate Systems (GCS)• X dimension units: X dimension units: degrees_eastdegrees_east• Y dimension units: Y dimension units: degrees_northdegrees_north
• Projected Coordinate Systems (PCS)Projected Coordinate Systems (PCS)• X dimension standard_name: X dimension standard_name: projection_x_coordinateprojection_x_coordinate• Y dimension standard_name: Y dimension standard_name: projection_y_coordinateprojection_y_coordinate• Variable has a Variable has a grid_mappinggrid_mapping attribute. attribute. • CF 1.6 conventions currently supports thirteen predefined coordinate CF 1.6 conventions currently supports thirteen predefined coordinate
systems (Appendix F: Grid Mappings)systems (Appendix F: Grid Mappings)
• Undefined Undefined • If not GCS or PCSIf not GCS or PCS
• ArcGIS writes (and recognizes) PE String as a variable attributeArcGIS writes (and recognizes) PE String as a variable attribute.
NetCDF and Coordinate Systems
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Time = 1
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
Changing Time Slice
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
What about Aggregation?
• Create a seamless multi-dimensional cube fromCreate a seamless multi-dimensional cube fromo files representing different regionsfiles representing different regionso files representing different time steps/slicesfiles representing different time steps/slices
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Supports netCDF, HDF and GRIBo Spatial Aggregationo Temporal Aggregation o On-the-fly analysis
• Accessible as Map Service
• Accessible as Image Service
• Supports direct ingest
• Eliminates data conversion
• Eliminates data processing
• Improves workflow performance
• Integrates with service oriented architecture
Scientific data support in Mosaic Dataset
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Multidimensional Mosaic Datasets
• Raster Types for netCDF, HDF & GRIB
• Define variables when adding Rasters
• Each Row is a 2D Raster with variables and dimension values
• Define on-the-fly processing
• Serve as Multidimensionalo Image Serviceo Map Serviceo WMS
Aggregate (mosaic) spatial, time, and vertical dimensions
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |
Demo
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Behaves the same as any layer or table • Display
o Same display tools for raster and feature layers will work on multi-dimensional netCDF raster and netCDF feature layers.
• Graphingo Driven by the table just like any other chart.
• Animationo Multi-dimensional data can be animated through time dimension
• Analysis Toolso Will work just like any other raster layer, feature layer, or table. (e.g.
create buffers around netCDF points, reproject rasters, query tables, etc.)
Using Scientific Data in ArcGIS
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Multidimensional Mosaic Dataset - Visualization
• Visualize temporal change of a variable
• Visualize a variable at any vertical dimension
• Visualize flow direction and magnitude variables
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• New Vector Field renderer for raster o Supports U-V and Magnitude-directiono Dynamic thinningo On-the-fly vector calculation
• Eliminates raster to feature conversion
• Eliminates data processing
• Improves workflow performance
Visualization of Raster as Vectors
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Several hundreds analytical tools available for raster, features, and table
• Temporal Modelingo Looping and iteration in ModelBuilder and Python
Spatial and Temporal Analysis
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Modeling with Raster function template (RFT)
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |
Demo
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Map Service (supports WMS)o Makes maps available to the web.
• Image Service (supports WMS)o Provides access to raster data through a web service.
• Geoprocessing Serviceo Exposes the analytic capability of ArcGIS to the web.
• Map Packageo To share complete map documents and the data referenced by the
layer it contains.
• Geoprocessing Packageo To share your geoprocessing workflow.
Sharing / WMS Support (for multi-dimensions)
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Publishing a WMS on ArcGIS Server
• Enable WMS capabilities on Service Editor or Manager
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Multi-dimensional data support in WMS
• getCapabilities
o Supports time, elevation and other dimensions (e.g. depth)
• getMap
o Returns map for any dimension value
&DIM_<dimensionName>=<value>&
o Supports CURRENT for time dimension
&TIME=CURRENT&
• getFeatureInfo
o Returns information about feature for any dimension value
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Multi-dimensional WMS in ArcMap
• Supports WMS layer like any other layer
• Animates a time enabled WMS layer using time-slider
• Slices for any dimension value are accessible with ArcObjectsPublic Sub UpdateWMSServiceLayerDimensionValue() 'UID for wms service layer type Dim pUid As New uid pUid = "{27ABB9EC-7A26-4cf8-8BD4-70EC1D274E17}"
Dim pWMSMapLayer2 As IWMSMapLayer2 'calling a function to find the layer from active dataframe Set pWMSMapLayer2 = GetLayer(pUid, "myWMSLayer")
'setting values to dimensions Dim pDimNameValues As IPropertySet Set pDimNameValues = New PropertySet pDimNameValues.SetProperty "Depth", "500" 'dimension#1 pDimNameValues.SetProperty "T1", "500" 'dimension#2 Set pWMSMapLayer2.DimensionValues = pDimNameValues 'calling a function to redraw the layer RefreshActiveDataFrameEnd Sub
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
WMS in Dapple Earth Explorer
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Multi-dimensional WMS in a Web Application
http://dtc-sci01.esri.com/MultiDimWMSViewer/
Depth
Time
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
ArcGIS Online
• Curated, authoritative content provided by Esrio Ready To Useo Highly scalableo Global to National
• Authoritative content provided by the communityo Hosted in your ArcGIS Online Organization accounto Hosted on your hardware and shared to ArcGIS Online
> 100 Tb of data> 150 millions maps per day
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Ready-to-Use Maps
http://www.arcgis.com/features/maps/index.html
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Ready-To-Use Analysis Services
• Esri hosted analysis on Esri hosted datao Simplify job of GIS Professionalso Can be used in models and scripts
just like any other toolo Extend spatial analysis to a
much broader audienceo Available in Desktop or as REST service
Best practices published to the Resource CenterAnalyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• GLDAS Noah Land Surface Model OutputsGLDAS Noah Land Surface Model Outputs
o EvapotranspirationEvapotranspirationo Soil MoistureSoil Moistureo Snow PackSnow Packo OtherOther
Ready-to-Use Scientific Data Maps
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Web Application
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Web Application
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Tell the story of your scientific data – Create Story Maps
http://dtc-sci01.esri.com/DeadZoneStoryMap/ Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |
Demo
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Supplemental tools
•OPeNDAP to NetCDF
•Make NetCDF Regular Point Layer
•Make NetCDF Station Point Layer
•Make NetCDF Trajectory Point Layer
•Describe Multidimensional Dataset
•Get Variable Statistics
•Get Variable Statistics Over Dimension
•Multidimensional Zonal Statistics
•Multidimensional Zonal Statistics As Table
http://blogs.esri.com/esri/arcgis/2013/05/24/introducing-the-multidimension-supplemental-tools-2/
Python and Geoprocessing Tools
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Python is used to build custom tools for specific tasks or datasets
Application Specific Script Tools
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Geoprocessing Resource CenterGeoprocessing Resource Center
http://resources.arcgis.com/geoprocessing/http://resources.arcgis.com/geoprocessing/
• Marine Geospatial Marine Geospatial Ecology Tools (MGET)Ecology Tools (MGET)•Developed at Duke Univ.Developed at Duke Univ.
•Over 180 tools for importOver 180 tools for importmanagement, and management, and analysis of marine dataanalysis of marine data
http://mgel.env.duke.edu/mgethttp://mgel.env.duke.edu/mget
• Australian Navy toolsAustralian Navy tools (not publicly available)(not publicly available)
Community Developed Tools
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• netCDF4-python is included in 10.3/Pro
• Read and write netCDF file
• Conversion time values to date
• Multi-file aggregasion
• Compression
https://www.unidata.ucar.edu/software/netcdf/workshops/2012/netcdf_python/netcdf4python.pdf
netCDF4-Python
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Create Space-Time Cube & Emerging Hot Spot Analysis
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Creating your own tool
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
OPeNDAP to NetCDF
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Ingest OPeNDAP ServiceIngest OPeNDAP Service
• Output dynamic multidimensional Output dynamic multidimensional rasterraster
• Support Sub-settingSupport Sub-setting
Next: Make OPeNDAP Layer
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• Embrace the Common Data Model (netCDF, HDF etc.)Embrace the Common Data Model (netCDF, HDF etc.)• Use Data and metadata standards (OGC, CF etc)Use Data and metadata standards (OGC, CF etc)
• Produce and use CF complainant data Produce and use CF complainant data
• Make your data “spatial” (by specifying geographic or a projected Make your data “spatial” (by specifying geographic or a projected coordinate system)coordinate system)
• Create sample tools where possibleCreate sample tools where possible
• Clearly define workflow and requirementsClearly define workflow and requirements
Things to Consider…
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |
Demo
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
• NetCDF, Kevin Sigwart, - Demo Theater – Federal Showcase, Tuesday, 15 Jul 2014, 11:30am – 12:00pm
• Atmospheric, Weather and Climate SIG- Room 24C, Tuesday, 15 Jul 2014, 12:00pm - 1:00pm
• Weather in GIS - See Weather in Esri's Maps & Apps, Sudhir Shrestha & Dan Zimble- Session, Ballroom 20D, Tuesday, 15 Jul 2014, 3:15pm - 4:30pm
• Using Rasters to Measure Impact of Weather on Military Operations, Matt Funk- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 11:30am - 12:00pm
• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17A, Wednesday, 16 Jul 2014, 1:30pm – 2:45pm
• ArcGIS for the Military: Analyzing Environmental Impact on Operations, John Fry & Matt Funk- Session, Omni Ballroom A/B, Wednesday, 16 Jul 2014, 3:15pm - 4:30pm
• Working with Scientific Data Using Mosaic Datasets, Hong Xu- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 3:30pm – 4:00pm
• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17B, Thursday, 17 Jul 2014, 8:30am – 9:45am
• Analyzing Maritime Weather, John Fry & Matt Funk- Demo Theater - Defense and Intel - National Security, Thursday, 17 Jul 2014, 11:30am - 12:00pm
Scientific Data Sessions
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop |
Thank you…
• Please fill out the session survey:
First Offering ID: 1309
Second Offering ID: 1414
Online – www.esri.com/ucsessionsurveys
Paper – pick up and put in drop box
Analyzing Multidimensional Scientific Data in ArcGIS
Esri UC 2014 | Technical Workshop | Analyzing Multidimensional Scientific Data in ArcGIS