o ak r idge n ational l aboratory u. s. d epartment of e nergy making usable data available and...
TRANSCRIPT
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Making Usable Data Available and Available Data Usable: The Application of Geospatial Standards in Near Real-Time
Sensor Networks
Daniel GetmanAaron Myers
Geographic Information Science and Technology Group
Oak Ridge National LaboratoryApril 20, 2007
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
What is SensorNet
Much more than what is presented here Stand Alone Sensor Networks
• Mobile Sensor Systems• Weigh Station Systems• Port Systems• Military Base Systems
Distributed Sensor Networks• Connecting stand alone systems in a larger
network Overarching Concepts
• Interoperability/Standards• Real Time Data Distribution and Alerting• Ubiquitous Data Access• Development of Global Scale Sensor Networks
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Describing the Stand Alone Systems
Multiple Sensors Chemical – Radiation – Biological – Weather Video with real time analysis capabilities
Multiple Viewers ArcMap Google Earth Browser based viewers
Integration with Models HPAC
Data Distribution Capabilities Mobile systems communicating with
operations centers and other systems
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Defining The Goals
Minimum System Goal Provision of near real-time data within the stand
alone system while also providing access to data to external users including Operation Centers and other emergency response personnel
Maximum System Goal Develop an open standards based system within
which a variety of sensors, viewers, and complete stand alone systems can plug-and-play can use and share near real-time sensor data
Stretch Goal Make the openness of the system transparent to
all users
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Standards Used in the Stand Alone Systems
Open Geospatial Consortium (OGC) Sensor Web Enablement (SWE)
• Includes Sensor Observation Service (SOS), Sensor Alert Services (SAS), and Sensor Planning Service (SPS)
Web Feature Service (WFS) Geography Markup Language (GML)
U.S. Department of Homeland Security (DHS) ANSI N42.42 (Radiation Data Representation)
Organization for the Advancement of Structured Information Standards (OASIS) Common Alerting Protocol (CAP)
National Institute of Standards and Technology (NIST) IEEE 1451 (Sensor Plug-n-play)
Courtesy: David Resseguie, ORNL
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Why Standards are Great
Standards are great once they are adopted Data sharing is not only easier but can be
easily automated Everything is less expensive…eventually Creativity is stimulated in application
development
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Why Standards Can Be Difficult to Use
Querying differences Data Size Differences Data Structure Differences
DB
User Viewer
SQL Query
Response Containing Desired Data
Traditional Database Query
WFS Database Query
User Viewer
WFS Query
Response Containing More Than Desired Data
DBWFS Request to SQL Conversion Process
SQL Query
Raw Data to XML Conversion Process
GML, or other standard, response
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
WFS and SQL Queries
Traditional SQL QuerySELECT Col1, Att1, Val1 FROM TABLE1 WHERE Val2 = “SomeValue”
WFS Query<wfs:GetFeature service="WFS" version="1.0.0" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:snet="http://www.sensornet.gov/snet" xmlns:wfs="http://www.opengis.net/wfs"> <wfs:Query typeName=“snet:Table1"> <ogc:Filter xmlns:ogc="http://www.opengis.net/ogc"> <ogc:PropertyIsLike> <ogc:PropertyName>//snet:Val2</ogc:PropertyName> <ogc:Literal>SomeValue</Literal> <ogc:PropertyIsLike> </ogc:Filter> </wfs:Query></wfs:GetFeature>
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Simple WFS Performance TestsXhive and Oracle bassed WFS and MySQL DB
Processing Time For n Concurrent Queries While Processing Ten Inserters at Ten Features Per Insert
0
5000
10000
15000
20000
25000
1 5 10 15 20 25 30 35 40 45 50 100
Concurrent Requesters
Ave
rag
e R
esp
on
se T
ime
(ms)
XHive based WFS
Oracle based WFS
MySQL DB
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Practical Difficulties
Clients can have conflicting expectations Things that are difficult with Standards Based Systems
• Legacy Software and Data– Few software packages will natively access
standards based data sources• Familiar/Robust/Highly Functional Interface
– It has to be as good as the proprietary system• Near Real-Time Dynamic Data
– Lots of queries, some very specific (alerting)• Database Design Limitations
– Designed for support of standard rather than support of user activities
– Large Data Capacity can be problematic» Keeping all of the data without impeding
system speed• Data Integration
– Leveraging of existing proprietary data
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Idealistic SensorNet System
Standards Based Viewer
Sensor
Standard Output FormatStandard Query Format
Standard Output Format
Everyone Else
WFS
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Where does this leave the ISS?
Standards Based Viewer
Sensor
Standard Query Format
Standard Output Format
Sensor
Sensor
Proprietary Output Format 1
Standard OutputFormat
Proprietary OutputFormat 2
ProprietaryViewer 1
ProprietaryViewer 2
??
? ?
Everyone Else
??WFS
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The Duct Tape Approach
Implement standards where the technology and client requirements allow
Implement component based solutions that handle the places in the system where standards either do not exist or are not fully implemented
Plan to remove the duct tape when the standards are implemented and adopted
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Slightly More Realistic SensorNet System
Standards Based Viewer
Sensor
Sensor
Sensor
Proprietary Output Format 1
Proprietary Output Format 2
Standard OutputFormat
ProprietaryViewer 1
ProprietaryData Source
ProprietaryViewer 2
Everyone Else
? ?Custom
ConversionUtility
Custom Conversion
Utility
WFS
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Tale of Two Deployments
JFHQ SNAPS in Washington DC Mobile SensorNet System (Actually in a trailer) JOC System designed to receive data from
Mobile and Base systems GPS enabled Radiation, Chemical, Weather,
Video HPAC Integration
SRRPP at the Port of Charleston Land and Marine based SensorNet Systems Vehicle and Officer GPS Tracking GPS enabled Radiation, Video
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
JFHQ SNAPS Deployment
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
SRRPP Deployment
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Lessons Learned with the SMRSS
Intrinsic problems include Data is duplicated Two conversion processes to maintain
Performance Issues Both WFS and ArcSDE are bottlenecks compared to direct
database interaction. Together, they can really hinder performance
The conversion to ArcSDE caused a large number of reads on the WFS. This had a negative impact on the overall performance of the system. More reads per second means less writes per second
Inserting data into ArcSDE is significantly slower than inserting data directly into a database• We found that ArcObjects inserts were faster than SDE API
inserts• Querying against the data tables directly in Oracle to gather
data used in the conversion process, rather than going through SDE, is a good shortcut
Conversion process can be used for other tasking that would otherwise require an additional WFS call• Alerting and interfacing with other systems (cameras for
example)
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Standards Based Viewer
Sensor
Sensor
Sensor
Proprietary Output Format 1
Proprietary Output Format 2
Standard OutputFormat
ProprietaryViewer 1
ProprietaryData Source
ProprietaryViewer 2
Everyone Else
Custom Conversion
Utility
WFS
Realistic Idealistic SensorNet System
No pounding the WFS with SDE Inserts
Single Conversion Utility
Everyone Gets What They Want
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Evolution of the RISS
Standards Based Viewer
Sensor
Sensor
Sensor
Proprietary Output Format 1
Proprietary Output Format 2
Standard OutputFormat
ProprietaryViewer 1
ProprietaryData Source
ProprietaryViewer 2
Everyone Else
Custom Conversion
Utility
WFS
Times They Are A-Changin'
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Evolution of the RISS
Standards Based Viewer
Sensor
Sensor
Sensor
Proprietary Output Format 1
Proprietary Output Format 2
Standard OutputFormat
ProprietaryViewer 1
ProprietaryViewer 2
Everyone Else
Custom Conversion
Utility
Many, ManyData Services
DB
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
New & Improved Idealistic SensorNet System
Standards Based Viewer
Sensor
Sensor
Sensor
Standard OutputFormat
ProprietaryViewer 1
ProprietaryViewer 2
Everyone Else
Standard OutputFormat
Standard OutputFormat
Many, ManyData Services
DB
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
- Questions and Comments -
Daniel [email protected]
865-241-1745