data science for epa big data analytics: oregon data dr. brand niemann director and senior data...
TRANSCRIPT
1
Data Science for EPA Big Data Analytics: Oregon Data
Dr. Brand NiemannDirector and Senior Data Scientist/Data Journalist
Semantic Communityhttp://semanticommunity.info/
http://www.meetup.com/Virginia-Big-Data-Meetup/ http://www.meetup.com/Federal-Big-Data-Working-Group/
http://www.meetup.com/Northern-Virginia-Semantic-Web-Meetup/ http://semanticommunity.info/Data_Science/Federal_Big_Data_Working_Group_Meetup
April 21, 2015
2
Data Science Process and Questions
• How was the data collected?– Dr. Joan Aron, Subject Matter Expert, helped me find it.
• Where is the data stored?– The Oregon Geospatial Data Clearinghouse and MindTouch,
Excel, and Spotfire.• What are the data results?
– The Oregon Geospatial Data Clearinghouse is an excellent source for data science work as these slides show.
• Why should we believe the data results?– The Oregon Geospatial Data Clearinghouse has high quality
data and I am a competent data scientist.
3
Oregon.gov: Home Page
http://www.oregon.gov/pages/index.aspx
Data
4
Oregon.gov: Search for Data
http://www.oregon.gov/Pages/index.aspx#search?q=Data
My Note: Let’s Look at:• Oregon Geospatial Data Clearinghouse• Oregon | Open Data• Oregon Spatial Data Library: Department of Administrative Services
5
Oregon Geospatial Data: Clearinghouse
http://www.oregon.gov/DAS/pages/irmd/geo/sdlibrary.aspx
6
Oregon Geospatial Data: Alphalist
http://www.oregon.gov/DAS/CIO/GEO/pages/alphalist.aspx
My Note: This can be a data catalog in a spreadsheet.
8
Oregon: Open Data Catalog
https://data.oregon.gov/browse
My Note: I would like the entire catalog in a spreadsheet. I made the GeoData one!The metrics suggest that only a small number of the 1732 are downloadable files.
9
Oregon Spatial Data Library: Department of Administrative Services
http://spatialdata.oregonexplorer.info/geoportal/catalog/main/home.page
My Note: This appears to be the same as the Alphalist.
10
Data Science for Oregon Data: MindTouch Knowledge Base
Data Science for EPA Big Data Analytics Oregon Data
11
Data Science for Oregon Data: MindTouch Knowledge Base Find
My Note: Find Forestry (44)1. First 4: Joan Aaron’s2. Next 3:• Forested Lands (Shape)• Forest Ownership (western
Oregon) (Shape)• Forest Types (1914) (Shape)3. Next 3:• Wildfires: Communities At Risk
Data, 2005. Oregon Department of Forestry (ODF) (22)• Classified Forestland - Urban
Interface (SB360) (Shape)• Value: Forest (GRID)
Data Science for EPA Big Data Analytics Oregon Data
My Note: Shape (112)
12
Data Science for Oregon Data: Spreadsheet Knowledge Base Find
OregonData.xlsx
14
Data Science for Oregon Data:Spotfire Cover Page
My Note: Content Analytics
Web Player
15
Data Science for Oregon Data:Spotfire GeoData Alpha List Data Set
My Note: This Excellent Catalog Needed to Be a Searchable Linked Data Set!
Web Player
16
Data Science for Oregon Data:Spotfire OWRI GDB-to-Shape Files
My Note: One Mapper (Ashley Seim) has 8914 rows of points!Cannot get OWSR Polygons to display.
Web Player
17
Data Science for Oregon Data:Spotfire Excel Spreadsheets
Web Player
18
Data Science for Oregon Data:Spotfire Forestland Shape File
My Note: The VEGNAME for most of the total area is not specified!
Web Player
19
Data Science for Oregon Data:Spotfire West Forestown Shape File
My Note: See the Dynamic Linking Between Visualizations!
Web Player
20
Conclusions and Recommendations
• Subject matter expert, Dr. Joan Aron, provides use case (Oregon forestry) and suggested links to geospatial data sets.
• Oregon.gov has excellent Open Data, Geospatial Data Clearinghouse and Spatial Data Library, but not a catalog data set.
• An Oregon Geospatial Catalog Data Set was created to aid in the selection of specific data sets and their metadata.
• MindTouch, Spreadsheet, and Spotfire Knowledge Bases were created to support the Data Science Data Publication for this use case.
• An Open Data Catalog Data Set could be used with the Oregon Geospatial Catalog Data Set to produce more Data Science Use Cases.
21
Oregon Land Use Data
Oregon Land Use Data
22
Oregon’s 2012 Integrated Report – Submitted to EPA for Review and Action
Oregon’s 2012 Integrated Report – Submitted to EPA for Review and Action
24
Data Science for Oregon GIS Data: Spotfire Cover Page
Web Player
25
Data Science for Oregon GIS Data: Oregon Land Use Access DB1
Web Player
26
Data Science for Oregon GIS Data: Oregon Land Use Access DB2
Web Player
27
Data Science for Oregon GIS Data: 2012 Assessment Geodatabase
Web Player
28
Conclusions and Recommendations
• The Oregon Land Use GIS Data and Oregon’s 2012 Integrated Report – Submitted to EPA for Review and Action GIS Data have been visualized in Spotfire.– See Data Science Data Publication (in process): Data
Science for EPA Big Data Analytics and Oregon Data• Data Science for Oregon Harmful Algal Bloom (HAB)
Data has begun to extract and integrate from Web (Word & Excel), and PDF for Spotfire analytics and visualizations.– See Data Science Data Publication (in process): Data
Science for EPA Big Data Analytics and Oregon HAB Data
29
Data Science for Oregon HAB Data: PDF Report to MindTouch Knowledge Base
Appendix B - Oregon Waters of Potential Concern for Harmful Algal Blooms
30
Data Science for Oregon HAB Data: PDF Report to Excel Knowledge Base
Oregon HAB Data
31
Data Science for Oregon HAB Data: Administrative Data 1
Web Player
32
Data Science for Oregon HAB Data: Administrative Data 2
Web Player
33
Data Science for Oregon HAB Data: HAB Advisories
Web Player
34
Data Science for Oregon HAB Data: Appendix B
Web Player
35
Data Science for Oregon HAB Data: Appendix C
Web Player
36
Conclusions and Recommendations
• HAB Strategy Report Appendices B and C Tables (PDF) and Web Downloads (Word & Excel), were extracted and integrated for Spotfire analytics and visualizations.
• The Spotfire interactive graphics support Exploratory Data Analysis.
• The data sets require semantic harmonization of Basin Names for further integration.
37
Oregon HAB Tenmile Lakes-Spotfire Manage Relations
38
Oregon HAB Tenmile Lakes-Project Info
Web Player
39
Oregon HAB Tenmile Lakes-Land Use
Web Player
40
Oregon HAB Tenmile Lakes-Participant
Web Player
41
Oregon HAB Tenmile Lakes-Activity Cost
Web Player
42
Oregon HAB Tenmile Lakes-Comment
Web Player
43
Oregon HAB Tenmile Lakes-Goal
Web Player
44
Oregon HAB Tenmile Lakes-In Stream Water Right
Web Player