1

Big Data Symposium:Analytics and Applications for Federal Big

Data - FEMADr. Brand Niemann

Director and Senior Enterprise Architect – Data ScientistSemantic Community

http://semanticommunity.info/AOL Government Blogger

http://gov.aol.com/bloggers/brand-niemann/March 5-6, 2013

http://semanticommunity.info/Big_Data_Symposia

2

Begin With the End in Mind• Ms. Jo Strang, Associate Administrator, Safety, Federal Railroad

Administration, Department of Transportation “Open Gov 2.0 and Safety.Data.Gov”– New safety data sources and challenge from the National Institute of

Justice and new data from Open FEMA• Open FEMA

– Could not find with Google search– Start with Fema.gov

• http://www.fema.gov/

• Safety.Data.Gov– Could not find at that Web site– Start with Bureau of Justice Statistics

• http://www.bjs.gov/

3

My Process• Found Data Feeds at Fema.gov• Documented the Web Site, Metadata, and Data Sources for:

– USGS Earthquakes– World Map– FEMA Disaster Declarations Summary– FEMA Hazard Mitigation Program Summary– Public Assistance Subgrantee Funded Projects Detail– Public Assistance Funded Projects Summary

• Used Google Earth for the FEMA and USGS KML Files• Imported the Above Data Sets Into Spotfire• Created a Separate Visualization Tab for Each Data Set in Spotfire• Started a Unified Data Architecture• Submitted a Question About the Data Definitions to FEMA• Documented My Data Science Work in PowerPoint Slides• Provided Conclusions and Recommendations

4

Fema.gov Home Page

http://www.fema.gov/

DataFeeds

5

Fema.gov Data Feeds

http://www.fema.gov/data-feeds

6

Fema.gov Data Feeds Examples• RSS:

– Disaster Declarations• Query Form and List of Links

• KMZ:– Disaster Declared Counties - National View

• Designated Counties

• META and XLS:– Disaster Declarations Summary

• Same as RSS Disaster Declarations– Hazard Mitigation Program Summary

• Web Page– Public Assistance Funded Projects Detail

• No Web Page– Public Assistance Funded Projects Summary

• Web Page

• KML and CSV:– Worldwide M1+ Earthquakes, Past 7 Days (USGS)

• KMZ and TXT (No Web Page) My Note: But I found a Way Cool One!– http://earthquake.usgs.gov/earthquakes/map /

7

RSS: Disaster Declarations

http://www.fema.gov/disasters

8

META: Disaster Declarations Summary

http://www.fema.gov/library/viewRecord.do?id=6292

XLS

9

XLS: Disaster Declarations Summary

View / Download / Print (xls 9004K)

InstructionsMetadata

10

Hazard Mitigation Program Summary

http://www.fema.gov/hazard-mitigation-grant-program

11

META: Hazard MitigationProgram Summary

http://www.fema.gov/library/viewRecord.do?id=6293

XLS

12

XLS: Hazard MitigationProgram Summary

View / Download / Print (xls 16563K)

13

META: Public AssistanceFunded Projects Detail

http://www.fema.gov/library/viewRecord.do?id=6295

XLS

14

XLS: Public AssistanceFunded Projects Detail

View / Download / Print (xls 129914K)

15

Public Assistance FundedProjects Summary

http://www.fema.gov/public-assistance-local-state-tribal-and-non-profit

16

META: Public Assistance FundedProjects Summary

http://www.fema.gov/library/viewRecord.do?id=6299

17

XLS: Public Assistance FundedProjects Summary

View / Download / Print (xls 18935K)

InstructionsMetadataProjects 1-9Applicants

18

Worldwide M1+ Earthquakes,Past 7 Days (USGS)

http://earthquake.usgs.gov/earthquakes/catalogs/eqs7day-M1.txt

TXT not CSVSpotfire handles that!

19

Import All Data Sets (15) Into Spotfire

20

Default View in Spotfire

Illustrates Data Visualization Design Using Shneiderman’s Mantra: Overview First, Zoom and Filter, Then Details-on-Demand

21

KMZ: FEMA Designated Counties and USGS Worldwide M1+ Earthquakes, Past 7 Days (USGS)

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

22

USGS Earthquakes: Spotfire

My Note: It is hard to see at thisscale, but the data points areall related in each visualization.

My Note: Could Spotfire do this? Can USGS do what Spotfire does?

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

23

World Map and Data Tables: Spotfire

My Note: I needed a world map and to see how to integrate it with the Earthquake and other data tables. This Spotfire functionality helps me do this and is the beginning of a “unified data architecture”(Scott Gnau, President, Terra Data Labs).

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

24

FEMA Disaster Declarations Program Summary: Spotfire

My Note: Thevisualizationsare connected!

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

25

FEMA Hazards Mitigation Program Summary: Spotfire

My Note: Thevisualizationsare connected!

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

26

FEMA Public AssistanceFunded Projects Detail: Spotfire

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

27

FEMA Public Assistance FundedProjects Summary: Spotfire

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

28

Conclusions and Recommendations

• USGS Earthquakes:– Could Spotfire do this? Can USGS do what Spotfire does?

• Google Earth of FEMA and USGS:– Need a way to access Google Earth within Spotfire.

• World Map:– One can easily find and import geospatial data files (Shape) into Spotfire.

• Unified Data Architecture:– Spotfire provides a way to produce this by displaying all the data tables and their columns.

• FEMA Disaster Declarations Summary:– I submitted a question about the data definitions to FEMA and am waiting for an answer.

• FEMA Hazard Mitigation Program Summary:– There is county data, but the names do not match those used in the standard Shape file for county maps

so I had to use the State designation.• Public Assistance Subgrantee Funded Projects Detail:

– Parts 1 and 2 of the Data Set need to be merged.• Public Assistance Funded Projects Summary:

– The nine additional tabs of spreadsheet data should be used in a separate Spotfire file.