wustl roundtable
TRANSCRIPT
Open Data
Sources and Use CasesChris KlausSolution Architect, Information BuildersResearch Scientist, Western Kentucky UniversityVisiting Faculty, Washington University in St. Louis
2
Information Builders
Lines of Business Software Products
Business Intelligence & Analytics Data Integration Data Integrity
Professional Services Consulting Services Education Customer Support Services
Corporate Summary Founded in 1975 Headquartered in New York City 50 global locations 1,600 employees 9,000+ direct customers Tens of thousands OEM customers Tens of millions product users
Focused on Innovation and our Customers
3
Portal Embedded InfoApps™
ApplicationsLegacy Systems Relational/Cubes Big Data Columnar/In Memory Unstructured Social Media Web Services Trading Partners
Integration
SocialHot
BadF
Predictive Analytics
Sentiment and Word Analytics
Search Location Analytics
Mobile Write-Back
Data Discovery Reporting Dashboards Casting and Archiving
Active Technologies
High-PerformanceData Store
Data Quality
Data Governance
Master DataManagement
Batch ETL Real-Time ESB
Integrity
Intelligence
4
Open Data
“Knowledge is open if anyone is free to access, use, modify, and share it — subject, at most, to measures that preserve provenance and openness. ” - http://opendefinition.org/
“Open data and content can be freely used, modified, and shared by anyone for any purpose” - http://opendefinition.org/
Definition
5
Global Open Data IndexRanking of the United States
6
Open Data
Data.gov is a result of an executive order: “Making Open and Machine Readable the New Default for Government Information”.
Data.Gov
7
Data.govBusiness Datasets
8
Department of Commerce
“DOC has been in the business of open data for a long time. DOC’s National Oceanic and Atmospheric Administration (NOAA) alone collects and disseminates huge amounts of data that fuel the global weather economy—and this information represents just a fraction of the tens of thousands of datasets that DOC collects and manages, on topics ranging from satellite imagery to material standards to demographic surveys.”
“The—what is now a $5-plus billion economy. You might know it as weather.com, Yahoo Weather, TV broadcasts, print, radio. This highly competitive, lucrative, for-profit industry, has a very interesting historical consistent context, which is all of the underlying data is sourced from the Commerce Department and, more specifically, NOAA.”
The impact of weather data from NOAA
9
forecast.ioOpen Data for weather is common
10
Department of CommerceDOC Agencies
11
Open Data Use CaseUS Census data identifies customers’ gender
12
Open DataUnited States Census Bureau
13
United States Census BureauFrequency of first names by gender
14
Customers’ Gender Identification
Businesses often have orders with missing gender information.
Utilizing the Census data, an attempt can be made to identify gender when it is not supplied.
The result on the following example data set showed that an account representative in Delaware had sales with a heavy gender bias.
Problem Statement
15
Customers’ Gender IdentificationData Integrity – Enhancing Gender
16
United States Census Bureau
While leveraging the first name can help enhance gender data, using the last name can supply a probable race.
While not 100% accurate for some purposes this will supply valuable information. For example, demographic based marketing.
Race probability by last name
17
US Census BureauMaps, Demographic, and Economic data
19
Open Datapatentsview.org
20
Department of the TreasuryBureau of the Fiscal Service
21
22
Department of Health and Human ServicesCenters for Medicare & Medicaid Services
23
National Provider Index
Memorial Health System (MHS) plans to use the NPI files to enhance data on referring providers, who are not MHS employees and who’s data is not available via other means.
Use Case
24
National Provider IndexArchitecture
EMR
iWay Service Manager
Data Quality Suite Enhance (NPI Data)
Real Time Integration
iWay Service Manager
Real Time Integration
iWay Service Manager
NPI Database
Lookup Provider
iWay Service ManagerCMS Website
Download NPI database from website.
Referring Provider
Provider Details
25
Department of Transportation
26
Department of EducationCollege Scorecard
27
Internal Revenue ServiceList of exempt organization
28
Open Data Use CaseStreamline Sales Tax enables real time sales tax calculations
29
Real Time Sales Tax Calculations
There are over 17,000 taxing jurisdictions in the United States with regularly changing sales and use tax requirements.
One of our clients was dealing with tax collection in two states. Their effort to comply with tax collection depended upon a manual processing of each and every invoice. The manual process resulted in inaccuracies and put the company at risk for audits and penalties.
There was a need to automate the sales and use tax process.
Given the two states had implemented the Streamlined Sales Tax agreement, our client utilized our tools to solve this problem.
Problem Statement
30
Streamlined Sales Tax
31
Open Data
5 states have no sales tax 24 states participate in the
Streamlined Sales Tax (SST) 1 state is in process of
complying with SST
Senator Wayne Wallingford (R-District 27) has sponsored Senate Bill 795 for the 2016 Legislative Session. The legislation would enact Streamlined for Missouri.
Streamlined Sales Tax
32
Real Time Sales Tax CalculationsSolution Architecture
Order Management
System
iWay Service Manager
Data Quality Suite Validate (Address) Enhance (Geocode) Enhance (Sales Tax)
Real Time Integration
iWay Service Manager
Real Time Integration
iWay Service Manager
SSTP Boundary and Rate Tables
9 digit zip codes are needed to lookup the appropriate sales and use tax, which is met through address validation.
iWay Service ManagerSST FTP Site Sales Tax
StreamlinedSalesTax.org has sales and use tax data for many states.
Sales OrderRaw Address
Sales OrderValid Address
Geocode
33
State of Missouri
34
State of OhioVoter Registration Data
35
36
Adams County, ILFelony Warrants
38
ArcGIS Open Data
“As part of your ArcGIS Online subscription, you can use ArcGIS Open Data to share your live authoritative open data. Esri-hosted ArcGIS Open Data gives you a quick way to set up public-facing websites where people can easily find and download your open data in a variety of open formats.”
41
Federal Election Commission
43
Amazon Web ServicesOpen Data and AWS Public Data Sets
44
ProPublia Data Store
45
Department of CommerceData Usability Project
46
47
United NationsOpen Data
48
Open Spending
Almost a third of countries are supplying some open data about their spending.
49
