open data & data analytics in government...2016/04/28 · 2 nyc open data & data analytics...
TRANSCRIPT
1
NYC Open Data & Data Analytics Workgroups
Open Data & Data Analytics in Government
Session will begin at 1:00
2
NYC Open Data & Data Analytics Workgroups
Open Data & Data Analytics in Government
Agenda
• Transforming Data into Value for Government
Kevin Mergruen
Vice President, Public Sector, Information Builders
• Gaining Insights Through Open Data and Machine Data
Ashok Sankar
Solutions Director for Public Sector & Education, Splunk, Inc
• Leveraging Data Analytics to Measure Program Performance
Todd Spears
Vice President, Professional Services, 21 CT
• Speaker Q & A Panel
Kevin MergruenVP, Public SectorInformation Builders
Transforming Data into Value for Government
Agenda
• Changing the Game – Data
• Operationalizing Government Intelligence
• Results in Public Sector
• Summary
4
5
CHANGING THE GAME - DATACHANGING THE GAME - DATA
Copyright 2007, Information
Builders. Slide 6
Data redundancy across various
departments
Information is not a resource
Achieving Transformative Government Lack of Information Consistency
Types of Problems
Different definitions of data in different systems
Inaccurate information in a system
Missing or incomplete data records
Duplicate data in one or more applications
Non-Conforming Data
Fields that just aren’t useful anymore
7
Why Information Management
Source: Ventana Research Information
Management Benchmark Research
Islands of Citizen Data Reside in Different Systems
Document
Management
AFIS
Michael Johnson
Tax.tif
Tax &
Revenue
M.P. Johnson, USA
Tax ID : 234-JP-003
CRM
Call Center
Mike. Johnson
Last Interaction: 4/11/03
(Lottery proceeds on hold)
Civil,
Juvenile,
Criminal,
Probate
Michael P Johnson,
1623 Willow Lane
Deceased: 02/19/2007
Watch Lists
Michael Johnson
User ID: Mjohnso
! Personalized access
! Online Licenses
! Sub: Newsletter
DMV
Michael Percy Johnson
DL: N123-29-22-129-10
1623 Willow Lane
Ovenhurst, NY 23432
! Opt-Out flag
! Organ DonorUnstructured Data
ERP System 311 System
ExternalQuery
Courts CaseData
DMV Registration
Data
Warehouse
Michael A. Johnson
1400 54rd Avenue
NY NY
212 995-3345
3rd PartyInformation
Web
Applications
Michael Johnson
User id :mjohnson
License:: JP987
Online Transactions
Master Data Management (MDM) is the creation of a single, accurate, andunified view of enterprise data, integrating information from various datasources into one master record. This master data is then used to feedinformation back to the applications, creating a consistent view of informationacross the organization.
The insights offered by MDM allows government entities to:
• Improve revenue generation
• Reduce risk
• Lower cost and increase efficiencies
• Uncover opportunities to maximize spend
• Enhance citizen relationships – create a “Single View of the Citizen”
MDM lets you harness this potential of your data, letting you:
• Work from a consistent unified view of all enterprise data
• Make more informed, predictive decisions
• Provide everyone in your organization with the data they need, when it's needed
Master Data Management
10
Data Management Lifecycle
11
Cloud-based systems
Legacy assets
ERP applications
Business partners
Cloud-based systems
Downstream applications
Data warehouses
Business partners
12
CHANGING THE GAME - DATACHANGING THE GAME - DATA
Mastering of “Locations” – Case Study
Organization:
The City of Charlotte, a municipal government in North Carolina
Challenge:
Lack of standard processes for addresses, including physical and mailing addresses resulted in redundant efforts, inconsistent information, and high aggregate costs
Solution:
Deploy Master Data Server to automate profiling, cleansing, matching, enrichment, and monitoring activities, creating a golden set of master address and location records to serve multiple consuming departments/systems
Leverage iWay integration suite to connect departmental systems and synchronize consuming services to the new master database of location information
Results:
A consistent set of master records that can feed multiple departments/systems
Improved accuracy of location data – quick turnaround for citizen critical services
Providing customer service representatives a more complete understanding of the services that are being delivered at or near each “location”
Prompt notification of new or updated locations to city departments, resulting in fewer ‘failed’ site visits
13
14
Café Project at LA DCFSCommon Access Front End
• CAFÉ requirements
• Common Access Front End for 10 major legacy applications
• Provide single point of access for DCFS Staff
• Provide a single master Client and Provider index
• Provide a common web-based front end
• Merge data from Legacy systems
• Provide “My Account” functionality for customer/providers to update information
Louisiana Department of Children & Family ServicesSingle View of the Citizen
LA_CAFE-2202
17
NYC’s Complicated Point to Point Solution
eJustice
OracleOmniform(NYPD)DB2/MVS
NYPD
CaseManagement(DA)
CJIS
(DoITT for Law) UDIIS
(CJA)
18B Web
(ACP)LAW
(HRA)
IIS(DOC)
Admins/VMS
OLBS
(NYPD)
Adabas/MVS
DataShare 2.0
CRIMS(OCA)
Datacpm/MVS
Arrests
Fingerprint
NYSID
Non-fingerprintarrests
NYS Arrests
NYSID
Arrests
Inmate tracking
Arrests
Jail status
Court actions
Arrests
Court actions
Arrests
SSN Inmatetracking
Court actionsArrests
Inmate tracking
Arrests
SSN
Inmatetracking
Court actions
SOA
Services
SOA
Services
SOA
Services
SOA
Services
SOA
Services
SOA
Services
SOA
Services
KCDA OCA
ACS
Courts
NYPDCorrections
DataShare
Probation
Integrating 25 agencies across justice system
Each agency is a sending and receiving agency
Intel Server installed at each agency
Phased approach
Legacy exchanges first
GJXDM & NIEM messages next
NYC’s Criminal Justice Integration System The Data Share Solution
20
Polling Question
21
How many people use “BI and Analytics” in your
organization?
A None
B Less than 50
C Between 50 and 500
D Between 500 and 1000
E 1000+
Polling Question
22
How many people work in your organization?
A None
B Less than 50
C Between 50 and 500
D Between 500 and 1000
E 1000+
Strategic
Analytical
Operational
Operationalize Government Intelligence
Improving Organizational Effectiveness
o Operational InfoApps
o Information in hands of your officers
o Data Discovery/Visual Analytics
o Key Performance
Indicators
o Strategic Dashboards
23 Web SocialMachine Hadoop &
Columnar
ERP & OperationalLegacy Data Warehouse
What happened?
Why did it happen?
Do something about it!
24
Results in Public SectorAchieving Transformative Government
25
• State of Florida deployed a state-wide financial reporting portal to provide transparency on spending of tax payer money to meet ARRA requirements.
Government Compliancy
• State of Louisiana Department of Children & Family Services implemented a single view of the citizen system to improve processing of services along with an EBT Fraud Application that saves over $3M per year.
Operational Efficiency
• County of Los Angeles Auditor & Controller Office reduced waste & fraud by $millions through a pervasive contract management dashboard application deployed across the county.
Deficit Management
• City of Irving, Texas since 2007, the City has generated an additional $29.6 in revenues in land use and economic development by establishing goals and creating scorecards to measure performance.
Revenue Generation
• Texas Workforce Commission & Higher Ed Coordinating Board Texas CREWS allows parents and students to make informed decisions about college to get the best return on their educational investment.
Citizen Services
26
LA County Auditor ControllerChallenges
Supports reporting and analytics for the human resources and payroll aspects of a 100,000+ employee population
Need to help all departments better manage their contracts and expenditures
Identify waste, fraud, and abuse across departments
Move away from Excel spreadsheets to do contract management
Excel was cumbersome & error prone
Multiple versions of the truth
Dashboard DemoOverview
29
Dashboard DemoOverview
30
32
State of Louisiana Dept. of Children & Family ServicesSNAP Fraud Analytics
Challenge
Large transaction volume
Over 1 million beneficiaries, 4,000 retailers, $1 billion federal funding
Difficult to visualize trends and suspicious transactions
Geographic business intelligence application to catch the illegal trafficking of EBT transactions across the state
Field investigators can easily establish suspicious behavior based on EBT transaction amounts, times and locations
Became the State’s EBT Disaster Management system after Katrina, helping to track the refugees and ensure proper services were being provided.
Funding provided by USDA FNS – Became public domain solution. Deployed in Mississippi and Oklahoma
KSLA news report
http://www.ksla.com/Global/story.asp?S=15955474
Louisiana Department of Children & Family ServicesSNAP Fraud Analytics
35
36
37
38
39
40
By evaluating programs and institutions on the
basis of resultant wages and student loan levels,
Texas CREWS allows parents and students to make
informed decisions about college and get the best
return on their educational investment.
41
42
43
Data Value Chain
Connect
RDBMS
Application
E-Business
Legacy
SaaS
Big Data
Data
Move
Batch
Transactional
Event Driven
SOA
Automate
Orchestrate
Fix
Profile
Clean
Enrich
“DQ
Firewall”
Relate
Master Data
Organize
Synchronize
“360 View”
Govern
Monitor
Visualize
Alert
Remediate
Report
History
Business Intelligence
Dashboards
Analytics
Ad Hoc Reports
Enterprise Search
Mobile
Visualize
Predictive
Social Intelligence
Performance Mgt
Business Value
Integration Integrity Intelligence
Defining Your Transformation Strategy Step 1 – Which Information Will Deliver the Most Value?
Step 2 – Where is the Data Coming From?
Step 3 – Is the Data Ready to Be Shared?
Step 4 – Can All Stakeholders Participate & Benefit?
Step 5 – Are There Opportunities for Collaboration?
Achieving Transformative Government Getting Started
Contact Information
Kevin Mergruen, VP Public Sector
Phone: (917) 339-5311
Email: [email protected]
46
Questions
Gaining Insights Through
Open Data and Machine Data
Ashok Sankar
Solutions Director for Public Sector
& Education
Splunk, Inc.
Let’s Investigate These Areas:
• Open Data and Machine Data
• Accelerating open data initiatives with machine data
• Examples of machine data and open data in use
What is Open Data?
• Open Data is data that is publicly available and structured
in a way that enables its discovery and usability by end
users inside and outside of government
• In general, open data is consistent with the following
principles: public, accessible, described, reusable,
complete, timely, and managed post-release
Source: Project Open Data
What is Machine Data?
Logs files from computers
and network devices
Usage and Access Data from
Personal electronics
Tablets / Smartphones / Watches
All of these
produce data on
their own
Lots of data
Valuable data
Data from sensors and
remote metering devices Energy
MetersRFID Weather
sensors
Machine Data can be Open Data
The Most Immediate ROI
• IT security and IT Operations
• But, there are many other uses as well
• Business analytics
• Innovation
• Efficiencies
• Cost reductions
• Error reductions
• Fast time to resolution
• Pinpoint root cause
• Revenue generation
• Optimization
Almost all devices today generate data
Internet of ThingsCity Services Data
Transportation | Energy | Utilities | Building
Management
Water & Sewage | Public Safety
Wearables, Home Appliances,
Consumer Electronics, Gaming Systems,
Personal Security, Set-Top Boxes,
Vending
Machines, Mobile Point of
Sale, ATMs,
Personal Vehicles
Sensors, Pumps, GPS, Valves, Vats, Conveyors,
Pipelines, Drills, Transformers, RTUs, PLCs,
HMIs, Lighting, HVAC, Traffic
Management, Turbines,
Windmills, Generators,
Fuel Cells,
UPS
53
Retail | Home | Consumer
Telemedicine | Connected Cars
Where can I find them?
• There are many sources of machine data
from nearly all city departments
• Real-time sensors, monitors, activities
• And new ones are coming…
Optimizing Business placement
• Source: NYC transit data, traffic pattern monitors
• Real-time data from metro, taxi & traffic sensors
• Open data offers location of businesses,
demographic data
• Insights for optimal business placement, tracking
development application times
Monitor noise ordinance in real-time
• Device: Noise & Vibration Monitor
• Where: Construction sites
• Data is automatically sent to a compliance website
• Provides e-mail alerts, reports, and warnings of excessive noise
or vibration
• Real-time use near schools, hospitals, noise ordinance
regulated sites
Improve citizen commute experience
• Enable citizens to record and transmit ride experience on city and
state roads
• Any issues reported can now be prioritized for repair/maintenance
based on many attributes such as type, severity, priority, etc.
• This information can be published as open data along with any time to
resolution metrics
Compliance with City Mandates
• Temperature Sensor & Heating System Control
• Where: All NYC apartment building
• Monitor compliance with NYC Department of Housing
Preservation and Development mandates
• Post violations for citizens as open data
• Notifications to owners/operators
Citizen Quality of Health and Increasing
Automated Billing Accuracy
• Device: Water Meter Transmission Unit
• Where: >800k NYC buildings
• Transmits data from water sensors inside the building to the DEP
for more accurate billing
• Check water quality in real-time and let citizen’s know of any issues
• Reduce revenue leakage – citizen’s who may be claiming
secondary residence but using city as primary residence
Real-time traffic and trip resolutions
• Use real-time traffic data to alert subscribers
on traffic bottlenecks and detours
• Insights to traffic patterns and real-time
routing
• Emergency notifications
• Traffic patterns, road conditions, travel
experience across travel route state-wide
through data sharing
Senior Farmer’s Market Nutrition Program
• Program from USDA for low-income
seniors
• Exchange for eligible foods at farmers'
markets, roadside stands, and
community-supported agriculture
programs
• Scan to understand buying patterns
• Gives farmers insights on what is
popular
• Insights into access to markets,
placements
Real-time Citizen Assistance
• Open data provides locations of rest areas,
refreshment stalls
• Machine data from wearable tracks citizen
health in real-time
• Proactively assist with preventing health
issues
• Alert first responders and officers on patrol
Improving Recreation Experience
• Open data provides locations of trails,
parks, fishing streams
• Track where people prefer to bike or walk
• Insights to traffic patterns, popularity of
trails
• Prioritize improvement, expansion and
tourism efforts
Real-time Audit and Business Ratings
• Businesses are required to report activities to government on
regular basis to ensure compliance
• Audits are time-consuming and costly
• Monitor transactions automatically in real time, looking for
unusual or non-compliant behavior
• Easier oversight, faster audits
• Make available compliance as open data
Public Health
• Share data from cases with researchers, doctors and experts to
help develop new treatments and make better decisions on care
• Personalize care from data on drug treatments from government
sources, research, and patient input
• Predictive analytics to help health plans and providers make better
care decisions
Source: 9 Healthcare Innovations Driven By Open Data, InformationWeek
What can I do with it?
• Real-time machine data is powerful
• Its available to you now
• Open data can be real-time -> decisions based on
current information
• How you use it is up to your imagination
• What problem are you trying to solve?
How do you do it?
• A facility that can aggregate and correlate seemingly
disparate data
• Ask questions you never thought of before
• Analytics to deliver real-time, powerful insights
• Access to the citizen - portal, mobile, real-time alerts
Leveraging Data Analytics to
Measure Program Performance
Todd Spears
21CT
VP of Professional Services
70
70
Big Data – Big Challenges
2.5x
1018
bytes of data created each day
90%of world’s datacreated in the last 2 years
How fast is your data growing?
Turn Your Data into an Asset
71
Data Analytics:
What You Should Expect
• Stores and organizes your
data
• Interactive tools that let you
explore your data and ask
complex questions
• Simple and elegant UI’s
that deliver insights through
a contextually immersive
experience
Data Analytics Solution Stack
Data Prep
Data Analysis
Data Governance
Visualization
Turn Your Data into an Asset
What You
Should Expect
• Useful to any role and level
of professional within your
organization
• Provides you intelligence
and knowledge...not just
more data
Data Analytics:
What You Should Expect
72
• Increase revenue
• Reduce costs
• Increase effectiveness
• Improve services
Jumping into Data Analytics
Consider 4 tips when starting a
data analytics initiative
for your organization...
Tip #1: Identify Your ObjectivesIn
sig
hts
/ Re
sults
Data
Analysts
Managers
Executives
Data
Scientists
• What do you want to accomplish?
• What does success look like? How will you measure?
• Which objectives are critical to address first?
• What can you accomplish quickly?
• How much time do you have to demonstrate value?
• Who are your users?
70-90%of big data / data analytics projects fail
Tip #2: Select Data Wisely
• ID data sources that meet your objectives
• You don’t need ALL the data – start small and then expand
• Which data is accessible and useful today?
• Aim for highest value and lowest cost
75
Enterprise Data Warehouse Model:
Fill First ‣ Analyze Second
Analysis Begins
Medicaid Dept of
Health PDMP DMV DOJ SoS
Tip #3: Select Answers and Insights
76
• Sophisticated data analyticsprovide the answer andcontextually relevant information
Tip #4: Select a Way to See Your Data
• Visualization is a critical component
• Pick a solution that gives you lots of choices
• Avoid black boxes; seek traceability
• Consider an integrated solution
77
Using Data Analytics to Manage Your Program
78
Ex 1: Using Your Data to Find Waste
• Potential fraud, waste, abuse in Medicaid
• Data analytics flag suspect claims from provider
• Provider billing behavior tracked over time
• Simple filters allow exploration
• Dashboards for details and summary metrics
79
Ex 2: Using Your Data to Manage Resources
• Manage and prioritize resources
• Insights at a program level
• Providers divided into 4 risk groups
• Drill to get full list of providers and individual score
80
Ex 3: Using Your Data to Assess Efficacy of Decisions
• Retrospective analysis of expenditures
• $5M spent on inappropriate services over 3 years
• Policy revised in September 2014
• Analytics reveal immediate and effective change
81
Policy change
implemented
Ex 4: Using Your Data to Answer “What If” Questions
• Predict the impact of potential future changes
• Historical data serves as your baseline
• Model what you think will change
• Compare your predictions to baseline
82
Summary
1. Nail down your objectives and constraints
2. Keep your data small to start and show success early
3. Select a data analytics product that can serve many different users and consumers
4. Seek an interactive solution that provides answers, insights and traceability
83
85
NYC Open Data & Data Analytics Workgroups
Kevin Mergruen
Information Builders
Todd Spears
21CT
Ashok Sankar
Splunk, Inc.
Q & A