how to manage open police data - tips for data qa/qc and automation
TRANSCRIPT
How to Manage Open Police Data – Tips for Data QA/QC
and Automation
Alice Pence, City of AuburnRobert Schultz, RCMP in BC
Cat Gracey, Safe SoftwareDale Lutz, Safe Software
Police Data Initiative 53 U.S. jurisdictions. 41 million people. 90 datasets. And growing!
www.publicsafetydataportal.org
Why Should Police Data be Shared?
● Increase transparency and build trust● Help provide improved public services● Fuel more intelligent policing decisions
The Ethics:
● Need to protect victim identity● Only share when it’s lawful to do so● Must be accurate, free of errors
Quick Bio
• BS in Geography from University of Idaho w/Certificate in GIS• In GIS field for 3+ years – been in Auburn, WA for over 1 year
Alice Pence
Our Crimes Database• Stored in a Proprietary Linux system• Raw data with only addresses• Can’t directly dump into GIS- needs to have data transformed for public/internal viewing, QA/QC, a
spatial component to view on maps and applications, attributes renamed…
A Python ScriptThe Good• It worked. Can add new offenses, change and update as needed.• Ran in windows scheduler to automate.
The Bad• Long, cumbersome script. • Needed multiple databases (.gbd → SDE) to clean and move data.
The Ugly• Can’t easily change connection to Crimes Database, no direct connection to source (connected
through ArcCatalog -then built through Model Builder- then exported out as a Python Script).• No custom alerts or in depth logging capabilities.• Not user friendly for non-python users.
First ChallengePython FME
This continues on for a few more pages…
How to rename and group 4 letter offense codes
Attribute Mapper Transformer
• As an example: RKBKCrime Type= Felony Crimes Against PersonCrime Name= Robbery
1 2 3
*double bonus: I was able to automatically import codes by a CSV file, AND…FME automatically read in all the source values for me
Into the GIS EnvironmentFME → SDE → MXD → Web Map Service → Online Application = Reporting and Analysis
ComparisonPython• Cumbersome/long script• Not very easy to
customize alerts or logs• Hard to edit- other users
might not know the python language
• No friendly user interface
FME• Easy to update codes and
field changes• Other users can easily edit-
even w/minimal knowledge of FME
• Share directly to FME Server for logs and alerts
• Friendly user interface for FME Server and easy to set up
Open Data Portal: Crimes Dataset
Updated nightly with new data, thanks to FME Desktop and FME Server
Thank you!Alice PenceGIS/Database SpecialistCity of Auburn, [email protected]
Links to sites discussed in slides: Socrata Open Data Portal: https://data.auburnwa.gov/
Auburn Crimes Map:https://maps.auburnwa.gov/html5viewer/Index.html?viewer=crimes
Background.
● City’s official open data portal is SF OpenData.
The Vision.
● Empower use of City data.
“Our vision is that the City’s data is understood, documented, and high quality.”
The Project: Increase number and timeliness of datasets
● Few workspace authors● Safe place to try out workspaces
The Results.
● Improved ETL job platform, design and architecture
● 100-500 ETL jobs● Simple ETL Jobs● Use best-practices
View the original presentation by Samuel Valdez and
Janine Heiser
(Presented at the FME World Tour 2016)
Background.
● The Citywide Street Centerline Database (CSCL) is the authoritative source for location data used by New York City's public safety agencies (NYPD, FDNY) for 911 call-taking and dispatch.
The Vision.
Make CSCL data easily accessible for the day-to-day GIS user.
View the original presentation by Chris Rado
(Presented at the FME World Tour 2016)
The Results.
New version of CSCL created that is better suited for general GIS tasks and made it available to all city agencies, to the public via NYC Open Data and to Batman.
Thank you!Live chat us at www.safe.com
Free Resources:
FME Desktop Training (June 7-8)
FME Server Training (June 14-15)
Blog post: Guide to Open Data: Using it, Sharing it, and Creating a Portal
Open Data Webinar
Open Data eBook