brief on linked data for u.s. epa's chief data scientist
TRANSCRIPT
Bernadette Hyland
CEO & co-founder
11911 Freedom Drive, Suite 850
Reston, VA 20190
Tel. +1-571-331-3758
[email protected]@BernHyland
[email protected]@3RoundStones
Extend Your Reach.
Linked Data for Smarter Decisions.
Follow up information prepared forRobin Thottungal, Chief Data Scientist / Director of Analytics
US Environmental Protection Agency - Feb 26, 2016
Today’s reality at EPA
»Tens of thousands of sources
»Many formats - JSON, XML, CSV, PDF, PPT, SHP, SHX, text, binary…
»Thousands of data silos
»No single source of truth
»Varied interpretations
»Brittle interfaces - lack of interoperability
Image Credit: Smart Data Collective
Wide Variety of Data at EPA
3
Image Credit: MarkLogic, see http://www.marklogic.com/resources/marklogic-semantics-datasheet/resource_download/datasheets/
Credit: Frederick Giasson, Data Scientist & Software Developer, http://fgiasson.com/blog/index.php/2014/07/23/big-structures-where-the-semantic-web-meets-artificial-intelligence/
Potential at EPA …
• Findable data
• Accessible data
• Interoperable data
• Re-usable data
• Shared context
• Data Platforms (HDFS, NoSQL)
Linked Data is helping to extend & augmentEPA’s significant investment in enterprise relational technologies
How?
By leveraging NoSQL Data Platforms that rigorously adhere to international data interoperability standards. *
* Relevant international data exchange standards are published by the W3C, OGC, IEEE
Image Credit: MarkLogic
Graph databases, as a subset of NoSQL databases, are the most efficient way to look at the relationships between data
items, patterns of relationships and interactions.Image Credit: Cray, see http://www.cray.com/blog/graph-databases-101/
Graph Databases 101
Hadoop Integration»While over 90% of the world’s data has been created in the last two
years, EPA has tremendous variety of data requires the “right tool for the job”
»Historic data (“short, wide, complex data”) vs.
»Granular sensor & GIS data (“long skinny data”)
»Core mission-based systems with robust historic data, includes:
»Toxics Release Inventory (TRI)
»Facilities Registry (FRS)
»RCRA Handler
»EPA’s enterprise information architecture should include a data platform that leverages Hadoop: HDFS and MapReduce, and accommodates EPA’s robust data landscape.
»Must support modern, open source tools for application development, visualizations, crowdsourcing, and deployment on the Web
8
One option - MarkLogic Integrates Hadoop Ecosystem &EPA’s Robust Data Landscape
Image Credit: http://www.marklogic.com/what-is-marklogic/features/hadoop-integration/
EPA Robust Data Ecosystem is adaptable using a Linked Data Approach
» Makes data integration faster and easier » By using a global addressing scheme, HTTP URIs.
» Uses semantics to “glue” together data faster.» Common semantic definitions link traditional relational
models.» No more out of data documentation using standard
vocabularies.
» Robust search and discovery by leveraging the semantic graph.
» Scales to the Web!
9
All modern data platforms deployed at EPA should
»Support options for data modeling - Linked Data (JSON-LD, RDF), SQL (JSON, XML)
»Native store and query of documents, blobs and structured data.
»Standards-based query interface across documents and data, e.g., Full support for SPARQL 1.1
»Offer enterprise functionality including high availability & disaster recovery, scalability & elasticity, ACID transactions
»Be deployable on FedRamp certified cloud provider certifying controls for security, high availability, disaster recovery
»Scale to billions of statements, triples, etc.
»Store unstructured data across clusters like Hadoop, making it easy to move data partitions.
»Much but not all of EPA’s data is well suited for a Linked Data approach.
»Linked Data is based on 20+ year old idea, a system of linked information systems
M A N N I N G
David WoodMarsha ZaidmanLuke RuthWITH Michael Hausenblas
FOREWORD BY Tim Berners-Lee
Structured data on the Web
Linked Data
Goals: Governmental transparency and/or improved internal efficiencies
Governments Worldwide are using a Linked Data Approach
Linked Data Apps use data from many
EPA programs and other Open Data Sources
Linked Data Management SystemFor government open data publishing
Funded by
Linked Data Platform is in QA now! https://usepa.3roundstones.net Anticipated to move to production in 2016.
shared innovation™
Search for facilities where we live. Unlike many EPA Web portals, linked data is human AND machine readable data. No screen scraping is required. Encourages re-use (discourages data silos)
The EPA Linked Data service CONNECTS data silos, and provides familiar map and table data views
Click to drill down to pollution reports that combine data from 5 previously unconnected data silos.
Click through to the source of the pollution data via the source reports (TRI).
EPA collects granular pollution data. Linked Data opens up the data to a much wider audience in a human readable format.
Previously, only people who employed complex screen scraping techniques could get at this data. Now, EPA open data is available using an international data standard, with one click!
Good news story!Pollution graphs created in one week using Open Source Software & EPA Linked Data
Use of shared vocabularies, e.g. Places, Geographis, Dublin Core, Geo, FOAF, ORG, Vcard are the “lingua franca” of data interoperability
Case StudyUsing EPA Linked Data to assist chronic asthma/COPD patients
with timely weather alerts
Funded by
User
NOAA US EPA AirNow
DBpediaNational Library of Medicine
US EPA SunWise
Case Study: OrgpediaAn open organizational data project
on public & private companies
Funded by
Using the Callimachus Open Source Data Platform, we rapidly built a crowdsourcing platform.
3 Round Stones provides commercial application
support on the cloud or behind the enterprise firewall using
@3RoundStones http://3RoundStones.com
CONTENT MANAGEMENT
SYSTEM
LINKED DATA MANAGEMENT
SYSTEM
Callimachus
UN
ST
RU
CT
UR
ED
T
EX
T
TE
XT
ST
RU
CT
UR
ED
D
AT
A
DA
TA
Callimachus supports
in-browser development
Callimachus Enterprise customers are creating data-driven applications with data from leading graph
databases:
Callimachus is a scalable Web application server for publishing and consuming open data
Who uses it?
• Government, international publishers, healthcare / life sciencesWhat pain does Callimachus address?
• Integration of data silos where a graph approach is needed• Rapid creation of visualizations, dashboards (mashups) & info graphics• Less expensive solution to a data warehouse
Example apps?
• Collaborative knowledge management • Publishing workflow• Drug discovery / clinical trials • Predictive Analytics
data interoperability & portability
Supports:• HTML5, XHTML5, CSS3, JavaScript• XQuery, XProc, XPath, XSLT• SPARQL 1.1 Query, Update, Federated Query,
Service Description, Property Paths, Graph Store HTTP Protocol
• RDF/XML, RDF/Turtle, JSON-LD, SPARQL XML, SPARQL JSON
Callimachus is fanatical about
Contractor (3 Round Stones, Inc.)
Public
Application, Script or automated client
Web Browser
SPARQL endpointREST APIResource URIs
Linked Data management systemlocated at a Tier 1 Cloud Provider
(FISMA compliant)
RDF Database
Registered developer
<HTML>
Enterprise Data Documents
Read/ Write
Point to, include
Callimachus Enterprise
“Big Data Is Important, but Open Data Is More Valuable” As change agents, enterprise architects can help
their organizations become richer through strategies such as open data.
David Newman, VP Research, Gartner
Open Source Enterprise License
Community supported Commercial support
in-browser development, deployment, backups
Linked Data publication
User profiles, social sharing
Document, app management
OpenAnnotation support
External datasources
Shared deployments
Realms (virtual hosts)
Enterprise management
Cloud deployments
Callimachus
Callimachus™, the Callimachus logo, Callimachus Enterprise™, the Callimachus Enterprise logo and tagline, are trademarks of 3 Round
Stones, Inc. and are registered in the United States and abroad. Copyright © 2011-2016 3 Round Stones, Inc. All rights reserved.
Callimachus Enterprise