mlw19 esri marklogic building a geospatial data hub final · 2019-06-18 · global leader in gis...
TRANSCRIPT
![Page 1: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/1.jpg)
17 June 2019© MARKLOGIC CORPORATION
Tim Murphy
Esri, Contextual Intelligence
James Kerr
Senior Technical Director, Partner Delivery,
MarkLogic
Tom Diepenbrock
Senior Principal Consultant, MarkLogic
Esri and MarkLogic: Building a Geospatial Data Hub
![Page 2: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/2.jpg)
EsriGlobal Leader in GIS
• Annual Revenue > $1.4 Billion
• Privately held with zero debt
• 4200 U.S. employees – 6200 overseas
• 350,000+ Organization Customers
• 6.7 Million ArcGIS Online Users
• 75% of Fortune 500
• Supporting 130+ Countries
• 2500+ Partners
• Reinvests >31% in R&D
![Page 3: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/3.jpg)
Apps
Desktop
APIs
ArcGISA Comprehensive Geospatial Platform
Supporting Individuals, Teams, and Organizations
Enterprise & Online
![Page 4: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/4.jpg)
Dashboards
Story Maps
Social MediaViewer AppsAnalytic Apps
Field Apps
Across Organizations and Beyond
Desktop Pro App
Web Services-Based
Supports Apps That Extend the Power of Your Data to Everyone
Portal
![Page 5: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/5.jpg)
ArcGIS Is Open and Interoperable
Certifications
. . . Successfully Integrated Into Thousands of Systems
AutoCAD
SAP HANA
Adobe Creative Cloud AWS
AzureMS Office
Oracle
SQL Server
Teradata
NetezzaSharePoint
Jupyter Notebook
Power BI
R Python
Revit PostgreSQL
Altibase
Dameng SQLite
Extensible ArchitectureOpen-Source Contributions
Open APIs & SDKsOpen Data Access Open-Source Integration
REST
OPeNDAPWMS
WCS CSW
GML
WFS
KMLWMTSSLD
GeoPackage
SOAP
IFC
LAS
netCDF
SQL
JSON WaterML CityGML
OneGeologyLERC
Shapefiles
Web Scene (I3S)
INSPIREXLSForm
WPSIMDF
Open Standards and Formats
Direct Product Integration
Open Software Architecture
Embeddable
. . . Many
Hadoop
developers.arcgis.com
![Page 6: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/6.jpg)
Web Apps
App Builders
Smart
Mapping
Portals
Web Maps
Online
Content
Hub
Open Data
Solution Apps
Distributed
Story Maps Visualization
Field Apps
Location Analytics
. . . a Whole New Approach
. . . Fueled by Powerful Data and Analytics
Web GISEasy, Accessible, and Interconnected . . .
![Page 7: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/7.jpg)
Connecting to Non-Traditional GIS Sources
![Page 8: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/8.jpg)
+
Global Leader in GIS
Multi-model data integration
![Page 9: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/9.jpg)
=
Location Analytics for ALL your Data
+
![Page 10: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/10.jpg)
MarkLogic Esri Integration
![Page 11: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/11.jpg)
MarkLogic Esri Connector
MARKLOGIC
ESRI
CONNECTOR
JSON
XML
RDF
TEXT
GEOSPATIAL
BINARY
ETC.
FEATURE
SERVICEFEATURE
SERVICEFEATURE
SERVICE
![Page 12: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/12.jpg)
Optic API and TDE
§ Single, integrated platform – single, unified
query interface
§ Access any shape of data, or use all
together
§ Combine strengths of each underlying
query mechanism
SEARCH SQL SPARQL
LENSES (TEMPLATES)
OPTIC API
DOCUMENTS(JSON OR XML)
![Page 13: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/13.jpg)
Data Architecture
Layer
FeatureFeature
VIEW
ASK ANYTHING
UNIVERSAL INDEX
DOCUMENTS (JSON/XML)Feature
Row
GeometryGeometry
GEOSPATIAL
INDEX
GeometryGeometry
ROW INDEX
FEATURE SERVICE
SPARQL RESULTS
FeatureRow
TRIPLE INDEX
![Page 14: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/14.jpg)
Esri Connector Architecture
TEMPLATES
VIEWS
SPARQL
OPTIC API
DOCUMENTS (JSON/XML)
GEO QUERY SERVICE
Service DescriptorService
DescriptorService
Descriptor
LAYER METADATA SERVICES
G
E
O
D
A
T
A
S
E
R
V
I
C
E
S
KOOP
MARKLOGIC KOOP
PROVIDER FEATURE
SERVICE
GEOSPATIAL
ASK ANYTHING
UNIVERSAL INDEX
ARCGIS PRO ADD-IN
![Page 15: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/15.jpg)
Service Descriptors
§ Define the Feature Services
§ Configure Layers
§ Set the Data Sources for each Layer
§ Set the Bounding Query for each layer
§ Set what Geospatial Indexes to use
§ Declarative way to generate Optic API
plans
Service Descriptor
{…
}
![Page 16: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/16.jpg)
From SPARQL
From
View
JoinCTS
Where GeoJSON Feature
GeoJSON
Feature
Optic Query Pipeline
From
View
Optic Where
GeoJSON
Feature
Convert SQL WHEREto Optic expressions
Fetch geometry from the documents (if requested
and required)
Order
ByOffset Limit
Join
DocSelect
View 1
SPARQL
View n
![Page 17: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/17.jpg)
From SPARQL
From
View
JoinCTS
Where
Optic Aggregation Pipeline
From
View
Optic Where
View 1
SPARQL
View n
Group
By
Convert Esri
group by and
stats parameters
to Optic
expressions
Order
ByOffset Limit GeoJSON
Feature
GeoJSON Feature
Stats JSON
![Page 18: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/18.jpg)
§ Expose the power of full-text indexing, geospatial indexing and row-based
indexing via one interface
§ Configuration-based
§ Standard interface to expose data to Esri tools
§ Try it out
- MarkLogic Koop Provider: https://github.com/koopjs/koop-provider-marklogic
- ArcGIS Pro Add-in: Available on request
Summary
![Page 19: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/19.jpg)
Demos
![Page 20: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/20.jpg)
Real World Example
Observation Management System
Create, Manage, Search/Discovery, Analysis
![Page 21: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/21.jpg)
Graph-based system
§ Complex, well-defined ontology
§ Approximately 600 object classes
§ Metadata, container, etc. classes
§ Mission-specific classes
§ Production Object Data
- Approximately 850M Triples
- 13M JSON-LD documents
![Page 22: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/22.jpg)
Pain Point: Search
§ Poor graph DB performance/accuracy
![Page 23: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/23.jpg)
MarkLogic Implements Search
§ Text search over JSON-LD
§ JSON-LD -> Triples
§ Geospatial, Temporal, Text Searches
§ Java Client API
![Page 24: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/24.jpg)
Pain Point: Esri Integration
§ Esri + RDMS backend
§ Complex ETL
§ Brittle code, high maintenance
§ RDMS forced implementation details in
graph data
§ Users never got what they wanted
![Page 25: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/25.jpg)
Graph DB à Relational DB ETL Process
![Page 26: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/26.jpg)
MarkLogic Esri Connector Solution
Observation
MarkLogic SPARQL
Document Search
Observation
Observation
Observation
MarkLogic Esri
Connector Services
![Page 27: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/27.jpg)
Multi-Model Database
§ All the data is already in MarkLogic
§ Added TDE
§ Added MarkLogic Esri Connector
§ No ETL
![Page 28: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/28.jpg)
Ontology Driven Templates
§ Dynamically generated TDE Templates
§ One template per object class
§ ≈600 object class templates
§ ≈300 mission-specific class templates
§ Templates update with ontology changes
§ One denormalized view, ≈700 sparse
columns defined
§ ≈150 columns actually populated with data
VIEW
![Page 29: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/29.jpg)
Ontology Driven Service Descriptors
§ Dynamically generated Service Descriptors
§ ≈600 static services based on object
classes
§ ≈100 Services based on specific mission
classes
§ 3 – 54 Layers per Service Descriptor
§ 30 attributes per layer
§ Columns with data identified via Optic API
aggregate functions
§ Service descriptors update with ontology
changes
Service DescriptorService
DescriptorService
Descriptor
![Page 30: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/30.jpg)
Benefits
No Baked in Silos Multi-Model Flexibility Core MarkLogic Capabilities
HA ACID Scalable
Search Secure Triples
Geo TDE Cloud
![Page 31: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/31.jpg)
Benefit:
Painless Esri Integration
![Page 32: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/32.jpg)
Service On The Fly
Unforeseen Benefit
![Page 33: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/33.jpg)
§ “Make me a personalized service for any observation created in the last ten
days mentioning the name ‘Humpty Dumpty’ within 30 words of the the word
‘horses’
§ “Show me all objects of type ‘Vehicle’ created in the last month that are
garaged at buildings of type ‘Single Family Home’ with four bedrooms”. **The
fact that the building is a Single Family Home may be in a different document
than the fact that the building has four bedrooms**
Service On The FlySearch Examples
![Page 34: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/34.jpg)
End Users Refine Searches
Client App ServerTransactional Apps
![Page 35: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/35.jpg)
Application Inserts Service Descriptors
Client App ServerTransactional Apps
![Page 36: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/36.jpg)
Application Inserts Service Descriptors
Client App ServerTransactional Apps
Completely new capabilities___________________________________________________________________________________________________________________________________________________________________________
ü “Breaks the bonds” of static services
ü Flexible Feature Attribution
• Based on what matches the backing query
• No longer tied to object class definitions
ü User-Specific Services
ü Service Discovery
ü Secure
• Retrieve only features each user is allowed to see at time of access
• Discover only services each user is allowed to see
___________________________________________________________________________________________________________________________________________________________________________
Users finally get what they want
![Page 37: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/37.jpg)
What’s Next?
![Page 38: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/38.jpg)
§ MarkLogic added to ICAP
§ ArcGIS Pro add-in published to the Esri Marketplace
§ Connector enhancements
- Support for time-aware layers
- SQL “bounding query”
- Improved deployment
Next Steps
![Page 39: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/39.jpg)
Questions?
![Page 40: MLW19 Esri MarkLogic Building a Geospatial Data Hub FINAL · 2019-06-18 · Global Leader in GIS Multi-model data integration = Location Analytics forALLyour Data + MarkLogic EsriIntegration](https://reader033.vdocument.in/reader033/viewer/2022060212/5f05080d7e708231d410eafd/html5/thumbnails/40.jpg)
Thank you