rdf database-as-a-service with s4

RDF Database-as-a-Service with S4

Marin Dimitrov, CTO of Ontotext

Apr 27th, 2015

RDF DBaaS with S4 / AKSW Colloquium #1 Apr 2015

• Self-Service Semantic Suite (S4)

• RDF DBaaS on AWS

• Demo

Contents

#2 RDF DBaaS with S4 / AKSW Colloquium Apr 2015

About Ontotext

• Provides products & solutions for content enrichment and metadata management

– 70 employees, headquarters in Sofia (Bulgaria)

– Sales presence in London, Washington & Boston

• Major clients and industries

– Media & Publishing

– Health Care & Life Sciences

– Cultural Heritage & Digital Libraries

– Government

– Education


The Self-Service Semantic Suite (S4)


• On-demand capabilities for text analytics, content enrichment and metadata management

– Text analytics for news, life sciences and social media

– RDF graph database as-a-service

– Access to large open knowledge graphs

• Available anytime, anywhere

– Simple RESTful services

• Simple, pay-per-use pricing

– No upfront commitments

What is S4?


• Enables quick prototyping

– Instantly available, no provisioning & operations required

– Focus on building applications, don’t worry about infrastructure

• Free tier

– Even bigger free quotas for research groups & projects

• Easy to start, shorter learning curve

– Various add-ons, SDKs and demo code

• Based on enterprise semantic technology by Ontotext

S4 benefits


• Text analytics services

– News annotation

– News categorisation

– Biomedical

– Twitter

• Entity linking & disambiguation

– Mappings to DBpedia & GeoNames instances

– Mappings to biomedical data sources (LinkedLifeData)

• HTML, MS Word, XML, plain text input

• Simple JSON output

Text analytics with S4


News analytics example

#8

S4 result

RDF DBaaS with S4 / AKSW Colloquium Apr 2015

• Available from AWS Marketplace

• Variety of hardware configurations

– 2 to 8 CPU cores / 8 to 61 GB RAM

– IOPS performance & encryption (EBS)

• Manage large data volumes

• Pay-per-hour pricing

Self-managed RDF DB in the Cloud


• Low-cost DBaaS available 24/7

• Ideal for small & moderate data volumes

• Instantly deploy new databases when needed

• Zero administration: automated operations, maintenance & upgrades

• Users pay only for the actual database utilisation

– Number of triples stored + number of queries per month

Fully managed RDF DB in the Cloud


• SPARQL query endpoint to the FactForge knowledge graph

– 500 million entities / 5 billion triples

• Key LOD datasets integrated

– DBpedia, Freebase, GeoNames, WordNet

– Dublin Core, SKOS, PROTON ontologies and vocabularies

Knowledge graphs with S4


• (available soon)

• Knowledge Graph bundles

– DBpedia, Wikidata, GeoNames, …

– GraphDB RDF database (self-managed @ AWS)

– 3rd party interactive data exploration tool (faceted search, data navigation, dynamic charts)

• Get instant & reliable access to KGs without dealing with provisioning, data import, maintenance, …

Knowledge graphs with S4


• Java & C# SDKs

• Sample code

– Java, C#, NodeJS, JavaScript, Python, PHP, Groovy

– Curl examples for the most impatient

• GATE & UIMA plugins

• Firefox & Chrome add-ons

• Online documentation

S4 for developers


• DaPaaS & ProDataMarket

– Goal: Open Data / Linked Data publishing & hosting

– S4 role: scalable Linked Data hosting infrastructure

• KConnect

– Goal: semantic annotation, search & analytics for healthcare data

– S4 role: scalable text analytics & RDF data management infrastructure

Research projects using S4


Fully Managed RDF Database-as-a-Service


• Elastic

– dynamically adapt to data & query volumes

• High availability & resilience

– no SPFs, “graceful degradation” of performance upon failures

• Cost efficient

– cost aware architecture

– Key aspect for Open Data scenarios like DaPaaS & ProDataMarket

• Isolation of the multi-tenant databases

• Fair use of shared resources

Requirements


• Micro DB

– Up to 1M triples

– FREE, available now

• Extra Small DB (10M triples)

• Small DB (50M)

• Medium DB (250M)

• Large DB (1B)

RDF DBaaS options on S4


• AWS based

– Storage, compute, load balancing, integration services…

• Ontotext GraphDB for the database instances

• OpenRDF REST services

• Docker for containerisation

• Network-attached volumes (EBS) for data storage

• A DBaaS on S4 is…

– A GraphDB instance

– Running within a Docker container

– With a private EBS data volume

Implementation


• Routing nodes

– Expose OpenRDF RESTful services to apps

– Access control & quota checks

– Forward client requests to the proper data node

– Temporarily queue requests when necessary

• Data nodes

– Multiple Docker containers (GDB+EBS) per node

• Coordinator (single)

– Distribute DB initialisation / creation tasks to data nodes

• Management Console

S4 DBaaS architecture


S4 DBaaS architecture


REST apps

3rd party RDF tools

Qu

ota

& A

cce

ss C

on

trol

routers

data nodes

coordinator

EBS

backups

SNS

Docker Repository

Account management

Quota management

reporting

Monitoring & Logging Dynamo

Amazon S3

images

• CRUD

– Router node receives a request

– Routes it to the proper data node & container

– Receives a response, forwards it back to client app

• Routing updates

– Data nodes push notification via SNS – “hearbeats” + changes regarding the hosted DBs (if any)

– Each routing node receives the notifications (via SNS) and updates its routing tables

– Coordinator also receives notifications, learns which DBs are operational / down for maintenance

Normal operations


Failure case #1 – data node crash


REST apps

3rd party RDF tools

Qu

ota

& A

cce

ss C

on

trol

routers

data nodes

coordinator

EBS SNS

Docker Repository

1 2

2

2

3

Recovery from a data node crash


REST apps

3rd party RDF Visualisation

Qu

ota

& A

cce

ss C

on

trol

routers

data nodes

Coordinator

EBS SNS

Docker Repository

1

2

3+4

5 6

6

6

7

Auto Scaling

Failure case #2 – router crash & recovery


REST apps

3rd party RDF tools

Qu

ota

& A

cce

ss C

on

trol

routers

data nodes

coordinator

EBS SNS

Docker Repository

1 3

Auto Scaling

4

5 6

7

8 2

• (open connections from client apps to the node are terminated)

• Auto-scaler starts a new router node

– New router subscribes to SNS for heartbeats & updates

• Load balancer starts sending new client requests to router

– Router puts them in the local queue (if routing table is still incomplete)

• Heartbeats from data nodes are received

– Routing information is now complete

– Router starts sending the queued requests to data nodes

Recovery from a router crash


Failure case #3 – coordinator crash & recovery


REST apps

3rd party RDF tools

Qu

ota

& A

cce

ss C

on

trol

routers

data nodes

coordinator

EBS SNS

Docker Repository

2

Auto Scaling

4

5

6

6

3

Create DB 1

• Routers can route requests to data nodes as usual

– … but new DBs cannot be created temporarily

– … and data nodes with free container slots can’t get info on DBs waiting for initialisation

• AWS Auto-scaler starts a new Coordinator node

– Coordinator reads a list of all registered DBs from the metadata store & subscribes to SNS

• Coordinator starts receiving heartbeats & updates from data nodes

– … learns which DBs are operational / pending

– … and resumes distributing new / pending DBs initialisation tasks to the data nodes with free slots

Failure case #3 – coordinator crash & recovery


• Combination of coordinator + data node + routing node crash – same as #1 + #2 + #3

• Routers depend on data nodes

• Data nodes depend on Coordinator

• Coordinator does not depend on other nodes

– No heartbeats coming, means all DBs are down

– Start distributing DB initialisation tasks whenever a request comes from a working data node

– Eventually, all data nodes are up, DBs initialised, heartbeats & routing updates start coming

– … and routers can start routing client requests

Composite failure & recovery


Management interface


Micro, XS, S, M, or L

I/O performance

R/O access to Open Data services or open knowledge

graphs

Management interface


DBaaS endpoint

DB details summary

Backup, export, change settings, delete

Run a test query

• Gradually introduce XS, S, M and L instances

• Integration with the GraphDB Workbench management UI

• LDF based containers

• Multi-datacenter deployment

• Replication across datacenters (single master)

Roadmap


• “On-demand Text Analytics and Metadata Management with S4” (ESaaSA @ CLOSER’2015)

• “Text Analytics and Linked Data Management As-a-Service with S4” (Wasabi @ ESWC’2015)

• “Low-cost Open Data As-a-Service in the Cloud” (SemDev @ ESWC’2015)

More Details


Demo


• (create an account & generate an API key pair)

• Create a new DB

• Create a new repository in the DB

– via the REST API / OpenRDF Java SDK / curl

– …or via UI tools like the OpenRDF Workbench

• Import sample data (REST / OpenRDF Workbench)

• Run a query through the public SPARQL endpoint

Demo scenario


Demo data – Universities in Saxony


#1 Create a database


#2a Create a repository & load data (curl)


@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix rep: <http://www.openrdf.org/config/repository#>. @prefix sr: <http://www.openrdf.org/config/repository/sail#>. @prefix sail: <http://www.openrdf.org/config/sail#>. @prefix graphdb: <http://www.ontotext.com/trree/owlim#>. [] a rep:Repository ; rep:repositoryID “test01" ; rdfs:label "Description of my repository" ; rep:repositoryImpl [ rep:repositoryType "openrdf:SailRepository" ; sr:sailImpl [ graphdb:ruleset "owl-horst-optimized" ; sail:sailType "owlim:Sail" ; graphdb:base-URL "http://example.org/graphdb#" ; graphdb:repository-type "file-repository" ; ] ].

Repository configuration file

config.ttl

• Repository name: ”test01” • OWL-Horst reasoning ruleset

#2a Create a repository & load data (curl)


API_KEY=… KEY_SECRET=… USER=… DATABASE=… REPOSITORY=… SERVICE_ENDPOINT="https://$API_KEY:[email protected]/$USER/$DATABASE" curl -X POST -H “Content-Type:application/x-turtle” -T config.ttl $SERVICE_ENDPOINT/repositories/SYSTEM/rdf-graphs/service?graph=http://example.com#g1 curl -X POST -H “Content-Type:application/x-turtle” -d “<http://example.com#g1> a <http://www.openrdf.org/config/repository#RepositoryContext>.” $SERVICE_ENDPOINT/repositories/SYSTEM/statements curl -X POST -H "Content-Type:application/rdf+xml;charset=UTF-8" -T example.rdf $SERVICE_ENDPOINT/repositories/$REPOSITORY/statements

Create a repository

Upload sample data from example.rdf

• User: 4730361296 • Database: demo01 • Repository: test01

• Configuration: config.ttl

#2b Create a repository & load data (OpenRDF Workbench)


DBaaS endpoint



DBaaS endpoint

#3a SPARQL query (OpenRDF Workbench)


#3b SPARQL query (from the S4 Management Console)


PREFIX dbpedia: <http://dbpedia.org/resource/> PREFIX dbp-prop: <http://dbpedia.org/property/> PREFIX dbp-ont: <http://dbpedia.org/ontology/> SELECT ?name ?numberOfStudents ?staff ?established WHERE { dbpedia:University_of_Leipzig rdfs:label ?name ; dbp-prop:students ?numberOfStudents ; dbp-prop:staff ?staff ; dbp-prop:established ?established . }

• S4 provides an enterprise RDF DBaaS

• Resilient design, high availability

• Instantly available whenever needed, easy to use, OpenRDF REST services

• Zero administration: automated operations, maintenance & upgrades

• Free DBs up to 1M triples (even more for research teams & projects)

• Check out http://s4.ontotext.com

Key takeaways


Thank you!


rdf database-as-a-service with s4

Technology

s4 aksw colloquium

s4 result rdf dbaas

dbaas available

ontotext s4 benefits

managed rdf db

graphdb rdf database

s4 marin dimitrov

data navigation