12.09.2016 SEMANTiCS 2016

Introduction to SHACLShapes Constraint Language

Data Quality TutorialSEMANTiCS 2016 Satellite Events

By Dimitris KontokostasAKSW/KILT - LeipzigDBpedia Association

12.09.2016 SEMANTiCS 2016

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 644055.This communication reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains.

12.09.2016 SEMANTiCS 2016

SHACLSHApes Constraint Language / (upcoming) W3C recommendation

Language for defining constraints (+++)

Validating RDF graphs against a set of shapes

Shapes are defined in an RDF graph called “Shapes Graph”

The graph to be validated is called “Data Graph”

12.09.2016 SEMANTiCS 2016

SHACL ScopeValidation (primary)

Extended (but not limited to) - interface building - data structure communication - code generation - data integration - ...

12.09.2016 SEMANTiCS 2016

Things you don’t need to know about SHACLNaming in CS is hard and SHACL didn’t came easy - SHACL was proposed by Richard Cyganiak

In the WG mailing lists there are roughly - 1K email conversations - 4.7K emails

The current spec prints in 129 pages

12.09.2016 SEMANTiCS 2016

Things you should know about SHACLSHACL is NOT yet a W3C standard / recommendation !!! - There are very strict guidelines for that

It is very close (but not there yet)

Until it does everything we present here might change - but don’t worry, most probably won’t ;)

12.09.2016 SEMANTiCS 2016

Presentation Overview● History (or how we ended up with SHACL)

○ OWL, SPARQL, SPIN, ShEx, RDFUnit

● SHACL Basics○ Terminology, Shapes, targets, filters, constraint (components)

● SHACL Core○ Predefined constraint components

● Use cases○ Live demo ...

● SHACL / SPARQL○ SPARQL extensions for constraints, components, targets, functions

12.09.2016 SEMANTiCS 2016

History(or how we ended up with SHACL)

12.09.2016 SEMANTiCS 2016

How we ended up with SHACL?● No official W3C recommendation for Validating RDF

○ Since 2001?

● But RDF Schema / OWL are schema languages...○ Or NOT?○ #OWA, #CWA, #UNA => inferencing not validation

● Standalone / ad-hoc solutions○ Most on top of SPARQL & RDFS/OWL

■ e.g. Stardog DB

● Semi-official specifications (W3C member submissions)○ SPIN○ Resource Shapes○ ShEx

12.09.2016 SEMANTiCS 2016

● OWL/RDFS & Inference/OWA○ ex:birthPlace rdfs:range ex:Place○ ex:John ex:birthPlace ex:Germany => ex:Germany a ex:Place○ ex:John ex:birthPlace ex:Bob => ex:Bob a ex:Place

● Validation after closing the world○ CWA / UNA’

● Works well in practice● What about proper semantics?

OWA: Open World AssumptionCWA: Closed World AssumptionUNA: Unique Name Assumption

RDFS/OWL-based validation

12.09.2016 SEMANTiCS 2016

SPIN - SPARQL Inferencing Notation● By TopQuadrant● SPARQL-based rule and constraint

language for the Semantic Web● Uses SPARQL to create constraints

○ Query for errors○ Every result is a violation○ ASK/CONSTRUCT queries

■ RDF vocabulary● High expressivity / low readability

Example

[] a sp:Ask ;

sp:text """

# must be at least 18 years old

ASK WHERE {

?this my:age ?age .

FILTER (?age < 18) .

}""" .

[] a sp:Ask ;

rdfs:comment "must be at least 18 years old" ;

sp:where ([ sp:object sp:_age ;

sp:predicate my:age ;

sp:subject spin:_this

] [ a sp:Filter ;

sp:expression [

sp:arg1 sp:_age ;

sp:arg2 18 ;

a sp:lt ]]) ]

12.09.2016 SEMANTiCS 2016

RS - IBM Resource Shapes 2.0● By IBM, Part of the OSLC suite● High level language for defining

constraints● Core building blocks of SHACL are based

on RS● High readability / Low expressivity

Example

<oslc-change-request> a oslc:ResourceShape ;

dcterms:title "..."^^rdf:XMLLiteral ;

oslc:describes oslc_cm:ChangeRequest ;

oslc:property [

a oslc:Property ;

oslc:propertyDefinition dcterms:title ;

oslc:name "title" ;

oslc:occurs oslc:Exactly-one . ] ;

oslc:property [

oslc:propertyDefinition oslc_cm:status ;

oslc:name "status" ;

oslc:occurs oslc:Zero-or-one ; ] .

12.09.2016 SEMANTiCS 2016

ShEx - Shape Expressions● Structural schema language for RDF

graphs● ShEx schemas /shapes language● Simplified language

○ non-RDF (RDF-Like)○ exports to RDF / JSON-LD

● Differences in semantics● Very high readability / not very high

expressivity

Example

# An <EmployeeShape>

<EmployeeShape> {

# at least one givenName.

foaf:givenName xsd:string+,

# one familyName.

foaf:familyName xsd:string,

# any number of phone numbers.

foaf:phoneIRI*,

# one FOAF mbox.

foaf:mboxIRI

}

12.09.2016 SEMANTiCS 2016

RDFUnit (a little bit self-promoting)● Implements a Unit-Testing methodology for RDF

○ Success, Fail, Error, Test Coverage, etc

● SPARQL-Based● Options to define custom SPARQL Tests● Focuses on automation

○ OWL/RDFS ○ LOV○ IBM Resource Shapes○ DSP (Dublin Core Description Set Profiles)○ SHACL (not complete yet)

● Used as a library○ Integrated with JUnit

12.09.2016 SEMANTiCS 2016

SHACL as a combination

Core Syntax: inspired mostly from IBM Resource Shapes

Extension Mechanism: inspired mostly from SPIN (some from RDFUnit)

Compact syntax: (will be) inspired by ShEx

Semantics are based in SPARQL (highly controversial decision)

12.09.2016 SEMANTiCS 2016

SHACL Basics / Terminology

12.09.2016 SEMANTiCS 2016

(Core) Terminology- Shapes- Targets (was scope)- Filters- Focus Nodes- Constraints

- Property / Focus node constraints

- Constraint Components- Predefined CCs in SHACL Core

12.09.2016 SEMANTiCS 2016

Shapes● Instances of sh:Shape

○ or referenced with e.g. sh:shape, sh:or, …● Focus node selection process

○ Targets○ Filters○ When referenced (sh:shape, sh:or, …)

● Groups constraints for the selected focus nodes together

○ Focus node constraints○ Property path constraints

ex:PersonShape a sh:Shape ;

sh:targetClass ex:Person ;

sh:stem “http://my.domain.com/person/”

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ;

sh:datatype xsd:string ; ] .

12.09.2016 SEMANTiCS 2016

Conventions / common presentation pattern

Context specific Details

Example shapes / shapes graph

related constructs are green

--------------------------------------------------

(Optional) example data / data graph

ex:a => focus node passing validationex:a => focus node failing validationex:a => not a focus node

12.09.2016 SEMANTiCS 2016

Node Selection with Targets & filters

12.09.2016 SEMANTiCS 2016

Targets (previously known as scopes)● Define the initial target nodes for a Shape● Predefined set of common selectors

○ Node targets○ Class instance targets○ Subjects-of targets○ Objects-of targets

● Custom SPARQL-Based selectors (Advanced parts of SHACL)

ex:PersonShape a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ;

sh:datatype xsd:string ; ] ;

--------------------------------------------------

ex:Bob a Person ;

ex:name “Bob” .

ex:Alice a Person ;

ex:name “Alice” .

ex:Ted a Person ;

ex:Jack ex:address “John Doe St.”.

12.09.2016 SEMANTiCS 2016

Node targets sh:targetNodeBob and Alice must:

- be employees of ACME- have an age

[] a sh:Shape ;

sh:targetNode ex:Bob, ex:Alice ;

sh:property [

sh:predicate ex:worksIn ;

sh:hasValue ex:ACME ; ] ;

sh:property [

sh:predicate ex:age ;

sh:minCount 1 ; ] .

--------------------------------------------------

ex:Bob ex:worksIn ex:Disney .

ex:Ted ex:age 30 .

12.09.2016 SEMANTiCS 2016

Class targets sh:targetClassAll Persons must have

- at least one name, - exactly one birthDate and - at most one deathDate

SHACL means SHACL Instances here

=> ?s rdf:type/rdfs:subClassOf* ?class

Not compliant with the term instance as defined by RDFS(i.e. does not take into account sub-properties of rdf:type / rdfs:subClassOf)

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:birthDate ;

sh:minCount 1 ;

sh:maxCount 1 ; ] ;

sh:property [

sh:predicate ex:deathDate ;

sh:maxCount 1 ; ] ;

--------------------------------------------------

ex:Bob a ex:Person ;

ex:name “Bob” ;

ex:birthDate “last summer” .

12.09.2016 SEMANTiCS 2016

Subjects-Of targets sh:targetSubjectsOf

All Things that have a deathDate

- should have exactly one birthDate

[] a sh:Shape ;

sh:targetSubjectsOf ex:deathDate ;

sh:property [

sh:predicate ex:birthDate ;

sh:minCount 1 ;

sh:maxCount 1 ; ] ;

--------------------------------------------------

ex:Bob ex:birthDate “2016-01-01”;

ex:deathDate “2001-01-01” .

ex:Alice ex:birthDate “2010-01-01” .

ex:Ted ex:deathDate “2002-01-01” .

12.09.2016 SEMANTiCS 2016

Objects-of targets sh:targetObjectsOf

All “Things” that a thing knows

- should have at least one name

[] a sh:Shape ;

sh:targetObjectsOf ex:knows ;

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ; ] ;

--------------------------------------------------

ex:Bob ex:name “Bob” ;

ex:knows ex:Alice .

ex:Alice ex:knows ex:Bob .

ex:Ted ex:knows ex:Paul .

ex:foo ex:knows “Bar” .

12.09.2016 SEMANTiCS 2016

Filters (sh:filterShape)● Provide more flexibility in defining focus

nodes● Further extend targets● Can be defined for a whole shape or for a

single constraint

ExampleAll cartoons employed by ACME must have a nickname

[] a sh:Shape ;

sh:targetClass ex:Cartoon ;

sh:filterShape [ #applies on all shape constraints

sh:property [

sh:predicate ex:worksIn ;

sh:hasValue ex:ACME ;

]] ;

sh:property [

sh:predicate ex:acmeNickname ;

sh:minCount 1 ; ] ;

--------------------------------------------------

ex:Coyote a ex:Cartoon ;

ex:worksIn ex:ACME ;

ex:acmeNickname “Wile E. Coyote” .

ex:BagsBunny a ex:Cartoon ;

ex:worksIn ex:Disney .

12.09.2016 SEMANTiCS 2016

Filters (sh:filterShape)● Provide more flexibility in defining focus

nodes● Further extend targets● Can be defined for a whole shape or for

a single constraint

ExampleAll cartoons employed by ACME must have a ACME nickname and the ones employed by Disney must have a Disney nickname

[] a sh:Shape ;

sh:targetClass ex:Cartoon ;

sh:property [

sh:filterShape [ # Only for this Property

Constr.

sh:property [

sh:predicate ex:worksIn ;

sh:hasValue ex:ACME ;

]] ;

sh:predicate ex:acmeNickname ;

sh:minCount 1 ; ] ;

sh:property [

sh:filterShape [ # Only for this Property

Constr.

sh:property [

sh:predicate ex:worksIn ;

sh:hasValue ex:Disney ;

]] ;

sh:predicate ex:disneyNickname ;

sh:minCount 1 ; ] ;

12.09.2016 SEMANTiCS 2016

Constraints & moreon selected nodes

12.09.2016 SEMANTiCS 2016

Shape constraints(recap)

- Shapes define targets & filters- targets & filters generate focus nodes

- Focus nodes can be generated by other means i.e. by reference (later on that)

- For every focus node we can define:- Focus node constraints: Constraints

about the focus nodes directly - Property (path) constraints: Constraints

on values connected to the focus node with a property (path)

ex:PersonShape a sh:Shape ;

# target

sh:targetClass ex:Person ;

# Focus Node constraint

sh:stem “http://my.domain.com/person/”

# Property constraint

sh:property [

sh:predicate ex:name ; #property

sh:minCount 1 ; ] ;

# Property path constraint

sh:property [

sh:path [ sh:inversePath ex:child ] ; #

^ex:child

sh:minCount 1 ; ] .

12.09.2016 SEMANTiCS 2016

Focus node constraintsValidate the focus nodes directly

i.e. if all persons is the target of a shape=> validate the focus node as a value

Attached directly on the Shape - or can be grouped with sh:shape (more later)

Some constraints do not make sense in focus node constraints - e.g. cardinality or set-based constraints

ex:PersonShape a sh:Shape ;

# target

sh:targetClass ex:Person ;

sh:stem “http://my.domain.com/person/”

# valid shape but always wrong

sh:minCount 2 ; # is always 1, the current node

sh:maxCount 3 ; # is always 1, the current node

12.09.2016 SEMANTiCS 2016

Property (Path) ConstraintsThe most common constraints...

Validate the values of a path starting from the focus nodes

The path can be simple i.e. sh:predicate(simple path of length 1)

The path can also be complex i.e. sh:path (SPARQL 1.1. Property paths)

ex:PersonShape a sh:Shape ;

# target

sh:targetClass ex:Person ;

# Property constraint

sh:property [

sh:predicate ex:name ; #property

sh:minCount 1 ; ] ;

# Property path constraint

sh:property [

sh:path [ sh:inversePath ex:child ] ; #

^ex:child

sh:minCount 1 ; ] .

12.09.2016 SEMANTiCS 2016

Property Paths ● Subset of SPARQL 1.1 Property Paths

○ PredicatePath ○ InversePath○ SequencePath○ AlternativePath○ ZeroOrMorePath○ OneOrMorePath ○ ZeroOrOnePath

● Denoted with sh:path● sh:predicate a shortcut for a

simple predicate path of length 1● Algorithm defined for parsing a path

declaration

# ex:parent

[] sh:path ex:parent .

# ^ex:parent

[] sh:path [ sh:inversePath ex:parent ] .

# ex:parent/ex:firstName

[] sh:path ( ex:parent ex:firstName ) .

# rdf:type/rdfs:subClassOf*

[] sh:path ( rdf:type [ sh:zeroOrMorePath rdfs:subClassOf

]).

# ex:father|ex:mother

[] sh:path [ sh:alternativePath ( ex:father ex:mother ) ] .

12.09.2016 SEMANTiCS 2016

Property Paths (example)

All twins must have a parent with at least two children

[] a sh:Shape ;

sh:targetClass ex:Twin ;

sh:property [

# ^ex:child / ex:child

sh:path ( [ sh:inversePath ex:child ] ex:child ) ;

sh:minCount 2 ; ] ;

--------------------------------------------------

ex:Bob a ex:Twin .

ex:Ted a ex:Twin .

ex:Alice ex:child ex:Bob, ex:Ted .

ex:Patrick a ex:Twin .

12.09.2016 SEMANTiCS 2016

Constraints & conjuction (at minor risk)Constraints are conjuctive by default

- All apply- Order / grouping does not matter- sh:or is used for disjunction

ExampleAll things linked by ex:customer must:

- Have exactly one first name- Have exactly one last name- Be of type Person and of type Customer

[] a sh:Shape ;

sh:targetObjectsOf ex:customer ;

sh:property [

sh:predicate ex:firstName ;

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:firstName ;

sh:maxCount 1 ; ] ;

sh:property [

sh:predicate ex:lastName ;

sh:minCount 1 ;

sh:maxCount 1 ; ] ;

sh:property [

sh:predicate rdf:type ;

sh:hasValue ex:Customer ;

sh:hasValue ex:Person ; ] ;

12.09.2016 SEMANTiCS 2016

Constraints groupingGrouping constraints is allowed

- Documentation / annotation- sh:name, sh:comment, ...

- Different severities- Violation (default), Warning, Info

Property constraints grouped with sh:property

Focus Node constraints grouped with sh:shape

[] a sh:Shape ;

sh:targetClass ex:Employee ;

sh:stem “http://my.company.com/empl/” ;

sh:shape [ # same as above

sh:stem “http://my.company.com/empl/” ; ] ;

sh:property [

sh:predicate ex:name ;

sh:name “Name” ;

sh:comment “Name of the employee” ;

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:name ;

sh:severity sh:Warning ;

sh:maxCount 1 ; ] ;

12.09.2016 SEMANTiCS 2016

Severities3 predefined severities

- Violation, Warning, Info- Default to Violation if unspecified

Passed to the violation results as annotation

- Severity does not matter

Example - Must have a name - Should have an address - nice-to-have a postal code

[] a sh:Shape ;

sh:targetClass ex:Employee ;

sh:property [

sh:predicate ex:name ;

sh:severity sh:Violation ; # Optional / default

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:address ;

sh:severity sh:Warning ;

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:postCode ;

sh:severity sh:Info ;

sh:minCount 1 ; ] ;

12.09.2016 SEMANTiCS 2016

Annotation constructs (i.e. for form building)sh:name: a short name

sh:description: a short description

sh:order: the suggested display order

sh:group: The suggested display group

sh:PropertyGroup: defines a property group

[] a sh:Shape ;

sh:targetClass ex:Employee ;

sh:property [

sh:predicate ex:firstName ;

sh:order 1 ;

sh:group ex:PersonalDetails ;

sh:name “First Name” ;

sh:comment “First name of the employee” ; ] ;

sh:property [

sh:predicate ex:lastName ;

sh:order 2 ;

sh:group ex:PersonalDetails ;

sh:name “Last Name” ;

sh:comment “Last name of the employee” ; ] ;

ex:PersonalDetails

a sh:PropertyGroup ;

sh:order 0 ;

rdfs:label "Employee Personal Details" .

12.09.2016 SEMANTiCS 2016

Core SHACL Constraint Components

12.09.2016 SEMANTiCS 2016

Value type constraint componentssh:classThe value is of type provided (SHACL Instance)

sh:datatypeThe value is of the datatype provided

sh:nodeKindThe value is one of - sh:BlankNode, sh:IRI, sh:Literal - sh:BlankNodeOrIRI, sh:BlankNodeOrLiteral sh:IRIOrLiteral

-

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:child ;

sh:class ex:Person ; ] ;

sh:property [

sh:predicate ex:birthDate ;

sh:datatype xsd:date ; ] ;

sh:property [

sh:predicate ex:knows ;

sh:nodeKind sh:BlankNodeOrIRI ; ] ;

12.09.2016 SEMANTiCS 2016

Cardinality constraint componentssh:minCountThere must be at least the values specified

sh:maxCountThere must be at most the values specified

NoteAbsence of cardinality constraints assumes values are unbound [0, unlimited)

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ; ] ;

sh:property [

sh:predicate ex:birthDate ;

sh:maxCount 1 ; ] .

12.09.2016 SEMANTiCS 2016

String-based constraint componentssh:minLength / sh:maxLengthNumber of min/max value characters

Sh:pattern (+ sh:flags)The value conforms to provided pattern

sh:stemThe value is an IRI and starts with the provided value (namespace-like)

sh:uniqueLangAt most one value per language (e.g. only 1 English)

-

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:name ;

sh:minLength 3 ;

sh:maxLength 20 ;

sh:uniqueLang true ; ] ;

sh:property [

sh:predicate ex:id ;

sh:pattern “^ABC” ;

sh:flags “i” ; ] ;

sh:property [

sh:predicate ex:mother ;

sh:stem “http://my.ex.com/mother/” ; ] .

12.09.2016 SEMANTiCS 2016

Value-range constraint componentssh:minExclusive / sh:minInclusiveValue more than (or equal) than

sh:maxExclusive / sh:maxInclusiveValue less than (or equal) than

-

[] a sh:Shape ;

sh:targetClass ex:Adult ;

sh:property [

sh:predicate ex:age ;

sh:minInclusive 18 ; ] .

[] a sh:Shape ;

sh:targetClass ex:Teenager ;

sh:property [

sh:predicate ex:age ;

sh:minInclusive 10 ;

sh:maxExclusive 18 ; ] .

12.09.2016 SEMANTiCS 2016

Property-pair constraint componentssh:equalsValues of a property equals the values of another property

sh:disjointNo values of a property are equals with any value of another property

sh:lessThan / sh:lessThanOrEqualsAll values of a property are less (or equal) than all values of another property

-

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:firstName ;

sh:equals ex:givenName ; ] ;

sh:property [

sh:predicate ex:child ;

sh:disjoint ex:parent ; ] ;

sh:property [

sh:predicate ex:birthDate ;

sh:lessThanOrEquals ex:deathDate ; ] .

12.09.2016 SEMANTiCS 2016

Shape reference constraint component (sh:shape)

The value must comply with the given shape

ExampleThings people know should have a name

Values of sh:shape are shapes that can have - focus node constraints - property (path) constraints

[] a sh:Shape ;

sh:targetClass ex:Person ;

sh:property [

sh:predicate ex:knows ;

sh:shape [

sh:property [

sh:predicate ex:name ;

sh:minCount 1 ;

] ;

] ;

.

12.09.2016 SEMANTiCS 2016

Negation constraint component (sh:not)Negation

The value must not have a given shape

A dog likes only things that are not cats

[] a sh:Shape ;

sh:targetClass ex:Dog ;

sh:not [

a sh:Shape ;

sh:property [

sh:predicate ex:likes ;

sh:minCount 1 ;

sh:class ex:Cat ;

]]] .

12.09.2016 SEMANTiCS 2016

Disjunction constraint component (sh:or)Disjunction

The value must conform to at least one of the given shapes

A person must have at least one first name or a given name

[] a sh:Shape ;

sh:targetClass ex:Dog ;

sh:or (

[ a sh:Shape ;

sh:property [

sh:predicate ex:firstName ;

sh:minCount 1 ; ]]

[ a sh:Shape ;

sh:property [

sh:predicate ex:givenName ;

sh:minCount 1 ; ]]

) .

12.09.2016 SEMANTiCS 2016

Many details left (due to time)- Filters & targets from referenced shapes (sh:or, sh:shape, sh:not)- Recursion (unspecified)

- Recursion with sh:path (zero/one or more paths)

- More sh:or examples- disjunction with sh:path (alternate path)

- Focus node constraint limitations (more examples)- Shapes/Data graphs details- SHACL Violation vocabulary (quick overview on next slide)

12.09.2016 SEMANTiCS 2016

Violation vocabularysh:ViolationResult - sh:focusNode - sh:path - sh:value (not available always) - sh:message - sh:severity - sh:sourceConstraintComponent - sh:sourceShape

[

a sh:ValidationResult ;

sh:severity sh:Violation ;

sh:focusNode ex:Bob ;

sh:path ex:age ;

sh:value "twenty two" ;

sh:message "ex:age should be xsd:integer." ;

sh:sourceConstraintComponent

sh:DatatypeConstraintComponent ;

sh:sourceShape ex:PersonShape .

]

12.09.2016 SEMANTiCS 2016

Example with RDFUnit

12.09.2016 SEMANTiCS 2016

Run SHACL from your IDE (intelliJ, Eclipse) as JUnit- Code to run- Example SHACL graph- Example data graph- How RDFUnit/JUnit works- Sample screencast

(SHACL Support is not complete but you mix with RDFS/OWL axioms & RDFUnit TestCases)

12.09.2016 SEMANTiCS 2016

12.09.2016 SEMANTiCS 2016

Recap ...- SHACL and some history- Good overview of

- Shapes - Targets- Filters- Focus Node constraints- Property Constraints

- SHACL Core constraint components

12.09.2016 SEMANTiCS 2016

Slides at slideshare.com/jimkont

aligned-project.eu