Michael Schmidt1, Olaf Görlitz2, Peter Haase1, Günter Ladwig3, Andreas Schwarte1, Thanh Tran3

FedBench

A Benchmark Suite for Federated Semantic Data Processing

1 2 3

10th Intl. Semantic Web Conference, Oct 26, 2011, Bonn

Linked Data Evaluation Strategies

Central Repository

RDF Data

RDF Data

RDF Data

Query

Centralized Linked Data Processing

Linked Data Evaluation Strategies

Local Rep.

RDF Data

RDF Data

RDF Data

Query

Central Repository

RDF Data

RDF Data

RDF Data

Query

Federation Layer

SPARQL Endp.

SPARQL Endp.

Dynamic

HTTP Lookups

Centralized Linked Data Processing

Federated Linked Data Processing

Centralized vs. Federated Approaches

Centralized Processing Federated Processing

•  Data periodically crawled, gathered, and updated

•  Use of original data sources ensures that data is always „up-to-date“

•  High reliability and controllability •  No control over federation members

•  Inflexible set of data sources •  Ad-hoc integration of remote sources

•  Comprehensive knowledge about data, useful for query optimization

•  Requires careful optimization, but also offers opportunities (parallelization)

Centralized vs. Federated Approaches

Key Observations (1)  Both centralized and federated Linked Data processing have practical use cases (2)  Radically different requirements, challenges, and characteristics

Centralized Processing Federated Processing

•  Data periodically crawled, gathered, and updated

•  Use of original data sources ensures that data is always „up-to-date“

•  High reliability and controllability •  No control over federation members

•  Inflexible set of data sources •  Ad-hoc integration of remote sources

•  Comprehensive knowledge about data, useful for query optimization

•  Requires careful optimization, but also offers opportunities (parallelization)

Benchmarking Linked Data Evaluation

Local Rep.

RDF Data

RDF Data

RDF Data

Query

Central Repository

RDF Data

RDF Data

RDF Data

Query

Federation Layer

SPARQL Endp.

SPARQL Endp.

Dynamic

HTTP Lookups

Centralized Linked Data Processing

Federated Linked Data Processing

BSBM, LUBM, SP2Bench, ... So far no benchmarks proposed

Challenges in Federated Linked Data Benchmarking: Heterogeneity of Use Cases

Query level ¨  (Q1) Query Language

¤  Expressiveness ¤  Complexity

¨  (Q2) Result Completeness

¨  (Q3) Ranking

¨  Various other characteristics ¤  Join types

¤  Result size ¤  ...

Data level ¨  (D1) Physical Distribution

¤  Local vs. remote

¨  (D2) Data Access Interfaces ¤  Native repository

¤  SPARQL Endpoint ¤  Linked Data (HTTP)

¨  (D3) Knowledge about Data Source Existence

¨  (D4) Data Statistics

Query level ¨  (Q1) Query Language

¤  Expressiveness ¤  Complexity

¨  (Q2) Result Completeness

¨  (Q3) Ranking

¨  Various other characteristics ¤  Join types

¤  Result size ¤  ...

Data level ¨  (D1) Physical Distribution

¤  Local vs. remote

¨  (D2) Data Access Interfaces ¤  Native repository

¤  SPARQL Endpoint ¤  Linked Data (HTTP)

¨  (D3) Knowledge about Data Source Existence

¨  (D4) Data Statistics

Need for a flexible benchmark suite rather than “one-size-fits-all“ benchmark scenario!

Challenges in Federated Linked Data Benchmarking: Heterogeneity of Use Cases

FedBench Components (ctd)

Data Sets •  Vary in structuredness,

domain, size, etc. •  Grouped in collections

Data Collections

•  Synthetic Data •  Split into sub-datasets

according to types

Cross-Domain Collection

Life Science Collection SP2Bench Data Collection

FedBench Components (ctd)

Data Sets •  Vary in structuredness,

domain, size, etc. •  Grouped in collections

Queries •  Operate on the data

collections •  Logically grouped

Example Query

SELECT ?pres ?party ?page

WHERE {

?pres rdf:type dbpedia-owl:President .

?pres dbpedia-owl:nationality dbpedia:United_States .

?pres dbpedia-owl:party ?party .

?x nytimes:topicPage ?page .

?x owl:sameAs ?pres

}

List all US presidents including their party and associated news.

Queries

¨  Partially taken from prototype systems, partially designed to capture challenges in federated query processing

¨  Four sets of queries ¤  Life Science

n  Life Science query set (full SPARQL): 7 queries (LS) ¤  Cross Domain

n  Cross Domain query set (full SPARQL): 7 queries (CD) n  Linked Data query set (BGPs): 11 queries (LD)

¤  SP2Bench n  SP2Bench query set (full SPARQL): 14 queries (SP)

¨  Focus on different functional aspects ¤  General federated query processing requirements ¤  Pure Linked Data processing

Queries

Operators: A – AND, U – UNION, O – OPTIONAL, F – FILTER Solution Modifiers: Or – ORDER BY, D – DISTINCT, L – LIMIT, Of – OFFSET

Queries

Benchmark Driver •  Allows to execute FedBench in a unified way •  Java, Open Source à easily adjustable and extensible

FedBench Components (ctd)

Data Sets •  Vary in structuredness,

domain, size, etc. •  Grouped in collections

Queries •  Operate on the data

collections •  Logically grouped

Evaluation Framework

¨  Parametrizable benchmark driver ¨  Implemented in Java using the Sesame framework ¨  Highly customizable via config files

¤ Data and query sets ¤ Number of runs, timeouts ¤ Deployment method of data sets ¤ Metrics (loading time, evaluation time, #requests)

¨  Highly extendable, which makes it easy to connect new systems on demand

Benchmark Driver •  Allows to execute FedBench in a unified way •  Java, Open Source à easily adjustable and extensible

FedBench Components (ctd)

Data Sets •  Vary in structuredness,

domain, size, etc. •  Grouped in collections

Queries •  Operate on the data

collections •  Logically grouped

CSV RDF

Benchmark Results

Benchmark Driver •  Allows to execute FedBench in a unified way •  Java, Open Source à easily adjustable and extensible

FedBench Components (ctd)

Data Sets •  Vary in structuredness,

domain, size, etc. •  Grouped in collections

Queries •  Operate on the data

collections •  Logically grouped

CSV RDF

Benchmark Results

Publishing

•  Wiki-based platform for Linked Data

•  Publishing and discussion of benchmark results

Evaluation

¨  Goal: prove practicability & flexibility of benchmark ¤ Cover a variety of scenarios ¤ Assess first state-of-the-art results ¤  Identify weaknesses and strengths of systems

¨  Measures ¤ Query evaluation time ¤ Number of requests sent to remote sources

¨  Hardware ¤  ILO2 HP server ProLiant DL360 ¤  4Core CPU with 2000MHz ¤  64bit Windows Server 2008, running 64bit JVM 1.6.0_22 ¤  32GB RAM (20GB for federation mediator, rest distributed

among federation members)

Evaluation: Scenario A

¨  “Centralized vs. Federated“ query processing ¤ Scenario A1: Centralized processing

n Sesame 2.3.1

¤ Scenario A2: Local federation n Sesame 2.3.1 + AliBaba

¤ Scenario A3: SPARQL Endpoint federation (HTTP) n Sesame 2.3.1. + AliBaba n SPLENDID from WeST

¨  10min timeout per query ¨  Average over three runs (after warm-up phase)

Scenario A: Life Science Queries

#Requests to Endpoints LS1 LS2 LS3 LS4 LS5 LS6 LS7

Endpoint Federation (AliBaba) 13 61 (410) 21k 17k (130) (876)

Endpoint Federation (SPLENDID) 2 49 9 10 4778 322 4889

Data size: 50M triples in total

Evaluation: Scenario B

¨  Scenario B: Linked Data query set on CD collection ¤ Bottom-up approach ¤ Top-down approach ¤ Mixed approach

¨  Local CumulusRDF Linked Data server ¨  Systems: dedicated prototype implementations* ¨  Major findings

¤ Top-down approach most performant ¤ Mixed approach competitive, bringing the merits of

earlier result reporting

* G. Ladwig, T. Tran: Linked Data Query Processing Strategies. In Proc. ISWC, 2010.

Summary: Central Findings

¨  Effective join ordering often impossible when no intelligent source selection strategy is given

¨  In such cases: often very high number of requests (104+) caused by iterative, nested-loop evaluation strategy of AliBaba

¨  Limited capabilities of Sesame to deal with parallelization cause problems (locking issues)

In the following talk: FedX – a federated query processing system that tackles these issues!

Conclusion

¨  Benchmark flexible enough to cover a wide range of semantic data use cases/applications

¨  Evaluation reveals severe deficiencies of today‘s approaches

¨  Upcoming tasks/future work ¤ General SPARQL 1.1 extensions ¤ SPARQL 1.1 federation extensions ¤ Distributed reasoning

¨  Laid out as community project: you are invited to contribute with your own data & queries!

Questions ?

http://code.google.com/p/fbench/