Data marketplaces: models and concepts
Hong-Linh Truong
Distributed Systems Group,
Vienna University of Technology
[email protected] http://www.infosys.tuwien.ac.at/staff/truong
1 ASE WS 2012
Advanced Services Engineering,
WS 2012, Lecture 6
Outline
Data marketplaces
Description models
Exchange data agreement
Data contract
ASE WS 2012 2
Data service unit
3
Recall – data service units in
clouds/internet
data
Internet/Cloud
Data service unit
People
data
Data service unit
Things
ASE WS 2012
data data
Data-as-a-Service – service models
Recall – data as a service
ASE WS 2012 4
Storage-as-a-Service
(Basic storage functions)
Database-as-a-Service
(Structured/non-structured
querying systems)
Data publish/subcription
middleware as a service
Sensor-as-a-Service
Private/Public/Hybrid/Community Clouds
deploy
Data marketplaces
More than just DaaS
DaaS focuses on data provisioning features
Data marketplaces
Multiple data providers and consumers
Multiple DaaS
Complex interactions among DaaS, data providers
and consumers
Complex billing and pricing models
Market dynamics
ASE WS 2012 5
WHAT ARE IMPORTANT ISSUES IN DATA
MARKETPLACES?
Discussion time
ASE WS 2012 6
DAAS DESCRIPTION MODEL
Some important issues
ASE WS 2012 7
DATA AGREEMENT EXCHANGE
DATA CONTRACT
Description Model for DaaS (1)
State of the art:
Providers have their own way to describe DaaS,
mainly in HTML
Existing service description techniques are not
adequate in supporting description for DaaS
Problems
Service and data discovery cannot be done
automatically
On-demand data integration, service integration, and
query optimization cannot be supported well.
Service/data information and DaaS engineering
cannot be tied. ASE WS 2012 8
Description Model for DaaS (2)
Which levels must be covered?
ASE WS 2012 9
Data
items
Data
items
Data
items
Data resource
Data
assets
Data resource Data resource
Data resource Data resource
Consumer
Consumer
DaaS
Here
Description Model for DaaS – types
of information
Which types of information must be covered?
ASE WS 2012 10
Quality of
data Ownership
Price License ....
Service
interface Service
license Quality of
service ....
DEMOS – a description model for
Data-as-a-Service
ASE WS 2012 11
See prototype:
http://www.infosys.tuwien.ac.at/
prototype/SOD1/demods/
Quang Hieu Vu, Tran Vu Pham, Hong
Linh Truong,, Schahram Dustdar,
Rasool Asal: DEMODS: A Description
Model for Data-as-a-Service. AINA
2012: 605-612
Description model and data
marketplaces
ASE WS 2012 12
DEMODS – prototype (1)
ASE WS 2012 13
DEMODS – prototype (2)
ASE WS 2012 14
Check: http://demodsmanagement.appspot.com/
WHICH TYPES OF DAAS INFORMATION
ARE DYNAMIC? AND THEIR IMPACT ON
DESCRIPTION MODELS?
Discussion time
ASE WS 2012 15
Exchange data agreement (1)
ASE WS 2012 16
DaaS
Consumer
DaaS
Sensor
DaaS
Consumer DaaS provider Data
provider
How they interact w.r.t. data concerns?
How their data agreements look like?
Exchange data agreement (2)
Lack of models and protocols for data
agreement in data marketplaces
Constraints for data usage are not clear
Inadequate data/service description → hindering data
selection and integration
Existing techniques are not adequate for
dynamic data agreement exchange in data
marketplaces
Need generic exchange models suitable for different
ways of data provisioning in data marketplaces
ASE WS 2012 17
Data Agreement Exchange as a
Service (DAES)
Metamodel for data agreement exchange
Techniques for enriching and associating data
assets with agreement terms
Interaction models for data agreement exchange
Hong Linh Truong, Schahram Dustdar, Joachim Götze, Tino Fleuren, Paul Müller, Salah-Eddine Tbahriti, Michael Mrissa,
Chirine Ghedira: Exchanging Data Agreements in the DaaS Model. APSCC 2011: 153-160
ASE WS 2012 18
Metamodel for data agreements
Different
category of
agreements
Licensing,
privacy, quality
of data
Extensions
Languages
Different types
of agreements
Different
specifications
ASE WS 2012 19
Associating data with data
agreements
Solutions
(a) directly inserting agreements into data assets
(b) providing two-step access to agreements and data
assets
(c) linking data agreements to the description of DaaS
(d) linking data agreements to the message sent by
DaaS
ASE WS 2012 20
Possible interaction models for data
enriched with data agreements
ASE WS 2012 21
DAES – conceptual architecture
Jersey, JAX-RS Restful WS Weblogic
Using URIs to identify agreements ASE WS 2012 22
DAES – managed information
Specific applications: agreement creation, agreement validation,
agreement compatibility analysis, agreement management
Implementation: Jersey, JAX-RS Restful WS Weblogic
ASE WS 2012 23
Illustrating examples – insert
agreement into data asset
A pay-per-use consumer uses dataAPI of DaaS
search for data
The consumer pays the use APIs
Each call can return different types of data
Example with
People Search in
Infochimps
But a strong consequence
for data service engineering
techniques: dealing with
elastic requirements!
ASE WS 2012 24
Illustrating examples – link
agreements to geospatial data Domain-specific DaaS: different agreements for different data requests
Vector data of geographic features via Web-Feature-Service (WFS)
Terrain elevation data via Web-Coverage Services (WCS)
ASE WS 2012 25
Illustrating examples – link
agreements to geospatial data
Consumers can interpret and
reason if the data can be
used for specific purposes
ASE WS 2012 26
Illustrative examples – develop an
app for policy compliance (1)
ASE WS 2012 27
Illustrative examples – develop an
app for policy compliance (2) Configuration
Results
ASE WS 2012 28
HOW NEAR-REALTIME DATA IMPACTS ON
DATA AGREEMENT EXCHANGE?
Discussion time
ASE WS 2012 29
Data contract
How to specific data contract?
ASE WS 2012 30
Data
items
Data
items
Data
items
Data resource
Data
assets
Data resource Data resource
Data resource Data resource
Consumer
Consumer
DaaS
Data contracts
Give a clear information about data usage
Have a remedy against the consumer where the
circumstances are such that the acts complained
of do not
Limit the liability of data providers in case of
failure of the provided data;
Specify information on data delivery,
acceptance, and payment
31 ASE WS 2012
32
Data contracts
Well-researched contracts for services but not
for DaaS and data marketplaces
But service APIs != data APIs =! data assets
Several open questions
Right to use data? Quality of data in the data
agreement? Search based on data contract? Etc.
➔ Require extensible models
➔ Capture contractual terms for data contracts
➔ Support (semi-)automatic data service/data selection
techniques.
Hong-Linh Truong, Marco Comerio, Flavio De Paoli, G.R. Gangadharan, Schahram Dustdar, "Data Contracts for
Cloud-based Data Marketplaces ", International Journal of Computational Science and Engineering, 2012 Vol.7, No.4,
pp.280 - 295
ASE WS 2012
Study of main data contract terms
Data rights
Derivation, Collection, Reproduction, Attribution
Quality of Data (QoD)
Not mentioned, Not clear how to establish QoD metrics
Regulatory Compliance
Sarbanes-Oxley, EU data protection directive, etc.
Pricing model
Different models, pricing for data APIs and for data assets
Control and Relationship
Evolution terms, support terms, limitation of liability, etc
33
Most information is in human-readable form
ASE WS 2012
34
Data contract study
ASE WS 2012
35
Developing data contracts in cloud-
based data marketplaces
Follow community-based approach for data
contract
Propose generic structures to represent data
contract terms and abstract data contracts
Develop frameworks for data contract applications
Incorporate data contracts into data-as-a-service
description
Develop data contract applications
ASE WS 2012
36
Community view on data contract
development
Community users can develop:
Term categories, term names, values, and units
Rules for data contracts
Common contract and contract fragments
Community users =!
novice users
ASE WS 2012
37
Representing data contract terms
Contract term: (termName,termValue)
Term name: common terms or user-specific terms
Term value: a single value, a set, or a range
ASE WS 2012
38
Structuring abstract data contracts
Concrete data contracts
can be in RDF, XML or
JSON
generates
Use Identifiers and
Tags for identifying
and searches
ASE WS 2012
39
Development of contract
applications
Main applications:
Data contract compatibility evaluation, data contract
composition
Some common steps
Extract DCTermType in TermCategoryType
Extact comprable terms from all contracts,
- e.g., dataRight: Derivation, Composition and Reproduction
Use evaluation rules associated with DCTermType
from from rule repositories
Execute rules by passing comparable terms to rules
Aggregate results
ASE WS 2012
Prototype
RDF for representing term categories, term
names, term values, units
Allegro Graph for storing contract knowledge
ASE WS 2012 40
41
Illustrating examples
A large sustainability monitoring data platform
shows how green buildings are
Real-time total and per capita of CO2 emission
of monitored building
Open government data about CO2 per capita at
national level
We created contracts from
Open Data Commons Attribution License
Open Government License
ASE WS 2012
42
Existing
common
knowledge
about Open
Data
Commons
ASE WS 2012
43
Step 2: provide OpenBuildingCO2
OpenBuildingCO2 by
modifying quality of
data and data right
OpenGov for
government data
Data contract for green building data
ASE WS 2012
44
Experiments – composing data
contract terms
ASE WS 2012
CAN WE AUTOMATICALLY GENERATE
DATA CONTRACTS FOR NEAR-REALTIME
DATA?
Discussion time
ASE WS 2012 45
Exercises
Read mentioned papers
Examine existing data marketplaces and write
DEMODS-based specification for some of them
Develop some specific data contracts for open
government data
Work on some algorithms for checking data
contract compatiblity
ASE WS 2012 46
47
Thanks for your attention
Hong-Linh Truong
Distributed Systems Group
Vienna University of Technology
http://www.infosys.tuwien.ac.at/staff/truong
ASE WS 2012