Transcript
Page 1: Trust Evaluation through User Reputation and Provenance Analysis

Trust Evaluation through User Reputation andProvenance Analysis

Davide Ceolin, Paul Groth, Willem Robert van Hage, Archana

Nottamkandath, Wan Fokkink

Page 2: Trust Evaluation through User Reputation and Provenance Analysis

Outline

● Problem Definition

● Trust estimates as probability distributions

● Reputation-based trust

● Provenance-based trust

● Combining reputation- and provenance-trust

● Results

● Future Work

Page 3: Trust Evaluation through User Reputation and Provenance Analysis

Our Problem

Trusting crowdsourced information

Specifically: video annotations

We can not tag videos automatically (complex, many layers,...), so we refer to the crowd, but crowdsourced annotations need to be evaluated.

Page 4: Trust Evaluation through User Reputation and Provenance Analysis

Our Case Study

Waisda, a video tagging platform.http://waisda.nl

Users score if they tag simultaneously.

Tags are useful (e.g. to index video) but how to trust tags?

Page 5: Trust Evaluation through User Reputation and Provenance Analysis

Measuring tags trustworthiness

How to measure tags trustworthiness?

Let's exploit the game mechanism:

#matches ≈ trustworthiness

However, if a tag did not get any match it is not necessarily wrong.

Page 6: Trust Evaluation through User Reputation and Provenance Analysis

Trust as a Beta Distribution

Page 7: Trust Evaluation through User Reputation and Provenance Analysis

Reputation-based trust

A classical approach:

● Gather all the evidence about a user● Estimate the Beta distribution (user

reputation)● Predict the trustworthiness of the (other)

tags inserted by the user (using the Beta)

Page 8: Trust Evaluation through User Reputation and Provenance Analysis

Our hypothesis

We believe we can go beyond the classical reputation-based trust:

Who ≤ Who + How

Page 9: Trust Evaluation through User Reputation and Provenance Analysis

(How) Provenance-based trust

● “How”-provenance can be used as a stereotype for tags authorship● May be less precise than reputation but more easily available

Page 10: Trust Evaluation through User Reputation and Provenance Analysis

“How-provenance” is composed by several “features” (timestamp, typingDuration, ...)

We use Support Vector Machines to learn the model and make our estimates.

Computing provenance-based trust

Page 11: Trust Evaluation through User Reputation and Provenance Analysis

Combining reputation- and provenance-trust

Is reputation "solid" enough? (E.g. based on 20 obs.)

Use Reputation!(More fine-grained)

Use how-provenance-based estimate!

Page 12: Trust Evaluation through User Reputation and Provenance Analysis

Results

#{tag: Real_Trust(tag)>Threshold = Pred_Trust(tag) > Threshold}Accuracy =

#tags

Page 13: Trust Evaluation through User Reputation and Provenance Analysis

Discussion

Who =* How

Who ≤* Who + How

How ≤* Who + How

(At least in our case study)

=* statistically not different≤* statistically different (less)

Page 14: Trust Evaluation through User Reputation and Provenance Analysis

Future Work

● We plan to apply this approach to other case studies as well

● A generic platform for producing trust assessments based on annotated provenance graphs is under development

● We also plan to merge provenance-based estimates with semantic similarity-based ones in order to provide a tool for a wide range of applications.

Page 15: Trust Evaluation through User Reputation and Provenance Analysis

Thank you!Questions?

[email protected]


Top Related