The CVCE

1 Lars Wieneke, CVCE, Luxembourg

About CUbRIK

n  European Community's Seventh Framework Program FP7-ICT

n  15 European partners n  Multimedia search

processing: Putting humans in the loop

n  Demos: History of Europe and Fashion

2 Lars Wieneke, CVCE, Luxembourg

Point of departure

Images as sources

3 Lars Wieneke, CVCE, Luxembourg

4 Lars Wieneke, CVCE, Luxembourg

Goal: Reconstructing and exploring social ties through historical sources

5 Lars Wieneke, CVCE, Luxembourg

Towards the social graph 4 pillars

1. Close connection to the requirements ofresearchers in European Integration studies

2. Structured and referencable repository ofpersons, events and places in time

3.  Efficient indexation process that enables the association of faces with identities

4. Toolchain for analysis and visualization

6 Lars Wieneke, CVCE, Luxembourg

Towards the social graph Sourcing researcher requirements

Selection of target user group

First draft of the app scenario

Feedback on technical scope

Exploratory interviews

(daily work practices)Second draft of the

app scenario

Focus group(user needs and app

scenarios) Feedback on technical feasability

Lessons learned:issues and features

Specification

Implementation 1. demonstrator

Workshop: Review of app and features

Revised specification

Implementation 2. demonstrator

Evaluation and test

Stage 1

Stage 2

Stage 3

Stage 4

Stage 5

Users

Requirements

Technology

7 Lars Wieneke, CVCE, Luxembourg

Towards the social graph Structured repository

Towards the social graph

Indexation process

8

Raw content

High level features (automatic annotations)

Conflict (e.g., “Image contains

‘Romano Prodi’ ” Confidence = low) ?

Conflict store Conflict manager

Conflict resolution task store

Conflict resolution task: conflict,

required skill, priority, ..

CUbRIK app for Conflict resolution

Game Q&A Crowdtask

Lars Wieneke, CVCE, Luxembourg

9

Towards the social graph Indexation process II

n  Human in the loop added value: n  Verification of identities/places/events ambiguous and temporal only

possible by putting humans in the loop n  Integration of multiple perspectives

n  CUbRIK as an open toolbox allows follow-up and extension through third parties

n  “Vertical” integration: GUI, components, crowdsourcing

integrated in a platform

Lars Wieneke, CVCE, Luxembourg

10

Towards the social graph Visualization and analysis

Lars Wieneke, CVCE, Luxembourg

11

Challenges & Approach

n  Main challenges n  Detection and identification of identities/places/events in time n  Verification of identities/places/events in time n  Analysis of relationships (e.g. co-occurrences) n  Rights aware crawling and storage n  Verification of provenance and license information n  Truth and provenance

n  Approach n  Crowd-sourced verification of detected faces (false positives/negatives) n  Verification of identities through/places/events in time social networks of

experts n  Visual knowledge discovery/exploration n  Integrated rights aware crawling and storage n  Integrated license and provenance management

Lars Wieneke, CVCE, Luxembourg

Towards the social graph Bringing it all together

12 Lars Wieneke, CVCE, Luxembourg

Image Indexation

Media harvesting and upload

Face detection

Face identification

Clickworkers

Crowd Face position

validation

Copyright aware

crawler

Provenance checker

License checker

Content provider

tools

Metadata Entity

extraction

Identity reconciliation

Entity verification & annotation

Entitypedia Integration

CROWD pre-

filtering

Text Indexation

Connection to the CVCE collection

Entity anntation and

extraction

Expert Crowd

Expert CROWD

verification

Entitypedia Integration

CROWD Research Inquieries

Expert Crowd

CROWD Research Inquiry

Social Graph Network Analysis

Graph Visualization

Analysis of the social

graph

Graph Query (old: Query for Entities)

Graph Visualization

Query for entities

Context Expander

Expansion through

documents

Expansion through videos

Expansion through images

Expansion through related entities

Social Graph construction

Social Graph

Creation

Content Analysis and Enrichment Querying Feedback acquisition

and processing Y3 component

Graph Visualization

Query for spatial

constraints

EXP through SIMILAR images

WP Event detection

histoGraph demo

Time for a Demo!

13 Lars Wieneke, CVCE, Luxembourg

Pipelining the CUbRIK components: Human input from click-workers

Great choice for simple tasks: n  Face detection: false positives, false negatives n  Monetary motivation, via www.microtask.com

Poor performance on complex tasks: n  Low resolution images n  Different angles etc. n  Actors recurring over time

14 Lars Wieneke, CVCE, Luxembourg

Pipelining the CUbRIK components: Human input from experts

Capable of complex tasks: n  In-depth knowledge of key actors n  Context knowledge allows inferences

But: Different motivational models! n  Public goods n  Reputation

15 Lars Wieneke, CVCE, Luxembourg

Usage for historians

n  No one truth in history but interpretation, context and discussion

n  Therefore need to represent ambivalence, contradictions and discussion

n  Close ties between data representation (Social graph) and their original context (primary sources)

16 Lars Wieneke, CVCE, Luxembourg

Conclusion

n  Challenges n  What is truth? Humanities vs. Computer Science n  Gathering requirements for tools that haven‘t been

developed yet n  Engaging crowds n  Image copyrights n  Scientific value?

n  Refinement of the application n  Additional datasources n  Improvement of the interface n  Integration of the new components

17 Lars Wieneke, CVCE, Luxembourg

Outlook

18

Outlook

19

Outlook histoGraph

20

WWW.CUBRIKPROJECT.EU

Visit us on

21

@CUBRIKPROJECT

Or follow us on Twitter