Using a Reputation Framework to Identify Community Leaders in Ontology Engineering(short paper)Christophe Debruyne and Niels NijsVrije Universiteit Brussel STARLab

11-09-2013 @ ODBASE 2013

vrijdag 13 september 13

Introduction

‣ Ontology Engineering‣ ... is a social process‣ ... is far from trivial → requires appropriate methods & tools‣ Workflow, roles, and responsibilities

‣ Community Leaders‣ Members in the stakeholder group that drive the ontology project

‣ Problem: the identification of community leaders in ontology engineering for the automatic assignment of responsibilities.

2vrijdag 13 september 13

Introduction

‣ Trust and Reputation Systems‣ Are used to ‣ ... increase the reliability and trust between agents‣ ... improve contribution quality‣ ... build or increase co-operation‣ ...

‣ Have been applied to grant rights/privileges to certain users

‣ Are reputation frameworks suitable for identifying community leaders in an ontology-engineering project?

‣ Approach‣ Identify the characteristics of a community leader‣ Propose framework and “sensors” assigning scores‣ Apply the framework in an ontology engineering experiment‣ Compare the output with results from survey

3vrijdag 13 september 13

Towards a reputation framework

‣ Characteristics of a community leader:(C1) Energy, passionate persistence & optimism (C2) Goal-Driven(C3) Build Trust(C4) Willing to take risks(C5) Pull and communicate with others(C6) Work systematically(C7a) Share knowledge, power and credit(C7b) Work interdependently(C8) Understand others

‣ Sensors for ...(A1) Community activity(A2) “Quality” of interactions(A3) Engage others(A4) Quality of results (annotations, for instance)(A5) Cross-community activity

‣ Objective vs. Subjective Sensors (!)

4

A1 A2 A3 A4 A5

C1 X X X

C2 X X X

C3 X

C4

C5 X X X X

C6 X X

C7a

C7b X

C8 X

“Coverage”

vrijdag 13 september 13

Towards a reputation framework

‣ We define A as the set of all human agents ‣ We define P as the set of all platforms‣ Reputation results R is defined as [0; 100] ∪ {⊘}

‣ Platform configurations ⟨p,wp,S⟩

‣ p a platform in P‣ wp the weight of the platform‣ S a set of sensor configurations ⟨s,ws⟩

‣ s a reputation sensor and ws the weight of the sensor

‣ Compute the reputation scores of a user for a particular platform

5

platform p1platform p1platform p1s1 s2 s3

a1a2a3...

50 25 7560 80 ⊘

⊘ ⊘ ⊘

platform p1result

a1a2a3...

5070⊘

vrijdag 13 september 13

‣ For every user, we compute the result for every platform‣ We remove users with no results‣ We remove platforms with at most 1 result

‣ Compute z-scores for each platform and rescale‣ distance between given value and mean in number of st.devs

‣ Overall score is the weighted average of all rescaled scores

Towards a reputation framework

6

p1 p2 p3 p4a1a2a3a4

60 80 90 ⊘

50 40 ⊘ ⊘

⊘ ⊘ ⊘ ⊘

80 ⊘

p1 p2a1a2a4

60 8050 4080 ⊘

p1 p2a1a2a4

-0,27 1-1,07 -11,34 ⊘

p1 p2a1a2a4

45,5 33,3332,2 66,6772,3 ⊘

z-score rescale

vrijdag 13 september 13

Experiment

‣ Experiment: large ontology engineering project‣ 36 students in the MSc in Computer Science program‣ Held in the context on a course on ontology engineering‣ First develop own information system‣ Then develop ontologies to‣ ... enable semantic interoperability between the systems‣ ... annotate an existing system

‣ Survey‣ On voluntary basis, give at most three names of those considered to

have driven the project‣ Analyze “overlap” between responses and scores obtained by the

reputation framework

7vrijdag 13 september 13

Experiment

8vrijdag 13 september 13

‣ 17 of the 36participantsparticipatedin the survey

‣ Considerableoverlapencouragesfurther investigation

Data and Results

9vrijdag 13 september 13

Limitations and Conclusion

‣ Limitations‣ Interactions outside of the collaborative platform‣ Additional experiments (planned for March 2013)‣ Types of ontology-engineering projects and communities

‣ Conclusions‣ Proposed a means for identifying community leaders in ontology

engineering using a reputation framework‣ Objective and subjective reputation sensors‣ Applied the reputation framework in an ontology-engineering

project and validated the results using a survey

10vrijdag 13 september 13

Fin. Questions?

vrijdag 13 september 13