Relevance and Speed of Answers: How Can MAS Answering Systems

Deal With That?

Albert Trias i MansillaGirona

12th july of 2011

Search Paradigms• Library Paradigm:

– Search is based on Catalogues.– Search results on published content.– Trust is based on authority.

• Village Paradigm:– People ask with natural language.– Answers are generated in real-time by anyone in

the community.– Trust is based on intimacy– “It’s not what you know, but who you know”

[Nardi2000]

2

Village Paradigm

3

• C1: “The village paradigm (social search) has some advantages in front of the library paradigm”

– Example: People can address new questions.

• C2: “Village Paradigm can be automated”– Example: Recommender System.

Agents and Q&A Automation

4

• Agents are relevant for Social Search:• Reactivity• Sociability• Proactivity• Autonomy

• Agents enable the construction of information systems from multiple heterogeneous sources [Dignum2006]

• C3: “Agents are a natural approach for social search automation”.

Automated Q&A in Agents Social Network

5

• Approach• P2P Social Network.• Each user is represented by an agent.• Each agent has a FAQ list.

• Agents Can:• Send Questions (asker).• Forward Questions (mediator).• Answer Questions (answerer).

Automated Q&A in Agents Social Network

6

• Related Work:• MARS.

• P2P Multi-Agent Referral System.• 6Search.

• P2P web search (bookmark based)• BFS (Gnutella)• TTL

• [Walter2008]• Recommender System, filtering with trust using BFS.• Trust Path (in base of trust transitivity)

• [Mychlmir2007].• Query Routing in P2P• Ant Optimization techniques.

7

Question Waves

8

Question Waves

Assumptions:• Model does not consider context.• Agents are homogeneous.• Agents are benevolent and cooperative.• Agents are always online.• Answering time is constant.• Static Social Network

9

Question Waves

“T1: Answers relevancy (in village paradigm) is correlated with answer time”

• A question wave is an attempt to find an answer to a question.

• In every attempt, the same question is sent to a subset of acquaintances.

• The expectancy of finding appropriate answers decays after every attempt.

• In P2P, to request a question to all possible peers is not feasible because it can overload the system.

• However, reducing the number of recipients too far can provide the worst results.

10

Question Waves

T=1

T=1T=0.7

T=0.7

Example:

Shelly

Bob

Dale

Gordon

Laura Leo

T=0.2

0 1 5 6 11 40

11

Evaluation

Simulations:

Get answers and sort by answer relevance, compare the rankings using Spearman’s correlation

Agents use 4 waves: T={t1,t2,t3}

1st : after 1 simulation step; Trust > t12nd : after 5 simulation steps; t1 >Trust > t23rd : after 20 simulation steps; t2> Trust > t34th: after 40 simulation steps; t3>Trust > 0

12

Results• Correlation between answers sorted by relevance, and

sorted by the following heuristics:• Answer Distance (D)• Trust of the last sender (Tr).• Receiving Order (H).• Answer Distance and Trust (DT).• Receiving Order and Trust (HT). • Transitive Trust (TT).• Trust of the Last Mediator (TLM).

13

Results• Heuristics Example

T=1

T=1T=0.7

T=0.7

Shelly

Bob

Dale

Gordon

LauraLeo

T=0.2

0 1 5 6 11 40

Laura Gordon BobD 1 2 2

H 1 +1 6 +2 11 +2

Tr 1 0.7 (Dale) 0.7 (Dale)

TT 1 1* 0.7 0.7 * 0.7

TLM 1 1 0.7

14

Evaluation

Ev(a) T D H DT HT Tr TT TLM 𝝑𝝑

mean .8,.7,.6 .14 .67 .17 .66 .14 .52 .9 .66

mean .85,.8,.7 .10 .49 .16 .48 .17 .56 .91 .68

mean .85,.75,.5 .11 .43 .16 .43 .19 .53 .91 .67

mean .85,.7,.5 .12 .56 .16 .55 .16 .52 .9 .67

max .8,.7,.6 .23 .7 .27 .69 .14 .53 .83 .72

max .85,.8,.7 .13 .62 .2 .61 .2 .57 .87 .73

max .85,.75,.5 .15 .6 .23 .59 .22 .58 .87 .72

max .85,.7,.5 .19 .67 .24 .65 .16 .56 .85 .72

Spearman’s Correlation

Evaluation– Using Question Waves behavior and under

our assumptions, answer relevance is correlated with answering time.

– Benefits:• Relevant: answers come ranked• Faster: reduce the burden of questions• Robust: agents search answers persistently.

– Risks:• Different point of view as answer quality.• Trust is needed: Answering always the same is really fast.

15

16

Discussion

Answer velocity is affected by:• Answering time.

• Expertise (Algorithms with faster convergence)• Effort (Example: numerical analysis, more iterations more

precision)• State of answerer • Automated or “Manual” answer?• Implication: Most important tasks will be performed early.

• Communication time.• Answering delay (time after receiving and before trying to

answer)

• Can MAS use answering time to consider answer relevance?• Can MAS behavior be based on reciprocity?

Thank you very much for your

attention

17

18

Evaluation

19

Evaluation

Village Paradigm

• Proverbs:– “A teacher is better than 2 books”– “A library of books does not equal one good teacher”

• Researchers:– Sometimes information only can be accessed asking the right

people [Yu2003].– “It’s not what you know, but who you know” [Nardi2000]

20

Social Search

21

• Social search use social interactions (implicit or explicit) to obtain results.

• (Chi, 2009) Social Search Engines can be classified in:– Social Feedback Systems. (Sorting results).

• Immediate Answer.• Cannot adress new questions.

– Social Answering Systems. (People answers questions)

• Can handle new questions.• Answer not immediate• Experts can get several times same question.

22

Content

•Introduction•Social Search•Agents and Q&A Automation•Automated Q&A in Agents Social Network•Question Waves•Evaluation•Discussion and Future Work

23

Introduction

•Centralized Search Engines provide generally good results, but they go down with atypical searches.

•Interest is Social Networking sites is growing.

•Researchers and Companies show interest in the “village paradigm”.

Automated Q&A in Agents Social Network

24

• Why?• Reuse previous pairs of questions and answers.• 30% of the time that a query was performed, it had been

carried out before by the same user. [Smyth2005]• 70% of the time it was searched before by an acquaintance of

the user. [Smyth2005]

25

Evaluation

set of agents A={a0 , a1, …, ai}, connected in a p2p social network

Method Step For each Received Answers

If Own Question Update result and Trust Else Forward it and update trust

If I have a new Own question Select contacts in contact waves; Program messages

For each received question If I received the same question before

Ignore it Else If I am good enough for answering, Generate Answer Value; Send answer Else

Select contacts in contacts waves; Program messages

Send programmed messages

Ev(a) T D H DT HT Tr TT TLM 𝝑𝝑

mean .8,.7,.6 .12 .51 .12 .49 .10 .38 .72 .66

mean .85,.8,.7 .09 .37 .11 .34 .12 .41 .74 .68

mean .85,.75,.5 .1 .32 .11 .30 .14 .39 .74 .67

mean .85,.7,.5 .1 .42 .11 .39 .12 .38 .73 .67

max .8,.7,.6 .2 .54 .19 .52 .11 .39 .64 .72

max .85,.8,.7 .12 .47 .15 .44 .15 .41 .68 .73

max .85,.75,.5 .13 .46 .16 .43 .16 .42 .69 .72

max .85,.7,.5 .16 .52 .17 .48 .12 .41 .67 .72

26

EvaluationKendall’s Correlation