leveraging textual features for best answer prediction in community-based question answering
TRANSCRIPT
GEORGE GKOTSIS 1 , MARIA LIAKATA 2 , CARLOS PEDRINACI 3 , JOHN DOMINGUE 3
Leveraging Textual Features for Best Answer Prediction in
Community-based Question Answering
1King’s College London2Department of Computer Science, University of Warwick3Knowledge Media Institute, The Open University
ICCSS 2015
Outline
8-11June 2015
Motivation
Problem description
Proposed solution Evaluation
ACQUA
ICCSS 2015 8-11June 2015
Motivation
ICCSS 2015
Questions on social networking sites
8-11June 2015
Recommendations &opinions
Authoritative responses
Expert & Empirical knowledge
ICCSS 2015
Queries on CQA
8-11June 2015
ICCSS 2015 8-11June 2015
Problem description
ICCSS 2015 8-11June 2015
ICCSS 2015
Reputation based Answer Rating based
8-11June 2015
“…we observe significant assortativity in the reputations of co-answerers, relationships between reputation and answer speed, and that the probability of an answer being chosen as the best one strongly depends on temporal characteristics of answer arrivals.”
Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, Jure Leskovec
Discovering Value from Community Activity on Focused Question Answering Sites: A Case Study of Stack Overflow.
KDD 2012
“When available, scoring (or rating) features improve prediction results significantly, which demonstrates the value of community feedback and reputation for identifying valuable answers.”
Grégoire Burel, Yulan He, Harith Alani.
Automatic Identification of Best Answers in Online Enquiry
CommunitiesESWC 2012
State of the art solutions
ICCSS 2015
Best answer prediction in Social Q&A
8-11June 2015
Binary classification problem
Is it solved? Yes, partially
Current solutions depend on:
Answer Ratings
• Score, #comments
Knowledge is Future & Unknown
User Ratings
• User Reputation• UpVotes etc• Preferential
attachment
Knowledge is Past & Not always available
ICCSS 2015
State of the art solutionsSummary
8-11June 2015Our solution
Linguistic User Ratings Answer ratings0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Average Precision
ICCSS 2015
StackExchange network
8-11June 2015
SE “is all about getting answers, it’s not a
discussion forum, there’s no chit-chat”
123 Q&A sites5,622,330 users9.5 million questions16.3 million answers9.3 million visits per day
20 June 2014:
ICCSS 2015 8-11June 2015
StackOver-flow91%
The Rest9%
stackoverflow0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
8,000,000
3,375,817
3,795,276
Non Accepted Answers
Accepted Answers
September 2013 dumpQuestions with Accepted Answers
ICCSS 2015
Shallow Linguistic features
8-11June 2015
Long history, coming from studies on readability1. Average number of characters per word2. Average number of words per sentence3. Number of words in the longest sentence4. Answer length5. Log Likehood:
Pitler &Nenkova, 2008
ICCSS 2015
StackOverflowOverview of shallow features’ evolution
8-11June 2015
ICCSS 2015
Shallow features: Observations
8-11June 2015
Accepted answers tend to be: Longer Differ more from the community vocabulary Contain shorter words Have longer longest sentences Have more words per sentence
But how good are shallow features?
ICCSS 2015
But how good are shallow features?
8-11June 2015
58% macro precision (our baseline)
Possible reasons1. Evolution of language characteristics
Language becomes more eloquent
2. Variance is huge3. Universal classifier looks unreachable, e.g.:
SuperUser average length is 577 Skeptics average length is 2,154
Bad
Good
ICCSS 2015
StackOverflow vrs. SuperUser
8-11June 2015
ICCSS 2015 8-11June 2015
Proposed solution
ICCSS 2015
Objectives
8-11June 2015
Build a classifier which is:
1. Based on linguistic features solely2. Robust
Performs equally well to other classifiers that use user ratings (past knowledge) or answer ratings (future knowledge)
3. Universal Same classifier applicable to as many SE websites
possible (domain agnostic)
ICCSS 2015
Feature discretisationExample for Length
8-11June 2015
Group by question
Question Id
1
5
Answer Id
6
7
Length
2 200
3 150
4 250
150
100
Sort by Length in descending order
Rank
LengthD
1
2
3
1
2
ICCSS 2015
Feature discretisation
8-11June 2015
Category Name Information Gain
Linguistic
Length 0.0226
LongestSentence 0.0121
LL 0.0053
WordsPerSentence 0.0048
CharactersPerWord
0.0052
Linguistic Discretisation
LengthD 0.2168
LongestSentenceD 0.1750
LLD 0.1180
WordsPerSentenceD
0.1404
CharactersPerWordD
0.1162
20x increase
ICCSS 2015
User and answer rating features
8-11June 2015
Category Name
Other
Age
CreationDateD
AnswerCount
User Rating
UserReputation
UserUpVotes
UserDownVotes
UserViews
UserUpDownVotes
Answer rating
Score
CommentCount
ScoreRatio
ICCSS 2015 8-11June 2015
Evaluation
ICCSS 2015
Evaluation Comparison
8-11June 2015
Case Features Used P R FM AUC
1 Linguistic 0.58 0.60 0.56 0.60
2 Linguistic & Discretisation
0.81 0.70 0.74 0.84
3 Linguistic & Discretisation & Other
0.84 0.7 0.76 0.87
4 Linguistic & Other & User Rating(no discretisation)
0.82 0.69 0.75 0.86
5 Linguistic & Other & User Rating(with discretisation)
0.82 0.72 0.77 0.88
6 All features(Answer and User Rating with discretisation)
0.88 0.85 0.86 0.94
ICCSS 2015 8-11June 2015
ACQUAAutomatic Community-based Question Answering
https://acqua.kmi.open.ac.uk/
ICCSS 2015 8-11June 2015
ACQUA - Architecture
ICCSS 2015
ACQUA - Screenshot
8-11June 2015
ICCSS 2015
Read more about our work
8-11June 2015
It’s All in the Content: State of the Art Best Answer Prediction based on Discretisation of Shallow Linguistic Features. WebSci ’14
ACQUA: Automated Community-based Question Answering through the Discretisation of Shallow Linguistic Features. The Journal of Web Science, 1(1) (preprint available)
ICCSS 2015
Thank you
8-11June 2015
http://xkcd.com/386/