answer extraction
DESCRIPTION
Answer Extraction. Ling573 NLP Systems and Applications May 16, 2013. Roadmap. Deliverable 3 Discussion What worked Deliverable 4 Answer extraction: Learning answer patterns Answer extraction: classification and ranking Noisy channel approaches. Reminder. Rob Chambers - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/1.jpg)
Answer ExtractionLing573
NLP Systems and ApplicationsMay 16, 2013
![Page 2: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/2.jpg)
RoadmapDeliverable 3 Discussion
What workedDeliverable 4Answer extraction:
Learning answer patternsAnswer extraction: classification and rankingNoisy channel approaches
![Page 3: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/3.jpg)
ReminderRob Chambers
Speech Tech talk & networking eventThis evening: 6:00pm Johnson 203
Speech Technology and Mobile Applications: Speech in Windows Phone
![Page 4: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/4.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:
![Page 5: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/5.jpg)
Deliverable #3Question Answering:
Focus on question processingWhat was tried:
Question classification
![Page 6: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/6.jpg)
Deliverable #3Question Answering:
Focus on question processingWhat was tried:
Question classification Data: Li & Roth, TREC – given or hand-tagged Features: unigrams, POS, NER, head chunks, semantic
info Classifiers: MaxEnt, SVM {+ confidence}
Accuracies: mid-80%s
![Page 7: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/7.jpg)
Deliverable #3Question Answering:
Focus on question processingWhat was tried:
Question classification Data: Li & Roth, TREC – given or hand-tagged Features: unigrams, POS, NER, head chunks, semantic info Classifiers: MaxEnt, SVM {+ confidence}
Accuracies: mid-80%sApplication:
Filtering: Restrict results to have compatible class Boosting: Upweight compatible answers
Gazetteers, heuristics, NER
![Page 8: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/8.jpg)
Question ProcessingWhat was tried:Question Reformulation:
Target handling:Replacement of pronouns, overlapping NPs, etc
Per-qtype reformulations:With backoff to bag-of-words
Inflection generation + irregular verb handling
Variations of exact phrases
![Page 9: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/9.jpg)
What was triedAssorted clean-ups and speedups
Search result caching
Search result cleanup, dedup-ing
Google vs Bing
Code refactoring
![Page 10: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/10.jpg)
What workedTarget integration: most variants helped
Query reformulation: type specific
Qtype boosting, in some cases
Caching for speed/analysis
![Page 11: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/11.jpg)
ResultsMajor improvements over D2 baseline
Most lenient results approach or exceed 0.1 MRR
Current best: ~0.34
Strict results improve, but less than lenient
![Page 12: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/12.jpg)
Deliverable #4Answer extraction/refinement
Fine-grained passages
![Page 13: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/13.jpg)
Deliverable #4Answer extraction/refinement
Fine-grained passages
Lengths not to exceed100-char,250-char
![Page 14: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/14.jpg)
Deliverable #4Answer extraction/refinement
Fine-grained passages
Lengths not to exceed100-char,250-char
Evaluate on 2006 DevtestFinal held-out evaltest from 2007
Released later, no tuning allowed
![Page 15: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/15.jpg)
Deliverable #4Any other refinements across system
Question processing
Retrieval – Web or AQUAINT
Answer processing
Whatever you like to improve final scores
![Page 16: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/16.jpg)
PlugError analysis
Look at training and devtest data
What causes failures?Are the answers in any of the retrieval docs?
Web/TREC If not, why?
Are answers retrieved by not highly ranked?
![Page 17: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/17.jpg)
Last PlugsTonight: 6pm: JHN 102
Jay Waltmunson: Speech Tech and Mobile UW Ling Ph.D.
Presentation and Networking
Tomorrow: 3:30 PCAR 291UW/MS SymposiumHoifung Poon (MSR): Semantic Parsing Chloe Kiddon (UW): Knowledge Extraction w/TML
![Page 18: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/18.jpg)
Answer ExtractionPattern-based Extraction review
Learning Answer Reranking I
Noisy Channel Answer Extraction
Learning Answer Reranking II
![Page 19: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/19.jpg)
Answer Selection by Pattern
Identify question types and termsFilter retrieved passages, replace qterm by tagTry to match patterns and answer spansDiscard duplicates and sort by pattern precision
![Page 20: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/20.jpg)
Pattern Sets WHY-FAMOUS
1.0 <ANSWER> <NAME> called1.0 laureate <ANSWER> <NAME>1.0 by the <ANSWER> , <NAME> ,1.0 <NAME> - the <ANSWER> of1.0 <NAME> was the <ANSWER> of
BIRTHYEAR 1.0 <NAME> ( <ANSWER> - )0.85 <NAME> was born on <ANSWER> ,0.6 <NAME> was born in <ANSWER>0.59 <NAME> was born <ANSWER>0.53 <ANSWER> <NAME> was born
![Page 21: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/21.jpg)
ResultsImproves, though better with web data
![Page 22: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/22.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
![Page 23: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/23.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE type
![Page 24: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/24.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
![Page 25: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/25.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
Long-distance dependencies not practical
![Page 26: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/26.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
Long-distance dependencies not practicalLess of an issue in Web search
Web highly redundant, many local dependenciesMany systems (LCC) use web to validate answers
![Page 27: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/27.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
![Page 28: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/28.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
Requires information about:Answer length, type; logical distance (1-2 chunks)
![Page 29: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/29.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
Requires information about:Answer length, type; logical distance (1-2 chunks)
Also, Can only handle single continuous qterms Ignores caseNeeds handle canonicalization, e.g of names/dates
![Page 30: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/30.jpg)
Integrating Patterns IIFundamental problem:
![Page 31: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/31.jpg)
Integrating Patterns IIFundamental problem:
What if there’s no pattern??
![Page 32: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/32.jpg)
Integrating Patterns IIFundamental problem:
What if there’s no pattern??No pattern -> No answer!!!
More robust solution:Not JUST patterns
![Page 33: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/33.jpg)
Integrating Patterns IIFundamental problem:
What if there’s no pattern??No pattern -> No answer!!!
More robust solution:Not JUST patterns Integrate with machine learning
MAXENT!!!Re-ranking approach
![Page 34: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/34.jpg)
Answering w/Maxent
![Page 35: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/35.jpg)
Feature FunctionsPattern fired:
Binary feature
![Page 36: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/36.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval results
![Page 37: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/37.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)
![Page 38: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/38.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)Question word absent (binary):
No question words in answer span
![Page 39: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/39.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)Question word absent (binary):
No question words in answer spanWord match:
Sum of ITF of words matching b/t questions & sent
![Page 40: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/40.jpg)
Training & TestingTrained on NIST QA questions
Train: TREC 8,9; Cross-validation: TREC-10
5000 candidate answers/questionPositive examples:
NIST pattern matchesNegative examples:
NIST pattern doesn’t matchTest: TREC-2003: MRR: 28.6%; 35.6% exact top 5
![Page 41: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/41.jpg)
Noisy Channel QAEmployed for speech, POS tagging, MT, summ,
etcIntuition:
Question is a noisy representation of the answer
![Page 42: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/42.jpg)
Noisy Channel QAEmployed for speech, POS tagging, MT, summ,
etcIntuition:
Question is a noisy representation of the answerBasic approach:
Given a corpus of (Q,SA) pairsTrain P(Q|SA)Find sentence with answer as
Si,Aij that maximize P(Q|Si,Aij)
![Page 43: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/43.jpg)
QA Noisy ChannelA: Presley died of heart disease at Graceland in 1977,
and..Q: When did Elvis Presley die?
![Page 44: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/44.jpg)
QA Noisy ChannelA: Presley died of heart disease at Graceland in 1977,
and..Q: When did Elvis Presley die?
Goal:Align parts of Ans parse tree to question
Mark candidate answersFind highest probability answer
![Page 45: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/45.jpg)
ApproachAlignment issue:
![Page 46: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/46.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE units
![Page 47: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/47.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE unitsCreate ‘cut’ through parse tree
Every word –or an ancestor – in cutOnly one element on path from root to word
![Page 48: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/48.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE unitsCreate ‘cut’ through parse tree
Every word –or an ancestor – in cutOnly one element on path from root to word
Presley died of heart disease at Graceland in 1977, and..Presley died PP PP in DATE, and..When did Elvis Presley die?
![Page 49: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/49.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as Q
![Page 50: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/50.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as QSolution: (typical MT)
Assign each element a fertility 0 – delete the word; > 1: repeat word that many
times
![Page 51: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/51.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as QSolution: (typical MT)
Assign each element a fertility 0 – delete the word; > 1: repeat word that many times
Replace A words with Q words based on alignmentPermute result to match original QuestionEverything except cut computed with OTS MT code
![Page 52: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/52.jpg)
SchematicAssume cut, answer guess all equally likely
![Page 53: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/53.jpg)
Training Sample Generation
Given question and answer sentencesParse answer sentenceCreate cut s.t.:
Words in both Q & A are preservedAnswer reduced to ‘A_’ syn/sem class labelNodes with no surface children reduced to syn
classKeep surface form of all other nodes
20K TREC QA pairs; 6.5K web question pairs
![Page 54: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/54.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut process
![Page 55: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/55.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in tree
![Page 56: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/56.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in treeWhat’s a bad candidate answer?
![Page 57: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/57.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in treeWhat’s a bad candidate answer?
StopwordsQuestion words!
Create cuts with each answer candidate annotatedSelect one with highest probability by model
![Page 58: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/58.jpg)
Example Answer CutsQ: When did Elvis Presley die?SA1: Presley died A_PP PP PP, and …
SA2: Presley died PP A_PP PP, and ….
SA3: Presley died PP PP in A_DATE, and …
Results: MRR: 24.8%; 31.2% in top 5
![Page 59: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/59.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
![Page 60: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/60.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
Stats based:No restrictions on answer type – frequently ‘it’
![Page 61: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/61.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
Stats based:No restrictions on answer type – frequently ‘it’
Patterns and stats:‘Blatant’ errors:
Select ‘bad’ strings (esp. pronouns) if fit position/pattern
![Page 62: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/62.jpg)
Combining UnitsLinear sum of weights?
![Page 63: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/63.jpg)
Combining UnitsLinear sum of weights?
Problematic:Misses different strengths/weaknesses
![Page 64: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/64.jpg)
Combining UnitsLinear sum of weights?
Problematic:Misses different strengths/weaknesses
Learning! (of course)Maxent re-ranking
Linear
![Page 65: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/65.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
![Page 66: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/66.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
![Page 67: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/67.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
Qtype-specific:Some components better for certain types:
type+mod
![Page 68: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/68.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
Qtype-specific:Some components better for certain types: type+mod
Blatant ‘errors’: no pronouns, when NOT DoW
![Page 69: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/69.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
![Page 70: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/70.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
Combined reranking:All features (after feature selection to 31)
![Page 71: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/71.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
Combined reranking:All features (after feature selection to 31)
Patterns: Exact in top 5: 35.6% -> 43.1%Stats: Exact in top 5: 31.2% -> 41%Manual/knowledge based: 57%
![Page 72: Answer Extraction](https://reader035.vdocument.in/reader035/viewer/2022062410/56815e79550346895dccfbc4/html5/thumbnails/72.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
Combined reranking: All features (after feature selection to 31)
Patterns: Exact in top 5: 35.6% -> 43.1%Stats: Exact in top 5: 31.2% -> 41%Manual/knowledge based: 57%Combined: 57%+