Download - Question Classification
![Page 1: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/1.jpg)
Question Classification
Ling573NLP Systems and Applications
April 25, 2013
![Page 2: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/2.jpg)
Deliverable #3Posted: Code & results due May 10
Focus: Question processingClassification, reformulation, expansion, etc
Additional: general improvement motivated by D#2
![Page 3: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/3.jpg)
Question Classification:
Li&Roth
![Page 4: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/4.jpg)
RoadmapMotivation:
![Page 5: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/5.jpg)
Why Question Classification?
![Page 6: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/6.jpg)
Why Question Classification?
Question classification categorizes possible answers
![Page 7: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/7.jpg)
Why Question Classification?
Question classification categorizes possible answersConstrains answers types to help find, verify answer
Q: What Canadian city has the largest population?
Type?
![Page 8: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/8.jpg)
Why Question Classification?
Question classification categorizes possible answersConstrains answers types to help find, verify answer
Q: What Canadian city has the largest population?
Type? -> CityCan ignore all non-city NPs
![Page 9: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/9.jpg)
Why Question Classification?
Question classification categorizes possible answersConstrains answers types to help find, verify answer
Q: What Canadian city has the largest population?
Type? -> CityCan ignore all non-city NPs
Provides information for type-specific answer selectionQ: What is a prism?Type? ->
![Page 10: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/10.jpg)
Why Question Classification?
Question classification categorizes possible answersConstrains answers types to help find, verify answer
Q: What Canadian city has the largest population?
Type? -> CityCan ignore all non-city NPs
Provides information for type-specific answer selectionQ: What is a prism?Type? -> Definition
Answer patterns include: ‘A prism is…’
![Page 11: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/11.jpg)
Challenges
![Page 12: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/12.jpg)
ChallengesVariability:
What tourist attractions are there in Reims?What are the names of the tourist attractions in
Reims?What is worth seeing in Reims?
Type?
![Page 13: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/13.jpg)
ChallengesVariability:
What tourist attractions are there in Reims?What are the names of the tourist attractions in
Reims?What is worth seeing in Reims?
Type? -> Location
![Page 14: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/14.jpg)
ChallengesVariability:
What tourist attractions are there in Reims?What are the names of the tourist attractions in
Reims?What is worth seeing in Reims?
Type? -> Location
Manual rules?
![Page 15: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/15.jpg)
ChallengesVariability:
What tourist attractions are there in Reims?What are the names of the tourist attractions in
Reims?What is worth seeing in Reims?
Type? -> Location
Manual rules?Nearly impossible to create sufficient patterns
Solution?
![Page 16: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/16.jpg)
ChallengesVariability:
What tourist attractions are there in Reims?What are the names of the tourist attractions in
Reims?What is worth seeing in Reims?
Type? -> Location
Manual rules?Nearly impossible to create sufficient patterns
Solution?Machine learning – rich feature set
![Page 17: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/17.jpg)
Approach Employ machine learning to categorize by answer
typeHierarchical classifier on semantic hierarchy of types
Coarse vs fine-grained Up to 50 classes
Differs from text categorization?
![Page 18: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/18.jpg)
Approach Employ machine learning to categorize by answer
typeHierarchical classifier on semantic hierarchy of types
Coarse vs fine-grained Up to 50 classes
Differs from text categorization?Shorter (much!)Less information, but Deep analysis more tractable
![Page 19: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/19.jpg)
ApproachExploit syntactic and semantic information
Diverse semantic resources
![Page 20: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/20.jpg)
ApproachExploit syntactic and semantic information
Diverse semantic resourcesNamed Entity categoriesWordNet senseManually constructed word listsAutomatically extracted semantically similar word
lists
![Page 21: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/21.jpg)
ApproachExploit syntactic and semantic information
Diverse semantic resourcesNamed Entity categoriesWordNet senseManually constructed word listsAutomatically extracted semantically similar word
lists
Results:Coarse: 92.5%; Fine: 89.3%Semantic features reduce error by 28%
![Page 22: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/22.jpg)
Question Hierarchy
![Page 23: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/23.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only :
![Page 24: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/24.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only :Small set of entity types, set of handcrafted rules
![Page 25: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/25.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only :Small set of entity types, set of handcrafted rules
Note: Webclopedia’s 96 node taxo w/276 manual rules
![Page 26: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/26.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only :Small set of entity types, set of handcrafted rules
Note: Webclopedia’s 96 node taxo w/276 manual rules
Learning approaches can learn to generalizeTrain on new taxonomy, but
![Page 27: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/27.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only :Small set of entity types, set of handcrafted rules
Note: Webclopedia’s 96 node taxo w/276 manual rules
Learning approaches can learn to generalizeTrain on new taxonomy, but
Someone still has to label the data…
Two step learning: (Winnow)Same features in both cases
![Page 28: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/28.jpg)
Learning a Hierarchical Question Classifier
Many manual approaches use only : Small set of entity types, set of handcrafted rules
Note: Webclopedia’s 96 node taxo w/276 manual rules
Learning approaches can learn to generalize Train on new taxonomy, but
Someone still has to label the data…
Two step learning: (Winnow) Same features in both cases
First classifier produces (a set of) coarse labels Second classifier selects from fine-grained children of coarse tags
generated by the previous stageSelect highest density classes above threshold
![Page 29: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/29.jpg)
Features for Question Classification
Primitive lexical, syntactic, lexical-semantic featuresAutomatically derivedCombined into conjunctive, relational featuresSparse, binary representation
![Page 30: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/30.jpg)
Features for Question Classification
Primitive lexical, syntactic, lexical-semantic featuresAutomatically derivedCombined into conjunctive, relational featuresSparse, binary representation
WordsCombined into ngrams
![Page 31: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/31.jpg)
Features for Question Classification
Primitive lexical, syntactic, lexical-semantic featuresAutomatically derivedCombined into conjunctive, relational featuresSparse, binary representation
WordsCombined into ngrams
Syntactic features:Part-of-speech tagsChunksHead chunks : 1st N, V chunks after Q-word
![Page 32: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/32.jpg)
Syntactic Feature ExampleQ: Who was the first woman killed in the Vietnam
War?
![Page 33: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/33.jpg)
Syntactic Feature ExampleQ: Who was the first woman killed in the Vietnam
War?
POS: [Who WP] [was VBD] [the DT] [first JJ] [woman NN] [killed VBN] [in IN] [the DT] [Vietnam NNP] [War NNP] [? .]
![Page 34: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/34.jpg)
Syntactic Feature ExampleQ: Who was the first woman killed in the Vietnam
War?
POS: [Who WP] [was VBD] [the DT] [first JJ] [woman NN] [killed VBN] {in IN] [the DT] [Vietnam NNP] [War NNP] [? .]
Chunking: [NP Who] [VP was] [NP the first woman] [VP killed] [PP in] [NP the Vietnam War] ?
![Page 35: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/35.jpg)
Syntactic Feature ExampleQ: Who was the first woman killed in the Vietnam
War?
POS: [Who WP] [was VBD] [the DT] [first JJ] [woman NN] [killed VBN] {in IN] [the DT] [Vietnam NNP] [War NNP] [? .]
Chunking: [NP Who] [VP was] [NP the first woman] [VP killed] [PP in] [NP the Vietnam War] ?
Head noun chunk: ‘the first woman’
![Page 36: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/36.jpg)
Semantic FeaturesTreat analogously to syntax?
![Page 37: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/37.jpg)
Semantic FeaturesTreat analogously to syntax?
Q1:What’s the semantic equivalent of POS tagging?
![Page 38: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/38.jpg)
Semantic FeaturesTreat analogously to syntax?
Q1:What’s the semantic equivalent of POS tagging?
Q2: POS tagging > 97% accurate; Semantics? Semantic ambiguity?
![Page 39: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/39.jpg)
Semantic FeaturesTreat analogously to syntax?
Q1:What’s the semantic equivalent of POS tagging?
Q2: POS tagging > 97% accurate; Semantics? Semantic ambiguity?
A1: Explore different lexical semantic info sources
Differ in granularity, difficulty, and accuracy
![Page 40: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/40.jpg)
Semantic FeaturesTreat analogously to syntax?
Q1:What’s the semantic equivalent of POS tagging?Q2: POS tagging > 97% accurate;
Semantics? Semantic ambiguity?
A1: Explore different lexical semantic info sourcesDiffer in granularity, difficulty, and accuracy
Named Entities WordNet SensesManual word listsDistributional sense clusters
![Page 41: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/41.jpg)
Tagging & AmbiguityAugment each word with semantic category
What about ambiguity?E.g. ‘water’ as ‘liquid’ or ‘body of water’
![Page 42: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/42.jpg)
Tagging & AmbiguityAugment each word with semantic category
What about ambiguity?E.g. ‘water’ as ‘liquid’ or ‘body of water’Don’t disambiguate
Keep all alternatives Let the learning algorithm sort it outWhy?
![Page 43: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/43.jpg)
Semantic CategoriesNamed Entities
Expanded class set: 34 categoriesE.g. Profession, event, holiday, plant,…
![Page 44: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/44.jpg)
Semantic CategoriesNamed Entities
Expanded class set: 34 categoriesE.g. Profession, event, holiday, plant,…
WordNet: IS-A hierarchy of sensesAll senses of word + direct hyper/hyponyms
![Page 45: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/45.jpg)
Semantic CategoriesNamed Entities
Expanded class set: 34 categoriesE.g. Profession, event, holiday, plant,…
WordNet: IS-A hierarchy of senses All senses of word + direct hyper/hyponyms
Class-specific words Manually derived from 5500 questions
E.g. Class: Food {alcoholic, apple, beer, berry, breakfast brew butter candy cereal
champagne cook delicious eat fat ..} Class is semantic tag for word in the list
![Page 46: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/46.jpg)
Semantic TypesDistributional clusters:
Based on Pantel and LinCluster based on similarity in dependency relationsWord lists for 20K English words
![Page 47: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/47.jpg)
Semantic TypesDistributional clusters:
Based on Pantel and LinCluster based on similarity in dependency
relationsWord lists for 20K English words
Lists correspond to word sensesWater:
Sense 1: { oil gas fuel food milk liquid} Sense 2: {air moisture soil heat area rain} Sense 3: {waste sewage pollution runoff}
![Page 48: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/48.jpg)
Semantic TypesDistributional clusters:
Based on Pantel and LinCluster based on similarity in dependency
relationsWord lists for 20K English words
Lists correspond to word sensesWater:
Sense 1: { oil gas fuel food milk liquid} Sense 2: {air moisture soil heat area rain} Sense 3: {waste sewage pollution runoff}
Treat head word as semantic category of words on list
![Page 49: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/49.jpg)
EvaluationAssess hierarchical coarse->fine classificationAssess impact of different semantic featuresAssess training requirements for diff’t feature
set
![Page 50: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/50.jpg)
EvaluationAssess hierarchical coarse->fine classificationAssess impact of different semantic featuresAssess training requirements for diff’t feature
setTraining:
21.5K questions from TREC 8,9; manual; USC dataTest:
1K questions from TREC 10,11
![Page 51: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/51.jpg)
EvaluationAssess hierarchical coarse->fine classificationAssess impact of different semantic featuresAssess training requirements for diff’t feature setTraining:
21.5K questions from TREC 8,9; manual; USC dataTest:
1K questions from TREC 10,11Measures: Accuracy and class-specific precision
![Page 52: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/52.jpg)
ResultsSyntactic features only:
POS useful; chunks useful to contribute head chunksFine categories more ambiguous
![Page 53: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/53.jpg)
ResultsSyntactic features only:
POS useful; chunks useful to contribute head chunksFine categories more ambiguous
Semantic features:Best combination: SYN, NE, Manual & Auto word lists
Coarse: same; Fine: 89.3% (28.7% error reduction)
![Page 54: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/54.jpg)
ResultsSyntactic features only:
POS useful; chunks useful to contribute head chunksFine categories more ambiguous
Semantic features: Best combination: SYN, NE, Manual & Auto word lists
Coarse: same; Fine: 89.3% (28.7% error reduction)
Wh-word most common class: 41%
![Page 55: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/55.jpg)
![Page 56: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/56.jpg)
![Page 57: Question Classification](https://reader036.vdocument.in/reader036/viewer/2022062305/56815ea2550346895dcd3073/html5/thumbnails/57.jpg)
ObservationsEffective coarse and fine-grained categorization
Mix of information sources and learningShallow syntactic features effective for coarseSemantic features improve fine-grained
Most feature types help WordNet features appear noisy Use of distributional sense clusters dramatically
increases feature dimensionality