ling 573 deliverable 3 jonggun park haotian he maria antoniak ron lockwood
TRANSCRIPT
![Page 1: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/1.jpg)
LING 573 Deliverable 3
Jonggun Park Haotian He Maria Antoniak Ron Lockwood
![Page 2: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/2.jpg)
Closed Class filters 14
• Animals, colors, companies, continents, countries, sports team, languages, occupations, periodic table, race, us-cities, us-presidents, us-states, and us-universities.
![Page 3: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/3.jpg)
Query EXPAANSSION!
![Page 4: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/4.jpg)
Query Expansion
• Who is the president of the United States?• President united states nations council
• How long did it take to build the Tower of Pisa?• long build tower pisa women’s station
![Page 5: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/5.jpg)
Question Classification
Software package: Mallet
Classification algorithms: MaxEnt, NaiveBayes, Winnow, DecisionTree
Training Data:
- TREC-2004.xml
- Training set 5 (5500 labeled questions) (Li & Roth)
Test Data:
- TREC-2005.xml
- Testing set (Li & Roth)
![Page 6: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/6.jpg)
Feature selection:
- Unigram
- Bigram
- Trigram
- Question word
- NER tags
![Page 7: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/7.jpg)
Conclusion:
Maximum accuracy:
TREC-2005 as test file: 0.8535911602209945
- MaxEnt, Unigram + Bigram + Wh-words
TREC-10 as test file: 0.854
- MaxEnt, Unigram + Bigram + Wh-words
![Page 8: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/8.jpg)
Other findings:
1. Trigram does not helps and drags the accuracy down.
2. NER feature does not helps and causes a slight drop-down.
![Page 9: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/9.jpg)
Web Boosting
• Resources: jsoup, Bing.com• Query: original question + target string• Results: top 50 web snippets, stored in a text file
![Page 10: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/10.jpg)
Web Boosting Challenges and Successes
• Which search engine or answer website to use?• How to avoid throttling?• How to integrate results into our system?• How to edit results to make them more useful for
our answer ranking system?
![Page 11: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/11.jpg)
Main Changes• Use web query as input to the redundancy-based
answer extraction engine • This replaces our paragraph based index
• Answer type classification now feeds into answer extraction• Filtering of candidate answers by answer type in
combination with NER on the answers• Following types are handled: NUM, LOC, HUM, ENTY
![Page 12: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/12.jpg)
Main changes (continued)
• Filtering of closed class questions using lists• E.g. pro sports teams, colors, etc.
• Filtering out of terms with occurrences in less than 2 snippets• Return 250 char. answer instead of 1-4 words
![Page 13: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/13.jpg)
Answer Extraction Details• Input to the Extraction Engine• Query word list• Stop-word list• Focus-word list (e.g. meters, liters, miles, etc.)• Passage list – the paragraph results of the query
1. N-gram generation and occurrence counting2. Filtering out stop words and query words 3. Filter by answer type
![Page 14: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/14.jpg)
Answer Extraction Details
4. Combining unigram counts with n-gram counts5. Weighting candidates with idf scores6. Re-rank candidates
1. Eliminate ones that don’t have evidence in at least 2 snippets
2. Eliminate ones that don’t match a closed class list (for certain questions.)
7. Verifying candidates in documents3. Use bag of words query from the candidate sub-
snippet + query words against Lucene index
![Page 15: LING 573 Deliverable 3 Jonggun Park Haotian He Maria Antoniak Ron Lockwood](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f425503460f94c618e0/html5/thumbnails/15.jpg)
Results
D2: strict = 0.01lenient = 0.064
D3: strict = 0.133lenient = 0.371