question answering (and textual...

QuestionAnswering(andTextualEntailment)

Prof.SameerSinghCS295:STATISTICALNLP

WINTER2017

March14,2017

BasedonslidesfromDanJurafsky,YejinChoi,StephenClark,DanKlein,Niranjan Balasubramanian,andeveryoneelsetheycopiedfrom.

Upcoming…

• Homework4wasdueonlastnight• Lowestgradeofthehomeworks willbedroppedHomework

• Finalreportdueinaweek:March20,2017• Instructionscomingsoon:ACLstyle,5pages(+references)Project

• Papersummariesduetonight• Summary2gradedSummaries

CS295:STATISTICALNLP(WINTER2017) 2

TA/InstructorEvaluationsareavailable!

Outline

QuestionAnswering

TextualEntailment


IR-BasedQASystem

OtherExtensions

QuestionsinModernSystemsFactoidquestions◦ Whowrote“TheUniversalDeclarationofHumanRights”?◦ Howmanycaloriesarethereintwoslicesofapplepie?◦ Whatistheaverageageoftheonsetofautism?◦ WhereisAppleComputerbased?

Complex(narrative)questions:◦ Inchildrenwithanacutefebrileillness,whatistheefficacyofacetaminopheninreducingfever?

◦ WhatdoscholarsthinkaboutJefferson’spositionondealingwithpirates?


Commercialsystems:mainlyfactoidquestions

WhereistheLouvreMuseumlocated? InParis,France

What’stheabbreviation forlimitedpartnership? L.P.

What arethenamesofOdin’sravens? Huginn andMuninn

What currencyisusedinChina? Theyuan

Whatkindofnutsareusedinmarzipan? almonds

WhatinstrumentdoesMaxRoachplay? drums

WhatisthetelephonenumberforStanfordUniversity? 650-723-2300


ParadigmsforQA

IR-basedapproaches◦ TREC;IBMWatson;Google

Knowledge-basedandHybridapproaches◦ IBMWatson;AppleSiri;WolframAlpha;TrueKnowledgeEvi


Manyquestionscanalreadybeansweredbywebsearch


IR-basedQuestionAnswering


IR-basedFactoidQA


DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


Knowledge-basedQA(Siri)Buildasemanticrepresentationofthequery◦ Times,dates,locations,entities,numericquantities

Mapfromthissemanticstoquerystructureddataorresources◦ Geospatialdatabases◦ Ontologies(Wikipediainfoboxes,dbPedia,WordNet,Yago)◦ Restaurantreviewsourcesandreservationservices◦ Scientificdatabases


Hybridapproaches(Watson)Buildashallowsemanticrepresentationofthequery

GenerateanswercandidatesusingIRmethods◦ Augmentedwithontologiesandsemi-structureddata

Scoreeachcandidateusingricherknowledgesources◦ Geospatialdatabases◦ Temporalreasoning◦ Taxonomicalclassification


IBM’sWatson

WonJeopardy onFebruary16,2011!

13

WILLIAMWILKINSON’S“ANACCOUNTOFTHEPRINCIPALITIESOF

WALLACHIAANDMOLDOVIA”INSPIREDTHISAUTHOR’SMOSTFAMOUSNOVEL

BramStoker

CS295:STATISTICALNLP(WINTER2017)

MotivationforWatson


SingleSourceisnotSufficient


WatsonArchitecture


. . .

Models

Answer & Confidence

Question

Evidence

Sources

Models

Models

Models

Models

ModelsPrimarySearch

CandidateAnswer

Generation

Answer Sources Evidence

RetrievalEvidence Scoring

Learned Modelshelp combine and

weigh the Evidence

Hypothesis and Evidence Scoring

Merging &

Ranking

Synthesis

WatsonPerformance


Outline

QuestionAnswering

TextualEntailment


IR-BasedQASystem

OtherExtensions

IR-basedFactoidQA



DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation


Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


IR-basedFactoidQAQUESTIONPROCESSING◦ Detectquestiontype,answertype,focus,relations◦ Formulatequeriestosendtoasearchengine

PASSAGERETRIEVAL◦ Retrieverankeddocuments◦ Breakintosuitablepassagesandrerank

ANSWERPROCESSING◦ Extractcandidateanswers◦ Rankcandidates

◦ usingevidencefromthetextandexternalsources


FactoidQ/A



DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation


Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


QuestionProcessingAnswerTypeDetection◦ Decidethenamedentitytype(person,place)oftheanswer

QueryFormulation◦ ChoosequerykeywordsfortheIRsystem

QuestionTypeclassification◦ Isthisadefinitionquestion,amathquestion,alistquestion?

FocusDetection◦ Findthequestionwordsthatarereplacedbytheanswer

RelationExtraction◦ Findrelationsbetweenentitiesinthequestion


QuestionProcessing

AnswerType:USstate

Query:twostates,border,Florida,north

Focus:thetwostates

Relations:borders(Florida,?x,north)


They’rethetwostatesyoucouldbereenteringifyou’recrossingFlorida’snorthernborder

AnswerTypes:NamedEntities

WhofoundedVirginAirlines?PERSON

WhatCanadiancityhasthelargestpopulation?CITY


PartofAnswerTypeTaxonomy


LOCATION

NUMERIC

ENTITY HUMAN

ABBREVIATIONDESCRIPTION

country city state

datepercent

money

sizedistance

individual

title

group

food

currency

animal

definition

reason expression

abbreviation

Li,Roth.LearningQuestionClassifiers.COLING(2002)

AnswerTypes


MoreAnswerTypes


AnswertypesinWatson

2500answertypesin20,000Jeopardyquestionsample◦ Themostfrequent200answertypescover<50%ofdata

The40mostfrequentJeopardyanswertypes

he,country,city,man,film,state,she,author,group,here,company,president,capital,star,novel,character,woman,river,island,king,song,part,series,sport,singer,actor,play,team,show,actress,animal,presidential,composer,musical,nation,book,title,leader,game

CS295:STATISTICALNLP(WINTER2017) 28Ferrucci etal.BuildingWatson:AnOverviewoftheDeepQA Project.AIMagazine.2010

AnswerTypeDetection


Hand-writtenRules

Regularexpression-basedrulescangetsomecases:◦ Who{is|was|are|were}PERSON

Otherrulesusethequestionheadword:◦ (theheadwordofthefirstnounphraseafterthewh-word)◦ WhichcityinChinahasmostforeignfinancialcompanies?◦ WhatisthestateflowerofCalifornia?

MachineLearning

QuestionwordsandphrasesPart-of-speechtagsParsefeatures(headwords)NamedEntitiesSemanticallyrelatedwords

FactoidQ/A



DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation


Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


KeywordSelectionAlgorithm1.Selectallnon-stopwordsinquotations2.SelectallNNPwordsinrecognizednamedentities3.Selectallcomplexnominals withtheiradjectivalmodifiers4.Selectallothercomplexnominals5.Selectallnounswiththeiradjectivalmodifiers6.Selectallothernouns7.Selectallverbs8.Selectalladverbs9.SelecttheQFWword(skippedinallprevioussteps)10.Selectallotherwords

CS295:STATISTICALNLP(WINTER2017) 31Moldovan,Harabagiu,Pasca,Mihalcea,Goodrum,GirjuandRus.TREC(1999)

Choosingkeywords


Whocoinedtheterm“cyberspace”inhisnovel“Neuromancer”?

1 1

4 4

7

cyberspace/1Neuromancer/1term/4novel/4coined/7

FactoidQ/A



DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation


Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


PassageRetrieval

RetrievedocumentsusingIR◦ querytermsaskeywords


Step1

Step2Segmentthedocumentsintoshorterunits◦ somethinglikeparagraphs

Step3Passageranking◦ Useanswertypetohelprerank passages

FeaturesforPassageRanking

NumberofNamedEntitiesoftherighttypeinpassage

Numberofquerywordsinpassage

NumberofquestionN-gramsalsoinpassage

Proximityofquerykeywordstoeachotherinpassage

Longestsequenceofquestionwords

Rankofthedocumentcontainingpassage


Eitherinrule-basedclassifiersorwithsupervisedmachinelearning

FactoidQ/A



DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation


Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs


AnswerExtractionRunananswer-typenamed-entitytaggeronthepassages◦ Eachanswertyperequiresanamed-entitytaggerthatdetectsit◦ IfanswertypeisCITY,taggerhastotagCITY◦ CanbefullNER,simpleregularexpressions,orhybrid

Returnthestringwiththerighttype:◦ WhoistheprimeministerofIndia(PERSON)◦ ManmohanSingh,PrimeMinisterofIndia,hadtoldleftleadersthatthedealwouldnotberenegotiated.

◦ HowtallisMt.Everest?(LENGTH)◦ TheofficialheightofMountEverestis29035feet


RankingCandidateAnswersButwhatiftherearemultiplecandidateanswers!

Q:WhowasQueenVictoria’ssecondson?

AnswerType:Person


• Passage:TheMariebiscuitisnamedafterMarieAlexandrovna,thedaughterofCzarAlexanderIIofRussiaandwifeofAlfred,thesecondsonofQueenVictoriaandPrinceAlbert

FeaturesforMLAnswertypematch:Candidatecontainsaphrasewiththecorrectanswertype.

Patternmatch:Regularexpressionpatternmatchesthecandidate.

Questionkeywords:#ofquestionkeywordsinthecandidate.

Keyworddistance:Distanceinwordsbetweenthecandidateandquerykeywords

Noveltyfactor:Awordinthecandidateisnotinthequery.

Appositionfeatures:Thecandidateisanappositivetoquestionterms

Punctuationlocation:Thecandidateisimmediatelyfollowedbyacomma,period,quotationmarks,semicolon,orexclamationmark.

Sequencesofquestionterms:Thelengthofthelongestsequenceofquestiontermsthatoccursinthecandidateanswer.


ScoringCandidatesinWatsonEachcandidateanswergetsscoresfrom>50components◦ (fromunstructuredtext,semi-structuredtext,triplestores)◦ logicalform(parse)matchbetweenquestionandcandidate◦ passagesourcereliability◦ geospatiallocation◦ Californiais”southwestofMontana”

◦ temporalrelationships◦ taxonomicclassification


Outline

QuestionAnswering

TextualEntailment


IR-BasedQASystem

OtherExtensions

AskMSR


1 2

3

45

QuestionProcessing Search

AnswerExtractionAnswerScoring

Dumais, Banko, Brill, Lin, Ng,SIGIR(2002)

Step1:RewriteQueries


Intuition: Questions are often syntactically quite closeto sentences with the answer• Where istheLouvreMuseumlocated?

• TheLouvreMuseumislocated in Paris• Who createdthecharacterofScrooge?

• Charles DickenscreatedthecharacterofScrooge

FeedbackLoops:FALCON


AllenAIScienceChallenge


Whichobjectisthebestconductorofelectricity?(A) awaxcrayon(B)aplasticspoon(C)arubbereraser(D)anironnail



Whichobjectisthebestconductorofelectricity?(A) awaxcrayon(B)aplasticspoon(C)arubbereraser(D)anironnail



Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop(D)grass



Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop (D)grass

§ Informationretrievalmethodsfail§ Wordco-occurrencemethodsstruggle

Graders are commonly used in the construction and maintenance of dirt roads and gravel roads

Also strong correlations between:grass « racegravel « surface



Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto

(A)light(B)air(C)water(D)soil



Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto

(A)light (B)air(C)water(D)soil

Knowledge needed(forexample):§ nearsunnywindow® receivelight§ inadarkroom® nolight§ testX’sresponsetoY

® compareX+YwithX+not(Y)

Outline

QuestionAnswering

TextualEntailment


IR-BasedQASystem

OtherExtensions

NaturalLanguage&Meaning


Meaning

Language

AmbiguityVariability

interpretatio

n expression

InferencevsEntailment


MeaningRepresentation

Natural Language

Inference

TextualEntailment

TextualEntailment


• Adirectionalrelationbetweentwotextfragments:Text(t) andHypothesis(h):

t entails h (t Þ h) if humans reading t will infer that h is most likely true

• Assuming“commonbackgroundknowledge”–whichisindeedexpectedfromapplications

Example

CS295:STATISTICALNLP(WINTER2017) 56https://cogcomp.cs.illinois.edu/page/resource_view/9

MoreSentencePairs


1. Some students came to school by car.Some students came to school.

2. No students came to school by car. Some students came to school.

3. John drove legally. John drove.

4. John drove predictably.John drove.

5. Legally, John could drive.John drove.

EntailmentwithKnowledge


Fortextualentailmenttoholdwerequire:◦ text ANDknowledgeÞ h,but◦ knowledgeshouldnotentailh alone

Systemsare notsupposedtovalidateh’struthregardlessoft(e.g.bysearchinghontheweb)

t entails h (t Þ h) if humans reading t will infer that h is most likely true

Example


TEXT: …WhilenooneaccusesMadonnaofdoinganythingillegalinadoptingthe4-year-oldgirl,reportedlynamedMercy,therearequestionsnonethelessabouthowMadonnaisabletonavigateMalawi's18-to-24monthvettingperiodinjustamatterofdaysorweeks…

HYPOTHESIS:

Madonnais50yearsold.

ThirdLabel:Contradictions


t contradicts h (t ^ h) if humans reading t will find the relations/events described by h to be highly unlikely given t.

Example


MoreExamples


Applications


QuestionAnswering

InformationExtraction

MachineTranslation

InformationRetrieval

Question Expected answer formWho bought Overture? >> X bought Overture

Overture’s acquisitionby Yahoo

Yahoo bought Overture

text hypothesized answer

entails

Heabhorredthemen’sunctuousways.

Hedislikedthemen’sflatteringways.

Similaritybasedonwhetherdocumententailsthequery.

question answering (and textual...

Documents