interacve machines: from quesons to experience
TRANSCRIPT
![Page 1: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/1.jpg)
Interac(veMachines:FromQues(onstoExperience
GiuseppeRiccardiUniversityofTrento,Italy
![Page 2: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/2.jpg)
TheVision
• WhyNaturalLanguageQues(on/Answer?• Interac(vemachines
• Interac(veeco‐systems
![Page 3: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/3.jpg)
Forthesakeofaskingques(ons• Knowledgewilldoit?
– Whywaterincreasesitsvolumewhenitfreezes?
![Page 4: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/4.jpg)
Forthesakeofaskingques(ons• Knowledgewilldoit?
Why water increases its volume when it freezes?
– GOOGLE.comCHACHA.com
Q:WhenwaterfreezesitsvolumeincreasesbyhowmuchpercentAA:Whenwaterfreezesat0°Citsvolumeincreasesbyabout9%underSTP(StandardTemperatureandPressure).
Isthatcorrect?
![Page 5: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/5.jpg)
Knowledge‐BasedQ&A
• Precisionoftheanswers• Accountabilityoftheresources• Limita(onofDocumentspace(vsWEB)
• Bri^lenessofthenaturallanguageinterface
![Page 6: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/6.jpg)
“Computersareuseless.Theycanonlygiveanswers.”—PabloPicasso,(1881‐1973).
![Page 7: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/7.jpg)
SearchEngineQueries
Searchvastamountofdatausing(almost)naturallanguage
Thesearchenginesimple“drill”1. ShortQuery(“universita’Trento“,“rifugio
PassoSanPellegrino“)
2. HumanParseofRetrievedDocuments3. HumanDecisionwhetherto
A. FollowuponlinksORB. GOTO1
![Page 8: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/8.jpg)
Knowledge‐BasedQ&A
![Page 9: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/9.jpg)
![Page 10: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/10.jpg)
DEEPQAProject• IBMResearchFundedproject
– BuildQAengine(WATSON)capableofchallengingHumansintheJeopardy!Game.
– Domainisvirtuallyopen.NODBratherunstructureddata(lightorfullsupervision).
• OpenCollabora(veResearchAgreementbtwUTrentoandIBMYorkTown– MIT,UTexas,CMU,USC
• Utrento– Interac(veMachinestoresolveQues(onsintoAc(ons/Decisions/Tasks
– MachineLearningmodelsforParsingSentencesforQAClassifica(on,RerankingHypotheses(targetAnswers)
![Page 11: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/11.jpg)
NaturalLanguageParsingandapplica(onstoDEEPQA
wAlessandroMoschi=
![Page 12: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/12.jpg)
Outline• Mo(va(ons
• TwoimportantproblemsinJeopardy– Ques(onClassifica(on– AnswerSelec(on
• Results• Conclusions
![Page 13: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/13.jpg)
LetusConsideroneJeopardyCue
• Whenhitbyelectrons,aphosphorgivesoffelectromagneGcenergyinthisform
• Solu(ons:photons/light• Electrons,phosphorandelectromagneGcenergyareinarela(onshipwhichgivesthesolu(on
• Howrepresen(ngandusingsuchrela(oninamachine?
![Page 14: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/14.jpg)
Issuesandsolu(ons• Howexploi(ngshallowseman(cinforma(on?• Howdealingwithnoiseanderrorsofseman(cparsing?
• Designinghand‐crakedrulestodealwithanykindoferrorsornoiseisnotrealis(c
• Weneedsta(s(calmethodsasthey:– “canac(vatetherules”thatmaximizetheprobabilityofsuccess
– canautoma(callylearnsuchprobabili(esfromdata
![Page 15: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/15.jpg)
ImportantTasksinJeopardy
Ques?on
Ques(on/TopicAnalysis
Hypothesis&EvidenceScoring
SokFiltering Synthesis
FinalMerging&Ranking
QueryDecomposi(on
HypothesisGenera(on
Answer&Confidence
AnswerReranking
Ques(onClassifica(on
![Page 16: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/16.jpg)
Classifica(oninDefini(onvsnotDefini(oninJeopardy
• Usually,todothisistoloseagamewithoutplayingit
(solu(on:forfeit)
• Whenhitbyelectrons,aphosphorgivesoffelectromagneGcenergyinthisform
![Page 17: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/17.jpg)
OurApproach
• SupervisedApproach– Posi(veandnega(veexamplesofdefini(onques(ons
– Syntac(cinforma(onisintui(velyimportant• Applyoff‐the‐shelfparsers
– Asthisisanewtask,toextractfeatures,weexploittreekernels,e.g.thePar(alTreeKernel(Moschim,ECML2006)
![Page 18: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/18.jpg)
ParseTree
ROOT
SBARQ
WHADVP
WRB
When
S
VP
VBN
hit
PP
IN
by
NP
NNS
electrons
,
,
NP
DT
a
NN
phosphor
VP
VBZ
gives
PRP
RP
off
NP
NP
JJ
electromagnetic
NN
energy
PP
IN
in
NP
DT
this
NN
form
ROOT
SBARQ
WHADVP
WRB
When
S
VP
VBN
hit
PP
IN
by
NP
NNS
electrons
,
,
NP
DT
a
NN
phosphor
VP
VBZ
gives
PRP
RP
off
NP
NP
JJ
electromagnetic
NN
energy
PP
IN
in
NP
DT
this
NN
form
PAS
A0
electrons
predicate
hit
AM
When
PAS
A0
a phosphor
predicate
give off
A1
energy
AM
in this form
PAS
A0
electrons
predicate
hit
A1
it
AM
if
PAS
A0
a phosphor
predicate
give off
A1
photons
1
![Page 19: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/19.jpg)
Similaritybasedonthenumberofcommonsubstructures
NP
D N
VP
V
hit
a phosphor
![Page 20: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/20.jpg)
Apor(onofthesubstructureset
![Page 21: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/21.jpg)
ExperimentsonJeopardyQues(onClassifica(on
Model Precision Recall F1 RBC 28.27 70.59 40.38 BOW 46.55 50.65 48.51 CSK 39.07 77.12 51.86 PTK 47.84 57.84 52.37 PTK+WSK+CSK+RBC
67.66 66.99 67.32
RuleBasedClassifier(RBC)
OnlyWordOverlap(BOW) CategorySubsequences(CSK)
Par(alTreeKernel(PTK)
66.7%ofrela(veimprovementontherule‐basedclassifier
![Page 22: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/22.jpg)
OurApproachtoAnswerRe‐ranking
• Learnaclassifierof<ques(on,answer>pairs– Posi(ve:theansweriscorrect– Nega(ve:otherwise
• Kernelapproach– Severalkernelsappliedtobothques(onsandanswers
![Page 23: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/23.jpg)
AnexampleofJeopardyQues(on
![Page 24: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/24.jpg)
![Page 25: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/25.jpg)
Baseline Model
Ques(on
Answer
Methodology:1‐ApplyingPTKwithoutanyextraannota(onandevaluatethemodelasbaseline.
![Page 26: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/26.jpg)
!"#$%&'(
)'$*#+(
![Page 27: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/27.jpg)
Best Model Methodology:1‐Applyinglemma(za(onandstemminginleaveslevel.2‐Addananchortopre‐terminalandhigherlevelsifthesub‐treesaresharedinQandA.3‐Ignorestopwordsinmatchingprocedure.Ques(on
Answer
![Page 28: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/28.jpg)
!"#$%&'(
![Page 29: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/29.jpg)
Precision/Recall Evaluation
Precision Recall F1
BaselinewithPTK(Par(alTreeKernel)
17.1% 59.37% 26.56
BestModel(STK)Syntac(cTreeKernel
17.71% 5.59% 8.50
BestModelwithPTK 24.10% 61.12% 34.57
![Page 30: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/30.jpg)
Ranking Precision at 1 and 5 best
Evalua?onon1‐best
Evalua?onon5‐best
Accuracy Accuracy
Baseline 0.22 0.22
PTK(Baseline) 0.23 0.19
PTK(BestModel) 0.33 0.23
![Page 31: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/31.jpg)
Conclusions• WeuseofpowerfulMLalgorithms
– e.g.SupportVectorMachines– robusttonoise
• Abstractrepresenta(onsofexamples– Similarityfunc(ons(KernelMethods)
• ModelingQues(onseman(cswithadvancedsyntac(candshallowseman(cstructures
• SokwarealreadyintegratedintheJeopardySystem
![Page 32: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/32.jpg)
Ques(onstoAc(ons
![Page 33: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/33.jpg)
SocialNetworks
GiuseppeRiccardi 33
![Page 34: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/34.jpg)
SocialInterac(onsCustomer CompanyCi?zen PublicAdmin.Pa?ent DoctorCi?zen Poli?calPartyUser UserWorker SupervisorPeer PeerRela?ve Rela?ve…………… ……………
![Page 35: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/35.jpg)
Cancomputersbe
partofthesocialnetwork?
![Page 36: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/36.jpg)
NetworkofAgents
HAgent
MAgent2
Yahoo!
Kayak
HAgent2
Blog
HAgent1
MAgent1
GiuseppeRiccardi 36
HAgent
HAgent2
Chacha
Kayak
Blog1
Blog2
HAgent1
MAgent1
Informa(onBandwidth(Entropy)Cost($,(me2taskcomple(on)Reliability(Trust,Performance)
![Page 37: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/37.jpg)
Human–ComputerInterac(on
H
ARTICULATION
OBSERVATION
TYPING
GESTURES
SPEECH
VISION
VISION
HEARING
Spring2010 37……….
![Page 38: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/38.jpg)
Human–ComputerInterac(on
H C
ARTICULATION OBSERVATION
PRESENTATIONOBSERVATION
TYPING
GESTURES
SPEECH
VISION
VISION
GUI
TTSHEARING
Spring2010 38……….
……….
INTERFACE
![Page 39: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/39.jpg)
Spring2010 39
Automatic Speech Understanding
FeatureExtrac(on A
Pa^ernMatching Ŵ
Given the acoustic observation sequence A=a1,a2,…,am,
what is the most likely “word” sequence W=w1,w2,…,wn?
LEHTAHSPREYLetuspray
Le^ucespray
![Page 40: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/40.jpg)
Natural Language Query to DB
Find the best flight from New York to Paris tomorrow business class!
TASK:Transac?onal
USERCONSTRAINTS
Object
![Page 41: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/41.jpg)
Spring 2009 41
Machine Understanding • User:
“Find the best flight from New York to Paris tomorrow business class”
• Speech Recognition: “ Find the bass flight from Newark to Paris
tomorrow business class”
• Modeling Uncertainty: – @action=Request-Reservation (0.9) – @origin=Newark (0.5) – @time-departure=Tuesday (0.7) – @destination=Paris (0.8)
Humanerrstoo!
![Page 42: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/42.jpg)
Cooperative Task SW-HW Help Desk
UHiGoodMorningOHi,HowMayIHelpYou?UIamRobertaSicconicalling
fromCulturalAffairsatCityHall.
UIhadmadearequestforapasswordchangeyesterday
OOkdoyouhavetherequesttrackid?
UUhmNoIcannotfindOOkdoyouhavethedateof
therequest?UWellthatwasyesterdayO...okIthinkIcanfindit..IgotitOIt’sforapasswordreset.
URight.TheproblemisthatIchangedthepasswordwhenIfirstloggedin..
OYouweresupposedtochangefirst(meyouloggedin.Nowlet’strytogethertologin
OcanyoutellmeyouRVSofyourcomputer
UWellletmesee.ThisisanewPCtome.WherecanIfindit?
OUsuallythetagisrightnexttothebaseofthechassynexttothepowerswitch.Itreads“inventariose^oreinforma(co”.
UInventario..Se^ore...
PersonalIden?fica?on
ProblemStatementTicketRecordRetrieval
ProblemResolu?on(USER)
ProblemResolu?on(PARTI)
10/13/2008 GiuseppeRiccardi
![Page 43: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/43.jpg)
DialogModels(DM)
![Page 44: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/44.jpg)
MarkovDecisionProcesses
![Page 45: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/45.jpg)
Conversa(onalAgents
![Page 46: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/46.jpg)
Conversa(onalAgents
Observa(ons+Interpreta(on+Uncertainty+Decisions+Interac(on=Experience
![Page 47: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/47.jpg)
Evalua(on• Howgoodaremachines?
– Accomplishingtasks
– Acceptancebytheusers,bythenetwork– LifeSpan(Evolveover(me)
• “Research Systems are difficult to evaluate while Commercial systems are ea$y to evaluate “(Paek,2007)
![Page 48: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/48.jpg)
DEMOS
![Page 49: Interacve Machines: From Quesons to Experience](https://reader031.vdocument.in/reader031/viewer/2022022217/6213e0f08a0ae56dde7c10ee/html5/thumbnails/49.jpg)
Conclusion
• ThereareQ&Amachinetaskswhichwillgive(par(al)benefits
• Machinesmayfulfill(social)roles
• Theul?mategoalistounderstandthecollabora?veinterac?onofmachine/humanagents– Language– Interac?onModel– SocialEqua?on