CS388:NaturalLanguageProcessing

GregDurre8

Lecture20:Informa>onExtrac>on,SRL,etc.

Administrivia

‣ Finalprojectpresenta>onslotsannounced

‣ Project2duetoday

ThisLecture

‣ Howdowerepresentinforma>onforinforma>onextrac>on?

‣ OpenInforma>onExtrac>on

‣ Rela>onextrac>on

‣ Slotfilling

‣ Seman>crolelabeling/abstractmeaningrepresenta>on

Represen>ngInforma>on

Seman>cRepresenta>ons

Examplecredit:AsadSayeed

personBrutusCaesarObamaBush…

stabBrutus Caesar

…

‣ “World”isasetofen>>esandpredicates

president

Obama

Bush…

BrutusstabsCaesar stab(Brutus,Caesar)=>true

‣ Statementsarelogicalexpressionsthatevaluatetotrueorfalse

Caesarwasstabbed ∃xstab(x,Caesar)=>true

Neo-DavidsonianEvents

Examplecredit:AsadSayeed

BrutusstabbedCaesarwithaknifeatthetheaterontheIdesofMarch

∃estabs(e,Brutus,Caesar)

‣ Letsusdescribeeventsashavingproper>es‣ Unifiedrepresenta>onofeventsanden>>es:

∃xdriver(x)∧clever(x)∧loca>on(x,America)

somecleverdriverinAmerica

∧with(e,knife) ∧loca>on(e,theater)

∧>me(e,IdesofMarch)

RealText

BarackObamasignedtheAffordableCareactonTuesday.HegaveaspeechlaterthataCernoononhowtheactwouldhelptheAmericanpeople.SeveralprominentRepublicanswerequicktodenouncethenewlaw. whichTuesday?

‣ Needtoimputemissinginforma>on,resolvecoreference,etc.

‣ S>llunclearhowtorepresentsomethingspreciselyorhowthatinforma>oncouldbeleveraged(severalprominentRepublicans)

who?

???

whichaaernoon?

∃esign(e,BarackObama)∧pa>ent(e,ACA)∧>me(e,Tuesday)

OtherChallenges

BobandAlicewerefriendsunGlhemovedawaytoaHendcollege

‣ Howtorepresenttemporalinforma>on?

‣ Represen>ngtrulyopen-domaininforma>onisverycomplicated!Wedon’thaveaformalrepresenta>onthatcancaptureeverything

∃e1∃e2friends(e1,Bob,Alice)∧moved(e2,Bob)∧end_of(e1,e2)

BobandAlicewerefriendsunGlaroundthe+mehemovedawaytoaHendcollege

(Atleast)ThreeSolu>ons

‣ En>ty-rela>on-en>tytriples:focusonen>>esandtheirrela>ons(notethaten>>esispre8ybroad:canincludeeventslikeWorldWarII,etc.)

(LadyGaga,singerOf,BadRomance)

‣ Craaedannota>onstocapturesomesubsetofphenomena:predicate-argumentstructures(seman>crolelabeling),>me(temporalrela>ons),…

‣ Slotfilling:specificontology,populateinforma>oninapredefinedway

(Earthquake:magnitude=8.0,epicenter=centralItaly,…)

OpenIE‣ En>ty-rela>on-en>tytriplesaren’tnecessarilygroundedinanontology

BarackObamasignedtheAffordableCareactonTuesday.HegaveaspeechlaterthataCernoononhowtheactwouldhelptheAmericanpeople.SeveralprominentRepublicanswerequicktodenouncethenewlaw.

(BarackObama,signed,theAffordableCareact)

(SeveralprominentRepublicans,denounce,thenewlaw)

‣ Extractstringsandletadownstreamsystemfigureitout

IE:TheBigPicture

‣ Howdowerepresentinforma>on?Whatdoweextract?

‣ Slotfillers‣ En>ty-rela>on-en>tytriples(fixedontologyoropen)

‣ Seman>croles

‣ Abstractmeaningrepresenta>on

Seman>cRoleLabeling/ AbstractMeaningRepresenta>on

Seman>cRoleLabeling

LadyGagaperformedaconcertforstudents

‣ Performingevent

‣ Subject:LadyGaga‣ Object:aconcert‣ Audience:students

AconcertwasperformedbyLadyGagaforstudents

‣ Sameeventdescribedbuttherepresenta>onlooksdifferent

VARG1 ARG0 ARG2

VARG0 ARG1 ARG2

‣ Verb(predicate)associatedwithseveralarguments(roles):“Agent”,“Theme”,and“Beneficiary”

VerbNet

verbs.colorado.edu

‣ Definestheseman>csofverbs,argumentsforeveryverbinEnglish

Seman>cRoles

‣ Relatedtothetarolesinlinguis>cs

‣ Agent(~subject),pa>ent/theme(~object),goal(~indirectobject)

‣ “Postprocessing”layerontopofdependencyparsingthatexposesusefulinforma>on,canonicalizesacrossgramma>calconstruc>ons

ARG0 ARG1 ARG2+(seman>csvary)

Seman>cRoleLabeling

‣ Iden>fypredicate,disambiguateit,iden>fythatpredicate’sarguments

‣ VerbrolesfromPropbank(Palmeretal.,2005)

FigurefromHeetal.(2017)

quicken:

Seman>cRoleLabeling

FigurefromHeetal.(2017)

‣ Iden>fypredicates(love)usingaclassifier(notshown)

‣ Iden>fyARG0,ARG1,etc.asataggingtaskwithaBiLSTMcondi>onedonlove

‣ Othersystemsincorporatesyntax,jointpredicate-argumentfinding

SRLforQA

ShenandLapata(2007)

‣ Ques>onandseveralanswercandidates

Q:Whodiscoveredprions?

AC1:In1997,StanleyB.Prusiner,ascienGstintheUnitedStates,discoveredprions…

AC2:Prionswereresearchedby…

Scorebymatchingexpectedanswerphrase(EAP)againstanswercandidate(AC)

AbstractMeaningRepresenta>on

‣ Graph-structuredannota>on

Theboywantstogo

Banarescuetal.(2014)

‣ SupersetofSRL:fullsentenceanalyses,containscoreferenceandmul>-wordexpressionsaswell

‣ F1scoresinthe60s:hard!

‣ Socomprehensivethatit’shardtopredict,buts>lldoesn’thandletenseorsomeotherthings…

AbstractMeaningRepresenta>on

Flaniganetal.(2016),Lyuetal.(2018)

‣ Firstpredictmappingfromconceptstographnodes(many-to-many)

‣ Thenuseanedgescoringmodulesimilartodependencyparserstopredictedges

‣ Predic>ngacoherentgraphishard,lotsofconstraintsonitandnodynamicprogram

Summariza>onwithAMR

Liuetal.(2015)

‣MergeAMRsacrossmul>plesentences

‣ Summariza>on=subgraphextrac>on

‣ Norealsystemsactuallyworkthisway(morewhenwetalkaboutsummariza>on)

SlotFilling

SlotFilling‣Mostconserva>ve,narrowformofIE

IndianExpress—Amassiveearthquakeofmagnitude7.3struckIraqonSunday,103kms(64miles)southeastofthecityofAs-Sulaymaniyah,theUSGeologicalSurveysaid,reportsReuters.USGeologicalSurveyiniGallysaidthequakewasofamagnitude7.2,beforerevisingitto7.3.

magnitude time

epicenter

Speaker:AlanClark

“GenderRolesintheHolyRomanEmpire”

AllagherCenterMainAuditorium

Thistalkwilldiscuss…

speakertitle

location

FreitagandMcCallum(2000)

‣ Oldwork:HMMs,laterCRFstrainedperrole

SlotFilling:MUC

HaghighiandKlein(2010)

‣ Keyaspect:needtocombineinforma>onacrossmul>plemen>onsofanen>tyusingcoreference

SlotFilling:Forums

‣ Extractproductoccurrencesincybercrimeforums,butnoteverythingthatlookslikeaproductisaproduct

Portnoffetal.(2017),Durre8etal.(2017)Notaproductinthiscontext

Rela>onExtrac>on

Rela>onExtrac>on‣ Extracten>ty-rela>on-en>tytriplesfromafixedinventory

ACE(2003-2005)

DuringthewarinIraq,AmericanjournalistsweresomeGmescaughtinthelineoffire

Located_In

‣ Systemscanbefeature-basedorneural,lookatsurfacewords,syntac>cfeatures(dependencypaths),seman>croles

Na>onality

‣ Problem:limiteddataforscalingtobigontologies

‣ UseNER-likesystemtoiden>fyen>tyspans,classifyrela>onsbetweenen>typairswithaclassifier

HearstPa8erns

Hearst(1992)

Xsuchas[list] ciGessuchasBerlin,Paris,andLondon.

‣ Syntac>cpa8ernsespeciallyforfindinghypernym-hyponympairs(“isa”rela>ons)

Berlinisacity

otherciGesincludingBerlin

YisaX

otherXincludingY

‣ Totallyunsupervisedwayofharves>ngworldknowledgefortaskslikeparsingandcoreference(BansalandKlein,2011-2012)

DistantSupervision

Mintzetal.(2009)

[StevenSpielberg]’sfilm[SavingPrivateRyan]islooselybasedonthebrothers’story

‣ Iftwoen>>esinarela>onappearinthesamesentence,assumethesentenceexpressestherela>on

‣ Lotsofrela>onsinourknowledgebasealready(e.g.,23,000film-directorrela>ons);usethesetobootstrapmoretrainingdata

Allisonco-producedtheAcademyAward-winning[SavingPrivateRyan],directedby[StevenSpielberg]

Director

Director

DistantSupervision

Mintzetal.(2009)

‣ Learndecentlyaccurateclassifiersfor~100Freebaserela>ons‣ Couldbeusedtocrawlthewebandexpandourknowledgebase

FewRel‣ Treatsrela>onclassifica>onasafew-shotclassifica>onproblem

‣ 100classesx700instances,goalistogeneralizetoeachclasswithjustafewinstances

‣ BERTcanhandlethisfairlywell(Soaresetal.,2019)

‣ “FewRel2.0”:newdatasetwith“noneoftheabove”type,whichmakesthingsmuchharder

Hanetal.(2018),Gaoetal.(2019)

En>tyTracking/ProceduralText

En>tyTracking

Slidecredit:AdityaGupta

RecipesDataset

Kiddonetal.(2016),Bosselutetal.(2018)

‣ Informa>onextrac>onfor“proceduraltext”:textdescribingsomekindofprocess

‣ Forarecipe:whatingredientsareinvolvedateach>mestep?

‣ Involvesglobalconstraintsandbeingabletomodelcomplexen>tyinterac>ons

En>tyTracking

Slidecredit:AdityaGupta

ProParaDataset

Dalvietal.(2018),GuptaandDurre8(2019)

‣ Processparagraphs:predictwhenobjectsarecreated,moved,ordestroyedinascien>ficprocess

‣ Structuredpredic>onproblem,>edtothepar>cularinforma>onconveyedintheseparagraphs

‣ UseaneuralCRFtomakeacoherentpredic>onforeachen>ty

OpenIE

OpenInforma>onExtrac>on

‣ Typicallynofixedrela>oninventory

‣ “Open”ness—wanttobeabletoextractallkindsofinforma>onfromopen-domaintext

‣ Acquirecommonsenseknowledgejustfrom“reading”aboutit,butneedtoprocesslotsoftext(“machinereading”)

TextRunner‣ Extractposi>veexamplesof(e,r,e)triplesviaparsingandheuris>cs

BarackObama,44thpresidentoftheUnitedStates,wasbornonAugust4,1961inHonolulu

=>Barack_Obama,wasbornin,Honolulu

‣ TrainaNaiveBayesclassifiertofiltertriplesfromrawtext:usesfeaturesonPOStags,lexicalfeatures,stopwords,etc.

‣ 80xfasterthanrunningaparser(whichwasslowin2007…)

Bankoetal.(2007)

‣ Usemul>pleinstancesofextrac>onstoassignprobabilitytoarela>on

Exploi>ngRedundancy

Bankoetal.(2007)

‣ Concrete:definitelytrueAbstract:possiblytruebutunderspecified

‣ Hardtoevaluate:canassessprecision ofextractedfacts,buthowdoweknowrecall?

‣ 9Mwebpages/133Msentences

‣ 2.2tuplesextractedpersentence,filterbasedonprobabili>es

ReVerb

Faderetal.(2011)

‣Moreconstraints:openrela>onshavetobeginwithverb,endwithpreposi>on,becon>guous(e.g.,wasbornon)

‣ Extractmoremeaningfulrela>ons,par>cularlywithlightverbs

ReVerb

Faderetal.(2011)

‣ Foreachverb,iden>fythelongestsequenceofwordsfollowingtheverbthatsa>sfyaPOSregex(V.*P)andwhichsa>sfyheuris>clexicalconstraintsonspecificity

‣ Findthenearestargumentsoneithersideoftherela>on

‣ Annotatorslabeledrela>onsin500documentstoassessrecall

QAfromOpenIE

Choietal.(2015)

Takeaways

‣ Rela>onextrac>on:cancollectdatawithdistantsupervision,usethistoexpandknowledgebases

‣ Slotfilling:>edtoaspecificontology,butgivesfine-grainedinforma>on

‣ OpenIE:extractslotsofthings,buthardtoknowhowgoodorusefultheyare

‣ Cancombinewithstandardques>onanswering

‣ Addnewfactstoknowledgebases

‣ SRL/AMR:handleabunchofphenomena,butmoreorlesslikesyntax++intermsofwhattheyrepresent

‣Many,manyapplica>onsandtechniques