CS388:NaturalLanguageProcessingLecture16:Informa;onExtrac;on

GregDurrett

Recall:ABen;on

themoviewasgreat

h1 h2 h3 h4

<s>

h̄1

‣ Foreachdecoderstate,computeweightedsumofinputstates

eij = f(h̄i, hj)

ci =X

j

↵ijhj

c1

‣ Unnormalized scalarweight

‣Weightedsumofinputhidden states(vector)

le

↵ij =exp(eij)Pj0 exp(eij0)

P (yi|x, y1, . . . , yi�1) = softmax(W [ci; ¯hi])

themovie wa

sgreat

‣ Veryhelpfulforseq2seqmodels

Recall:Alterna;ves‣ WhendowecomputeaBen;on?CancomputebeforeoraWerRNNcell

Bahdanauetal.(2015)

<s>

h̄1

c1

<s>

c1

Luongetal.(2015)

‣ AWerRNNcell

‣ BeforeRNNcell;thisoneisaliBlemoreconvolutedandlessstandard

lele

ThisLecture

‣Howdowerepresentinforma;onforinforma;onextrac;on?

‣ OpenInforma;onExtrac;on

‣ Rela;onextrac;on

‣ Slotfilling

‣ Seman;crolelabeling/abstractmeaningrepresenta;on

Represen;ngInforma;on

Seman;cRepresenta;ons

Examplecredit:AsadSayeed

personBrutusCaesarObamaBush…

stabBrutus Caesar

…

‣ “World”isasetofen;;esandpredicates

presidentObamaBush…

BrutusstabsCaesar stab(Brutus,Caesar)=>true

‣ Statementsarelogicalexpressionsthatevaluatetotrueorfalse

Caesarwasstabbed ∃xstab(x,Caesar)=>true

Neo-DavidsonianEvents

Examplecredit:AsadSayeed

BrutusstabbedCaesarwithaknifeatthetheaterontheIdesofMarch

∃estabs(e,Brutus,Caesar)

‣ Letsusdescribeeventsashavingproper;es‣Unifiedrepresenta;onofeventsanden;;es:

∃xdriver(x)∧clever(x)∧loca;on(x,America)

somecleverdriverinAmerica

∧with(e,knife) ∧loca;on(e,theater)∧;me(e,IdesofMarch)

RealText

BarackObamasignedtheAffordableCareactonTuesday.HegaveaspeechlaterthataCernoononhowtheactwouldhelptheAmericanpeople.SeveralprominentRepublicanswerequicktodenouncethenewlaw. whichTuesday?

‣ Needtoimputemissinginforma;on,resolvecoreference,etc.

‣ S;llunclearhowtorepresentsomethingspreciselyorhowthatinforma;oncouldbeleveraged(severalprominentRepublicans)

who?

???

whichaWernoon?

∃esign(e,BarackObama)∧pa;ent(e,ACA)∧;me(e,Tuesday)

OtherChallenges

BobandAlicewerefriendsunGlhemovedawaytoaHendcollege

‣Howtorepresenttemporalinforma;on?

‣ Represen;ngtrulyopen-domaininforma;onisverycomplicated!Wedon’thaveaformalrepresenta;onthatcancaptureeverything

∃e1∃e2friends(e1,Bob,Alice)∧moved(e2,Bob)∧end_of(e1,e2)

BobandAlicewerefriendsunGlaroundthe+mehemovedawaytoaHendcollege

(Atleast)ThreeSolu;ons

‣ En;ty-rela;on-en;tytriples:focusonen;;esandtheirrela;ons(notethatprominenteventscans;llbeen;;es)

(LadyGaga,singerOf,BadRomance)

‣ CraWedannota;onstocapturesomesubsetofphenomena:predicate-argumentstructures(seman;crolelabeling),;me(temporalrela;ons),…

‣ Slotfilling:specificontology,populateinforma;oninapredefinedway

(Earthquake:magnitude=8.0,epicenter=centralItaly,…)

OpenIE‣ En;ty-rela;on-en;tytriplesaren’tnecessarilygroundedinanontology

BarackObamasignedtheAffordableCareactonTuesday.HegaveaspeechlaterthataCernoononhowtheactwouldhelptheAmericanpeople.SeveralprominentRepublicanswerequicktodenouncethenewlaw.

(BarackObama,signed,theAffordableCareact)(SeveralprominentRepublicans,denounce,thenewlaw)

‣ Extractstringsandletadownstreamsystemfigureitout

IE:TheBigPicture‣ Howdowerepresentinforma;on?Whatdoweextract?

‣ Slotfillers‣ En;ty-rela;on-en;tytriples(fixedontologyoropen)

‣ Seman;croles‣ Abstractmeaningrepresenta;on

Seman;cRoleLabeling/ AbstractMeaningRepresenta;on

Seman;cRoleLabeling‣ Iden;fypredicate,disambiguateit,iden;fythatpredicate’sarguments

‣ VerbrolesfromPropbank(Palmeretal.,2005)

FigurefromHeetal.(2017)

quicken:

Seman;cRoleLabeling

FigurefromHeetal.(2017)

‣ Iden;fypredicates(love)usingaclassifier(notshown)‣ Iden;fyARG0,ARG1,etc.asataggingtaskwithaBiLSTMcondi;onedonlove

‣Othersystemsincorporatesyntax,jointpredicate-argumentfinding

SRLforQA

ShenandLapata(2007)

‣Ques;onandseveralanswercandidates

Q:Whodiscoveredprions?

AC1:In1997,StanleyB.Prusiner,ascienGstintheUnitedStates,discoveredprions…

AC2:Prionswereresearchedby…

Scorebymatchingexpectedanswerphrase(EAP)againstanswercandidate(AC)

AbstractMeaningRepresenta;on‣Graph-structuredannota;on

Theboywantstogo

Banarescuetal.(2014)

‣ SupersetofSRL:fullsentenceanalyses,containscoreferenceandmul;-wordexpressionsaswell

‣ F1scoresinthe60s:hard!

‣ Socomprehensivethatit’shardtopredict,buts;lldoesn’thandletenseorsomeotherthings…

Summariza;onwithAMR

Liuetal.(2015)

‣ MergeAMRsacrossmul;plesentences

‣ Summariza;on=subgraphextrac;on

‣ Norealsystemsactuallyworkthisway(morewhenwetalkaboutsummariza;on)

SlotFilling

SlotFilling‣Mostconserva;ve,narrowformofIE

IndianExpress—Amassiveearthquakeofmagnitude7.3struckIraqonSunday,103kms(64miles)southeastofthecityofAs-Sulaymaniyah,theUSGeologicalSurveysaid,reportsReuters.USGeologicalSurveyiniGallysaidthequakewasofamagnitude7.2,beforerevisingitto7.3.

magnitude time

epicenter

Speaker:AlanClark“GenderRolesintheHolyRomanEmpire”AllagherCenterMainAuditoriumThistalkwilldiscuss…

speakertitle

location

FreitagandMcCallum(2000)

‣ Oldwork:HMMs,laterCRFstrainedperrole

SlotFilling:MUC

HaghighiandKlein(2010)

‣ Keyaspect:needtocombineinforma;onacrossmul;plemen;onsofanen;tyusingcoreference

SlotFilling:Forums

‣ Extractproductoccurrencesincybercrimeforums,butnoteverythingthatlookslikeaproductisaproduct

Portnoffetal.(2017),DurreBetal.(2017)Notaproductinthiscontext

Rela;onExtrac;on

Rela;onExtrac;on‣ Extracten;ty-rela;on-en;tytriplesfromafixedinventory

ACE(2003-2005)

DuringthewarinIraq,AmericanjournalistsweresomeGmescaughtinthelineoffire

Located_In

‣ Systemscanbefeature-basedorneural,lookatsurfacewords,syntac;cfeatures(dependencypaths),seman;croles

Na;onality

‣ Problem:limiteddataforscalingtobigontologies

‣ UseNER-likesystemtoiden;fyen;tyspans,classifyrela;onsbetweenen;typairswithaclassifier

HearstPaBerns

Hearst(1992)

Xsuchas[list] ciGessuchasBerlin,Paris,andLondon.

‣ Syntac;cpaBernsespeciallyforfindinghypernym-hyponympairs(“isa”rela;ons)

Berlinisacity

otherciGesincludingBerlin

YisaX

otherXincludingY

‣ Totallyunsupervisedwayofharves;ngworldknowledgefortaskslikeparsingandcoreference(BansalandKlein,2011-2012)

DistantSupervision

Mintzetal.(2009)

[StevenSpielberg]’sfilm[SavingPrivateRyan]islooselybasedonthebrothers’story

‣ Iftwoen;;esinarela;onappearinthesamesentence,assumethesentenceexpressestherela;on

‣ Lotsofrela;onsinourknowledgebasealready(e.g.,23,000film-directorrela;ons);usethesetobootstrapmoretrainingdata

Allisonco-producedtheAcademyAward-winning[SavingPrivateRyan],directedby[StevenSpielberg]

Director

Director

DistantSupervision

Mintzetal.(2009)

‣ Learndecentlyaccurateclassifiersfor~100Freebaserela;ons‣ Couldbeusedtocrawlthewebandexpandourknowledgebase

OpenIE

OpenInforma;onExtrac;on

‣ Typicallynofixedrela;oninventory

‣ “Open”ness—wanttobeabletoextractallkindsofinforma;onfromopen-domaintext

‣ Acquirecommonsenseknowledgejustfrom“reading”aboutit,butneedtoprocesslotsoftext(“machinereading”)

TextRunner‣ Extractposi;veexamplesof(e,r,e)triplesviaparsingandheuris;cs

BarackObama,44thpresidentoftheUnitedStates,wasbornonAugust4,1961inHonolulu

=>Barack_Obama,wasbornin,Honolulu

‣ TrainaNaiveBayesclassifiertofiltertriplesfromrawtext:usesfeaturesonPOStags,lexicalfeatures,stopwords,etc.

‣ 80xfasterthanrunningaparser(whichwasslowin2007…)

Bankoetal.(2007)

‣ Usemul;pleinstancesofextrac;onstoassignprobabilitytoarela;on

Exploi;ngRedundancy

Bankoetal.(2007)

‣ Concrete:definitelytrueAbstract:possiblytruebutunderspecified

‣Hardtoevaluate:canassessprecision ofextractedfacts,buthowdoweknowrecall?

‣ 9Mwebpages/133Msentences

‣ 2.2tuplesextractedpersentence,filterbasedonprobabili;es

ReVerb

Faderetal.(2011)

‣Moreconstraints:openrela;onshavetobeginwithverb,endwithpreposi;on,becon;guous(e.g.,wasbornon)

‣ Extractmoremeaningfulrela;ons,par;cularlywithlightverbs

ReVerb

Faderetal.(2011)

‣ Foreachverb,iden;fythelongestsequenceofwordsfollowingtheverbthatsa;sfyaPOSregex(V.*P)andwhichsa;sfyheuris;clexicalconstraintsonspecificity

‣ Findthenearestargumentsoneithersideoftherela;on

‣ Annotatorslabeledrela;onsin500documentstoassessrecall

QAfromOpenIE

Choietal.(2015)

Takeaways

‣ Rela;onextrac;on:cancollectdatawithdistantsupervision,usethistoexpandknowledgebases

‣ Slotfilling:;edtoaspecificontology,butgivesfine-grainedinforma;on

‣ OpenIE:extractslotsofthings,buthardtoknowhowgoodorusefultheyare

‣ Cancombinewithstandardques;onanswering

‣ Addnewfactstoknowledgebases

‣ SRL/AMR:handleabunchofphenomena,butmoreorlesslikesyntax++intermsofwhattheyrepresent

‣ Many,manyapplica;onsandtechniques