CS388:NaturalLanguageProcessingCoreferenceResolu8on

GregDurrett

RoadMap

POStagging

Syntac8cparsing

NER

Coreferenceresolu8on

Summarize

Extractinforma8on

Answerques8ons

Iden8fysen8ment

Translate

TextAnalysis Applica/onsText Annota/ons

‣ Analysis:syntax,seman8cs,discourse,pragma8cs

‣ Coreference:discourse+pragma8cs

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

DiscourseAnalysis

PresidentBarackObamareceivedtheServeAmericaActa5erCongress’svote.HesignedthebilllastThursday.ThepresidentsaiditwouldgreatlyincreaseserviceopportuniBesfortheAmericanpeople.

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

slidecredit:AriaHaghighi

DiscourseAnalysis

Events

En((es

Text

Discourse(rhetorical,temporalstructure)

slidecredit:AriaHaghighi

DiscourseAnalysis

Cluster1:en.wikipedia.org/wiki/Barack_Obama

Cluster2:…/wiki/Edward_M.Kennedy_Serve_America_Act

Cluster3:…/wiki/United_States_Congress

En88esPresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

‣ En88esarereal-worldthingsthatcanberesolvedtoanentryinaknowledgebase(Wikipedia),canrepeatedlyreferencetheminatext

CoreferenceResolu8on

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

‣ Input:textwithmen8ons

‣ Output:aclusteringofthosemen8ons

CoreferenceResolu8on

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.

‣ Input:textwithmen8ons

‣ Alterna8vely:answer“whoismyantecedent?”foreachanaphor

PresidentBarackObama

HeAnaphor

Antecedent

coreferenttheServe

AmericaActCongress’s

Possibleantecedents

Outline‣ Linguis8cphenomenaincoreference

‣ Incorpora8ngworldknowledge

‣ Buildingcoreferencemodels

PhenomenainCoreference

Pragma8cs101PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. PresidentBarackObamasignedtheServeAmericaActlastThursday. PresidentBarackObamasaid…

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. HesignedthebilllastThursday. Thepresidentsaid…‣Whenwespeak/write,wehaveanideaofwhat’scleartothelistener,andcommunicatemoreefficientlyasaresult

Pragma8cs101

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. HesignedthebilllastThursday. Thepresidentsaid…

ProperNamePresidentBarackObama thepresident he

Nominal Pronoun

Specificity

Saliencerequired

‣ Proper,nominal,andpronominalmen8onsallresolvedifferently

ProperMen8ons‣ Introducenewen88esandgiveinforma8on,iden8tyen88esunambiguously(mostly)

PresidentBarackObama,44thpresidentoftheUnitedStates,…

PresidentObama

Obama‣Whenmighttherebeambiguity?

Dellfoundedwhatwouldbecomehiseponymouscompanyin1984.Dellwaslatertakenprivateinaleveragedbuyout.

‣Maincues:lexicaloverlap,seman/ctypeagreement

Pronouns

‣Maincues:salience,number/genderagreement,eventseman/cs/commonsenseknowledge

PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.He…

PresidentObamametwithChancellorMerkel.He…

Thepoliceman8cketedthedriveraPerheranthestopsignheno8cedabrokentaillight

Thisisthehousewherethebombwasbuiltintotheboatthatcarriedit.

NominalMen8ons

‣Maincues:seman/ctypeagreement/worldknowledge,salience

PresidentObama…Thepresident…

ServeAmericaAct…Thebill

BarackObamaandAngelaMerkel…Theleaders

NBC…Thenetwork

‣ Basiclexicalseman8cs/hypernymy

‣Worldknowledge

‣ Combinesthetwo:Obamaisapresident,Merkelisachancellor,thecommontypeofthoseisleader

Phenomena‣ Salience:distancefeatures‣ Seman8ccompa8bility

‣ Gender:hevs.she‣ Animacy:he/shevs.it

‣ Seman8ctype:MichaelDell(person)vs.Dell(company)

‣ Commonsenseknowledge:abombcanbecarried,aboatcannotbe

‣ CoreferenceisachallengingNLPproblem!Severaldifferentsubproblems,lotsofsourcesofinforma8onthatweneedtoconsider

‣ Hypernymy:anactisabill

‣Worldknowledge:Merkelisaleader

BuildingCoreferenceModels

Rule-basedSystems

‣ Filterpossibleantecedentsbasedonsyntac8candseman8cinforma8on,resolvetotheclosestone

HaghighiandKlein(2008)

‣ Seman8cinforma8onused:numberandgender(automa8callyscraped),headword/stringmatch,someworldknowledge(NBC=network)

PresidentBarackObama

HetheServeAmericaAct

Congress’s

‣ inanimate‣ inanimate

En8ty-centricRuled-basedSystems

RahmanandNg(2009),Raghunathanetal.(2010),Leeetal.(2011)

Obamagaveaspeechonthe“Let’sMove!”program,praisingSamKass.

MichelleObamapromotedherfitnessandnutri8onprogramonThursday.

‣ Needtomakedecisionsglobally:en8ty-centric,“sieve-based”coreference,“easy-first”systemsallrelyonearlierdecisionstodothis

He…

FEMALE

FEMALE

‣ Coreferencedependsoniden8tyofObama,whichinturndependsonothercoreferencelinks

Men8on-RankingSystems

DenisandBaldridge(2008),Fernandesetal.(2012),DurrejandKlein(2013)

PresidentBarackObama

theServeAmericaAct

Congress’s He

New

a1

p(ai = j|x) / exp(w

>f(i, j, x))

1

New

a2

12

123New New

‣ Log-linearmodel

a3 a4

anaphorindex

antecedentindex

document

featuresofmen8onpair+document

FeaturesforLearning-basedSystems

DenisandBaldridge(2008),Fernandesetal.(2012),DurrejandKlein(2013)

PresidentBarackObama

PROPER,MALE,SINGULAR

Ment.distance=3

NoheadmatchAntecedentlength=3 Anaphlength=1

Salience

Seman8ccompa8bility

Pragma8cs

MALE—he Obama—heXreceived—he PROPER—Xsigned

Sent.distance=1

Nostringmatch

[new]PRONOUN[new]he[new]Xsigned[new].X[new]Length=1

receivedtheServe… .Hesignedthebill

PRONOUN,MALE,SINGULAR

NeuralNetworkModels

ClarkandManning(2016)

PresidentBarackObamareceivedtheServe… .Hesignedthebill

antecedentfeats anaphorfeats pairfeatsdistance,headmatch,etc.

‣ Similarinputstolog-linearmodel

Feedforwardneuralnetwork

score

‣Wordembeddings+nonlinearlayerscapturemorecomplexinterac8onsbetweenmen8onandantecedent

Performance

40

50

60

70

80

78.0

65.661.7

55.6

StanfordRule-based(2010) BerkeleyLog-linear(2014)StanfordDeepCoref(2016) Human

CoNLLF1

Incorpora8ngWorldKnowledge

AccuracyPerMen8onClass(Berkeley)

Anaphoricpronouns

Referring:headmatch

6.2%

}

theU.S.president president

Obama he

6.2

82.7

72.0

DavidCameron primeministerReferring:noheadmatch

AccuracyPerMen8onClass(Berkeley)

Anaphoricpronouns

Referring:headmatch

6.2%

}

theU.S.president president

Obama he

6.2

82.7

72.0

DavidCameron primeministerReferring:noheadmatch

AccuracyPerMen8onClass(Berkeley)

Anaphoricpronouns

Referring:headmatch

6.2%

}

theU.S.president president

Obama he

6.2

82.7

72.0

DavidCameron primeministerReferring:noheadmatch

Phenomena‣ Salience‣ Seman8ccompa8bility

‣ Gender‣ Animacy

‣ Seman8ctype

‣ Commonsenseknowledge‣ Hypernymy

‣Worldknowledge

((

))

‣Wordembeddingssortof dothese

‣ Basicfeaturesgetthese

WordEmbeddings

Russia

na8on

China

Iran

‣Wordvectorscapturetopicalsimilarity,arenottrainedtocapturereferen@aliden@ty

Russia’seconomyhasbeensluggish…

…suspectedcollusionwithRussia.The……atriptoRussiainthespring8me

‣ RussiaisnotIran!Possiblycompa8blepairsarelesssimilarthanmanyincompa8blepairs

Phenomena‣ Salience‣ Seman8ccompa8bility

‣ Gender‣ Animacy

‣ Seman8ctype

‣ Commonsenseknowledge‣ Hypernymy

‣Worldknowledge

((

))

XX

‣Wordembeddingssortof dothese

‣ …buttheydon’tdothese

‣ Basicfeaturesgetthese

LeveragingExternalResources‣ HowdowefigureoutwhatkindofthingNBCis?

‣ UseanexternalknowledgebaselikeWikipedia

‣ Knowledgecanimportthefeaturesneededtomakedifficultcoreferencedecisions

JointEn8tyLinkingandCoreference‣ TherearemanythingsNBCcouldmean!

‣ Needtotackleen@tylinkingaswell:figuringoutwhaten8tyagivenoccurrenceofNBCrefersto

‣ Jointmodelsresolveen88estoWikipediaandsimultaneouslyplacecoreferencelinks(DurrejandKlein,2014)

‣ Improvementfromen8tylinkingissmall:~1%onCoNLLmetric

Challenge:NeedComplexInferencesRussia’seconomyhasbeensluggish…TheEasternEuropeanna8on…

Russia…isacountryinnortheastEurasia.

countrystate

na8onland

country ruralarea

Conclusion‣ CoreferenceisachallengingNLPproblem

‣Manyphenomenatocapture,includingsalienceandseman8ccompa8bility

‣Men8on-rankingclassifiersworkprejywell(non-neuralorneural)

‣Worldknowledgeisneededtosolvemanyremainingerrors,butishardtoincorporate