CS388:NaturalLanguageProcessing

GregDurre8

Lecture14:Seman<csI

Recall:Dependencies

DT NNTOVBDDT NNthe housetoranthe dog

‣ Dependencysyntax:syntac<cstructureisdefinedbydependencies‣ Head(parent,governor)connectedtodependent(child,modifier)‣ EachwordhasexactlyoneparentexceptfortheROOTsymbol‣ Dependenciesmustformadirectedacyclicgraph

ROOT

Recall:ShiR-ReduceParsing

IatesomespagheTbolognese

ROOT

‣ State:Stack:[ROOTIate]Buffer:[somespagheTbolognese]

‣ LeR-arc(reduceopera<on):Letdenotethestack�‣ “Poptwoelements,addanarc,putthembackonthestack”

�|w�2, w�1 ! �|w�1 w�1w�2 isnowachildof,

‣ Trainaclassifiertomakethesedecisionssequen<ally—thatclassifiercanparsesentencesforyou

Wherearewenow?

‣ Earlyintheclass:bagsofword(classifiers)=>sequencesofwords(sequencemodeling)

‣Whyisthisuseful?Whatdoesthisallowustodo?

‣ Nowwecanunderstandsentencesintermsoftreestructuresaswell

‣We’regoingtoseehowparsingcanbeasteppingstonetowardsmoreformalrepresenta<onsoflanguagemeaning

Today

‣Modeltheore<cseman<cs

‣ CCGparsingfordatabasequeries

‣ Composi<onalseman<cswithfirst-orderlogic

‣ Lambda-DCSforques<onanswering

‣Montagueseman<cs:

ModelTheore<cSeman<cs

ModelTheore<cSeman<cs‣ Keyidea:cangroundoutnaturallanguageexpressionsinset-theore<cexpressionscalledmodelsofthosesentences

‣ NaturallanguagestatementS=>interpreta<onofSthatmodelsit

‣ Entailment(statementAimpliesstatementB)reducesto:inallworldswhereAistrue,Bistrue

Shelikesgoingtothatrestaurant

‣ Interpreta<on:defineswhosheandthatrestaurantare,makeitabletobeconcretelyevaluatedwithrespecttoaworld

‣ Ourmodelinglanguageisfirst-orderlogic

First-orderLogic

‣ singsisapredicate(withoneargument),func<onf:en<ty→true/false

‣ Powerfullogicformalismincludingthingslikeen<<es,rela<ons,andquan<fica<ons

‣ [[sings]]=denota7on,setofen<<eswhichsing(foundbyexecu<ngthispredicateontheworld—we’llcomebacktothis)

LadyGagasings

‣ sings(LadyGaga)=trueorfalse,havetoexecutethisagainstsomedatabase(world)

Quan<fica<on

‣ ∀xsings(x)∨dances(x)→performs(x)

‣ ∃y∀xfriend(x,y)

‣ Universalquan<fica<on:“forall”operator

‣ Existen<alquan<fica<on:“thereexists”operator

‣ ∀x∃yfriend(x,y)

‣ Sourceofambiguity!“Everyoneisfriendswithsomeone”

“Everyonewhosingsordancesperforms”

“Someonesings”‣ ∃xsings(x)

LogicinNLP‣ Ques<onanswering:

‣ Informa<onextrac<on: LadyGagaandEminemarebothmusicians

∀xmusician(x)=>performer(x)

musician(LadyGaga)∧musician(Eminem)

Then:performer(LadyGaga)∧performer(Eminem)

WhoarealltheAmericansingersnamedAmy?

λx.na<onality(x,USA)∧sings(x)∧firstName(x,Amy)

‣ Func<onthatmapsfromxtotrue/false,likefilter.Executethisontheworldtoanswertheques<on

‣ Cannowdoreasoning.Maybeknow:

‣ Lambdacalculus:powerfulsystemforexpressingthesefunc<ons

Composi<onalSeman<cswithFirst-OrderLogic

MontagueSeman<cs

Id Name Alias Birthdate Sings?e470 StefaniGermano8a LadyGaga 3/28/1986 Te728 MarshallMathers Eminem 10/17/1972 T

‣ Databasecontainingen<<es,predicates,etc.

‣ Sentenceexpressessomethingabouttheworldwhichiseithertrueorfalse

NP VP

NNP NNP

S

VBPLadyGaga sings

‣ Denota<on:evalua<onofsomeexpressionagainstthisdatabase

‣[[LadyGaga]] = e470

denota<onofthisstringisanen<ty

‣[[sings(e470)]] = True

denota<onofthisexpressionisT/F

MontagueSeman<cs

NP VP

NNP NNP

S

VBPLadyGaga sings

e470

λy. sings(y)takesoneargument(y,theen<ty)andreturnsalogicalformsings(y)

λy. sings(y)

sings(e470)

‣Wecanusethesyntac<cparseasabridgetothelambda-calculusrepresenta<on,buildupalogicalform(ourmodel)composi7onally

func<onapplica<on:applythistoe470ID

ParsestoLogicalForms

NP

VPNNP NNP

S

VBPLadyGaga

sings

e470

λy. sings(y)

sings(e470) ∧ dances(e470)

VP

CC VP

VBPdancesλy. dances(y)

and

VP:λy.a(y)∧b(y)->VP:λy.a(y)CCVP:λy.b(y)

λy. sings(y) ∧ dances(y)

‣ Generalrules:S:f(x)->NP:xVP:f

ParsestoLogicalForms

NP

NNP NNP

S

VBDLadyGaga was

e470

λx.λy. born(y,x)

born(e470,3/28/1986)

VP

NP

March28,1986born

λy. born(y, 3/28/1986)

VBN

VP

λy. born(y, 3/28/1986)

‣ Howtohandletense:shouldweindicatethatthishappenedinthepast?‣ Func<ontakestwoarguments:firstx(date),theny(en<ty)

3/28/1986

Trickythings‣ Adverbs/temporality:LadyGagasangwellyesterday

∃e. type(e,sing) ∧ agent(e,e470) ∧ manner(e,well) ∧ time(e,…)‣ “Neo-Davidsonian”viewofevents:thingswithmanyproper<es:

‣ Quan<fica<on:Everyoneisfriendswithsomeone

‣ Generic:Catseatmice(allcatseatmice?mostcats?somecats?)

∀x ∃y friend(x,y)∃y ∀x friend(x,y)(differentfriends)(onefriend)

‣ Samesyntac<cparseforboth!Sosyntaxdoesn'tresolveallambigui<es

sings(Lady Gaga, time=yesterday, manner=well)

‣ Indefinite:Amyateawaffle ∃w. waffle(w) ∧ ate(Amy,w)

Seman<cParsing

‣ Forques<onanswering,syntac<cparsingdoesn’ttellyoueverythingyouwanttoknow,butindicatestherightstructure

‣ Solu<on:seman7cparsing:manyformsofthistaskdependingonseman<cformalisms

‣ Twotoday:CCG(lookslikewhatwe’vebeendoing)andlambda-DCS

‣ Applica<ons:databasequerying/ques<onanswer:producelambda-calculusexpressionsthatcanbeexecutedinthesecontexts

CCGParsing

CombinatoryCategorialGrammar‣ Steedman+Szabolcsi(1980s):formalismbridgingsyntaxandseman<cs

‣ Syntac<ccategories(forthislecture):S,NP,“slash”categories

‣ S\NP:“ifIcombinewithanNPonmyleRside,Iformasentence”—verb

NP S\NP

Eminem singse728 λy. sings(y)

Ssings(e728)

‣ Parallelderiva<onsofsyntac<cparseandlambdacalculusexpression

‣Whenyouapplythis,therehastobeaparallelinstanceoffunc<onapplica<onontheseman<csside

CombinatoryCategorialGrammar‣ Steedman+Szabolcsi1980s:formalismbridgingsyntaxandseman<cs

‣ Syntac<ccategories(forthislecture):S,NP,“slash”categories‣ S\NP:“ifIcombinewithanNPonmyleRside,Iformasentence”—verb

‣ (S\NP)/NP:“IneedanNPonmyrightandthenonmyleR”—verbwithadirectobject

NP S\NP

Eminem singse728 λy. sings(y)

Ssings(e728)

NP (S\NP)/NP

Oklahoma borderse101

Texase89NP

λx.λy borders(y,x)

S\NPλy borders(y,e89)

Sborders(e101,e89)

CCGParsing

Ze8lemoyerandCollins(2005)

‣ “What”isaverycomplextype:needsanounandneedsaS\NPtoformasentence.S\NPisbasicallyaverbphrase(borderTexas)

CCGParsing

Ze8lemoyerandCollins(2005)

‣ “What”isaverycomplextype:needsanounandneedsaS\NPtoformasentence.S\NPisbasicallyaverbphrase(borderTexas)

‣ Lexiconishighlyambiguous—allthechallengeofCCGparsingisinpickingtherightlexiconentries

CCGParsing

Slidecredit:DanKlein

‣ “to”needsanNP(des<na<on)andN(parent)

CCGParsing

Ze8lemoyerandCollins(2005)

‣Manywaystobuildtheseparsers

‣ Oneapproach:runa“supertagger”(tagsthesentencewithcomplexlabels),thenruntheparser

‣ Parsingiseasyonceyouhavethetags,sowe’vereducedittoa(hard)taggingproblem

BuildingCCGParsers

Ze8lemoyerandCollins(2005)

‣Model:log-linearmodeloverderiva<onswithfeaturesonrules:

P (d|x) / expw>

X

r2d

f(r, x)

!

‣ CanparsewithavariantofCKYEminem sings

NP S\NPe728 λy. sings(y)

Ssings(e728)

f

f

f =Indicator(S\NP->sings)

=Indicator(S->NPS\NP)

BuildingCCGParsers

Ze8lemoyerandCollins(2005)

‣ Trainingdatalookslikepairsofsentencesandlogicalforms

WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)

‣ TexascorrespondstoNP|e89inthelogicalform(easytofigureout)

(S/(S\NP))/N|λf.λg.λx. f(x) ∧ g(x)‣Whatcorrespondsto

‣ Howdoweinferthatwithoutbeingtoldit?

‣ Problem:wedon’tknowthederiva<on

Lexicon

WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)

‣ Anysubstringcanparsetoanyoftheseinthelexicon

‣ Chunksinferredfromthelogicformbasedonrules:

‣ GENLEX:takessentenceSandlogicalformL.BreakuplogicalformintochunksC(L),assumeanysubstringofSmightmaptoanychunk

‣ Texas->NP:e89iscorrect‣ borderTexas->NP:e89‣WhatstatesborderTexas->NP:e89… Ze8lemoyerandCollins(2005)

‣ NP:e89 ‣ (S\NP)/NP:λx.λy. borders(x,y)

GENLEX

‣ Verycomplexandhand-engineeredwayoftakinglambdacalculusexpressionsand“backsolving”forthederiva<on

Ze8lemoyerandCollins(2005)

Learning

Ze8lemoyerandCollins(2005)

‣ Itera<veprocedureliketheEMalgorithm:es<mate“best”parsesthatderiveeachlogicalform,retraintheparserusingtheseparseswithsupervisedlearning

‣We’lltalkaboutasimplerformofthisinafewslides

Applica<ons

‣ GeoQuery:answeringques<onsaboutstates(~80%accuracy)

‣ Jobs:answeringques<onsaboutjobpos<ngs(~80%accuracy)

‣ ATIS:flightsearch

‣ CandowellonallofthesetasksifyouhandcraRsystemsanduseplentyoftrainingdata:thesedomainsaren’tthatrich

‣WhataboutbroaderQA?

Lambda-DCS

Lambda-DCS

Liangetal.(2011),Liang(2013)

‣ Dependency-basedcomposi<onalseman<cs—originalversionwaslesspowerfulthanlambdacalculus,lambda-DCSisaspowerful

‣ DesignedinthecontextofbuildingaQAsystemfromFreebase

‣ Freebase:setofen<<esandrela<ons

AliceSmith

BobCooper

Sea8le

March15,1961 WashingtonDateOfBirth

PlaceOfBirth

PlaceOfBirth

CapitalOf

‣ [[PlaceOfBirth]]=setofpairsof(person,loca<on)

Lambda-DCS

Liangetal.(2011),Liang(2013)

Lambda-DCS Lambdacalculus

Seattle λx. x = SeattlePlaceOfBirth λx.λy. PlaceOfBirth(x,y)

PlaceOfBirth.Seattle λx. PlaceOfBirth(x,Seattle)

‣ LookslikeatreefragmentoverFreebase

Sea8lePlaceOfBirth???

Profession.Scientist ∧ PlaceOfBirth.Seattle

λx. Profession(x,Scientist) ∧ PlaceOfBirth(x,Seattle)

Lambda-DCS

Liangetal.(2011),Liang(2013)

AliceSmith

BobCooper

Sea8le

March15,1961 WashingtonDateOfBirth

PlaceOfBirth

PlaceOfBirth

CapitalOf

Profession

Scien<st

Profession.Scientist ∧ PlaceOfBirth.Seattle“listofscien<stsborninSea8le”

‣ ExecutethisfragmentagainstFreebase,returnsAliceSmith(andothers)

???

Sea8lePlaceOfBirthProfession

Scien<st

ParsingintoLambda-DCS

Berantetal.(2013)

‣ Buildingthelexicon:moresophis<catedprocessthanGENLEX,butcanhandlethousandsofpredicates

‣ Log-linearmodelwithfeaturesonrules: P (d|x) / expw>

X

r2d

f(r, x)

!

‣ Deriva<ondonsentencex:

‣ SimilartoCRFparsers

‣ Nomoreexplicitsyntax inthesederiva<ons likewehadinCCG

ParsingwithLambda-DCS

Berantetal.(2013)

‣ Learnjustfromques<on-answerpairs:maximizethelikelihoodoftherightdenota<onywiththederiva<ondmarginalizedout

Foreachexample:Runbeamsearchtogetasetofderiva<ons

Letd*=highest-scoringderiva<oninthebeamwithcorrectdenota7onDoastructuredperceptronupdatetowardsd*awayfromd

Letd=highest-scoringderiva<oninthebeam

sumoverderiva<onsdsuchthatthedenota<onofdonknowledgebaseKisyi

Takeaways

‣ Canrepresentmeaningwithfirstorderlogicandlambdacalculus

‣ Usefulforqueryingdatabases,ques<onanswering,etc.

‣ Canbridgesyntaxandseman<csandcreateseman<cparsersthatcaninterpretlanguageintolambda-calculusexpressions

‣ Next<me:neuralnetmethodsfordoingthisthatrelylessonhavingexplicitgrammars