log-linear models with structured outputs (continued)dasmith/inlp2009/lect15-cs... ·...

Log-Linear Modelswith Structured Outputs

(continued)

Introduction to Natural Language ProcessingComputer Science 585—Fall 2009

University of Massachusetts Amherst

David Smith

Overview

• What computations do we need?

• Smoothing log-linear models

• MEMMs vs. CRFs again

• Action-based parsing and dependency parsing

Recipe for Conditional Training of p(y | x)

1. Gather constraints/features from training data

2. Initialize all parameters to zero

3. Classify training data with current parameters; calculate expectations

4. Gradient is

5. Take a step in the direction of the gradient

6. Repeat from 3 until convergence 43 43

Recipe for a Conditional

MaxEnt Classifier

1. Gather constraints from training data:

2. Initialize all parameters to zero.

3. Classify training data with current parameters. Calculateexpectations.

4. Gradient is


6. Until convergence, return to step 3.

43

43 43


MaxEnt Classifier




4. Gradient is



4343 43


MaxEnt Classifier




4. Gradient is



43





4. Gradient is




MaxEnt Classifier




4. Gradient is



43

43 43


MaxEnt Classifier




4. Gradient is



4343 43


MaxEnt Classifier




4. Gradient is



43

Where have we seen expected counts before?





4. Gradient is




MaxEnt Classifier




4. Gradient is



43

43 43


MaxEnt Classifier




4. Gradient is



4343 43


MaxEnt Classifier




4. Gradient is



43

Where have we seen expected counts before?

EM!

Gradient-Based Training

• λ <- λ + rate * Gradient(F)

• After all training examples? (batch)

• After every example? (on-line)

• Use second derivative?

• A big field: numerical optimization

Overfitting• If we have too many features, we can choose

weights to model the training data perfectly

• If we have a feature that only appears in spam training, not ham training, it will get weight ∞ to maximize p(spam | feature) at 1.

• These behaviors

• Overfit the training data

• Will probably do poorly on test data

Solutions to Overfitting• Throw out rare features.

• Require every feature to occur > 4 times, and > 0 times with ling, and > 0 times with spam.

• Only keep, e.g., 1000 features.

• Add one at a time, always greedily picking the one that most improves performance on held-out data.

• Smooth the observed feature counts.

• Smooth the weights by using a prior.

• max p(λ|data) = max p(λ, data) =p(λ)p(data|λ)

• decree p(λ) to be high when most weights close to 0

Smoothing with Priors• What if we had a prior expectation that parameter values

wouldn’t be very large?

• We could then balance evidence suggesting large (or infinite) parameters against our prior expectation.

• The evidence would never totally defeat the prior, and parameters would be smoothed (and kept finite)

• We can do this explicitly by changing the optimization objective to maximum posterior likelihood:

log P(y, λ | x) = log P(λ) + log P(y | x, λ)

Posterior Prior Likelihood

42 42

42

Parsing as Structured Prediction

Shift-reduce parsing

Stack Input remaining Action() Book that flight shift(Book) that flight reduce, Verb ! book, (Choice #1 of 2)(Verb) that flight shift(Verb that) flight reduce, Det ! that(Verb Det) flight shift(Verb Det flight) reduce, Noun ! flight(Verb Det Noun) reduce, NOM ! Noun(Verb Det NOM) reduce, NP ! Det NOM(Verb NP) reduce, VP ! Verb NP(Verb) reduce, S ! V(S) SUCCESS!

Ambiguity may lead to the need for backtracking.

Shift-reduce parsing

Stack Input remaining Action() Book that flight shift(Book) that flight reduce, Verb ! book, (Choice #1 of 2)(Verb) that flight shift(Verb that) flight reduce, Det ! that(Verb Det) flight shift(Verb Det flight) reduce, Noun ! flight(Verb Det Noun) reduce, NOM ! Noun(Verb Det NOM) reduce, NP ! Det NOM(Verb NP) reduce, VP ! Verb NP(Verb) reduce, S ! V(S) SUCCESS!

Ambiguity may lead to the need for backtracking.

Train log-linear model of p(action | context)

11

Word Dependency Parsing

He reckons the current account deficit will narrow to only 1.8 billion in September.Raw sentence

slide adapted from Yuji Matsumoto

11



Part-of-speech tagging

He reckons the current account deficit will narrow to only 1.8 billion in September. PRP VBZ DT JJ NN NN MD VB TO RB CD CD IN NNP .

POS-tagged sentence


11





POS-tagged sentence

Word dependency parsing


Word dependency parsed sentenceHe reckons the current account deficit will narrow to only 1.8 billion in September .

ROOT

11





POS-tagged sentence




SUBJ

ROOT

S-COMP

11





POS-tagged sentence




SUBJ

ROOT

S-COMP

SUBJCOMP

11





POS-tagged sentence




SUBJ

ROOT

S-COMP

SUBJCOMP

COMP

11





POS-tagged sentence




SUBJ

ROOT

S-COMP

SUBJ

SPEC

MODMOD

COMPCOMP

11

MOD





POS-tagged sentence




SUBJ

ROOT

S-COMP

SUBJ

SPEC

MODMOD

COMPCOMP

12 7

Great ideas in NLP: Log-linear models (Berger, della Pietra, della Pietra 1996; Darroch & Ratcliff 1972)

In the beginning, we used generative models. p(A) * p(B | A) * p(C | A,B) * p(D | A,B,C) * …

12 7



each choice depends on a limited part of the history

12 7



each choice depends on a limited part of the history

but which dependencies to allow?what if they’re all worthwhile?

p(D | A,B,C)?p(D | A,B,C)?

… p(D | A,B) * p(C | A,B,D)?

13 8


p(A) * p(B | A) * p(C | A,B) * p(D | A,B,C) * …which dependencies to allow? (given limited training data)

13 8


In the beginning, we used generative models.


13 8




(1/Z) * Φ(A) * Φ(B,A) * Φ(C,A) * Φ(C,B)

* Φ(D,A,B) * Φ(D,B,C) * Φ(D,A,C) *

…throw them all in!

13 8



Solution: Log-linear (max-entropy) modeling

Features may interact in arbitrary ways Iterative scaling keeps adjusting the feature weights

until the model agrees with the training data.


(1/Z) * Φ(A) * Φ(B,A) * Φ(C,A) * Φ(C,B)

* Φ(D,A,B) * Φ(D,B,C) * Φ(D,A,C) *

…throw them all in!

14 9

How about structured outputs?

14

Log-linear models great for n-way classification

9


14

Log-linear models great for n-way classification Also good for predicting sequences

9


find preferred tags

v a n

14


9


but to allow fast dynamic programming, only use n-gram featuresfind preferred tags

v a n

14


Also good for dependency parsing

9


but to allow fast dynamic programming, only use n-gram features

…find preferred links…

find preferred tags

v a n

14


Also good for dependency parsing

9


but to allow fast dynamic programming, only use n-gram features

but to allow fast dynamic programming or MST parsing,only use single-edge features…find preferred links…

find preferred tags

v a n

14 9

How about structured outputs?but to allow fast dynamic programming or MST parsing,only use single-edge features

…find preferred links…

15 10

How about structured outputs?but to allow fast dynamic programming or MST parsing,only use single-edge features…find preferred links…

16

Edge-Factored Parsers (McDonald et al. 2005)

Byl jasný studený dubnový den a hodiny odbíjely třináctou

“It bright cold day April and clocks were thirteen”was a in the striking

Is this a good edge?

16





yes, lots of green ...

17





17





jasný den(“bright day”)

18






V A A A N J N V C

18






jasný N(“bright NOUN”)

V A A A N J N V C

19







V A A A N J N V C

19







V A A A N J N V C

A N

20







V A A A N J N V C

A N

20







V A A A N J N V C

A Npreceding

conjunction A N

21




How about this competing edge?

V A A A N J N V C

21





V A A A N J N V C

not as good, lots of red ...

22





V A A A N J N V C

22





V A A A N J N V C

jasný hodiny(“bright clocks”)

22





V A A A N J N V C


... undertrained ...

23





V A A A N J N V C

byl jasn stud dubn den a hodi odbí třin



23





V A A A N J N V C




jasn hodi(“bright clock,”

stems only)

24





V A A A N J N V C


stems only)




24





V A A A N J N V C


stems only)


Aplural Nsingular



25







V A A A N J N V C


stems only)


Aplural Nsingular

25







V A A A N J N V C


stems only)


Aplural Nsingular A N

where N followsa conjunction

26

jasný


Byl studený dubnový den a hodiny odbíjely třináctou


V A A A N J N V C


Which edge is better? “bright day” or “bright clocks”?

27

jasný




V A A A N J N V C


27

jasný




V A A A N J N V C


Which edge is better?

27

jasný




V A A A N J N V C


Which edge is better? our current weight vector

27

jasný




V A A A N J N V C


Which edge is better? Score of an edge e = θ ⋅ features(e)

our current weight vector

27

jasný




V A A A N J N V C


Which edge is better? Score of an edge e = θ ⋅ features(e) Standard algos valid parse with max total score


28




28




can’t have both(one parent per word)

28





can‘t have both(no crossing links)

28






Can’t have all three(no cycles)

28







Thus, an edge may lose (or win) because of a consensus of other edges.

29

Finding Highest-Scoring Parse

The cat in the hat wore a stovepipe. ROOT

Convert to context-free grammar (CFG) Then use dynamic programming

29




Thecat

in

thehat

wore

astovepipe

ROOT

let’s vertically stretch this graph drawing

29




each subtree is a linguistic constituent(here a noun phrase)

Thecat

in

thehat

wore

astovepipe

ROOT

let’s vertically stretch this graph drawing

30



Thecat

in

thehat

wore

astovepipe

ROOT

30



Thecat

in

thehat

wore

astovepipe

ROOT


CKY algorithm for CFG parsing is O(n3)

30



Thecat

in

thehat

wore

astovepipe

ROOT


CKY algorithm for CFG parsing is O(n3) Unfortunately, O(n5) in this case

30



Thecat

in

thehat

wore

astovepipe

ROOT



to score “cat wore” link, not enough to know this is NP

30



Thecat

in

thehat

wore

astovepipe

ROOT



to score “cat wore” link, not enough to know this is NP must know it’s rooted at “cat”

30



Thecat

in

thehat

wore

astovepipe

ROOT



to score “cat wore” link, not enough to know this is NP must know it’s rooted at “cat” so expand nonterminal set by O(n): {NPthe, NPcat, NPhat, ...}

30



Thecat

in

thehat

wore

astovepipe

ROOT so CKY’s “grammar constant” is no longer constant



to score “cat wore” link, not enough to know this is NP must know it’s rooted at “cat” so expand nonterminal set by O(n): {NPthe, NPcat, NPhat, ...}

31



Thecat

in

thehat

wore

astovepipe

ROOT

31



Thecat

in

thehat

wore

astovepipe

ROOT


CKY algorithm for CFG parsing is O(n3) Unfortunately, O(n5) in this case Solution: Use a different decomposition (Eisner 1996)

Back to O(n3)

32

Spans vs. constituents

Two kinds of substring.»Constituent of the tree: links to the rest only

through its headword (root).

»Span of the tree: links to the rest only through its endwords.



Decomposing a tree into spans


The cat

wore a stovepipe. ROOTcat in the hat wore

+

+

in the hat worecat in +

hat worein the hat +

cat in the hat wore a stovepipe. ROOT

34


34




Back to O(n3)

34




Back to O(n3) Can play usual tricks for dynamic programming parsing

Further refining the constituents or spans Allow prob. model to keep track of even more internal information

A*, best-first, coarse-to-fine Training by EM etc.

34




Back to O(n3) Can play usual tricks for dynamic programming parsing

Further refining the constituents or spans Allow prob. model to keep track of even more internal information

A*, best-first, coarse-to-fine Training by EM etc. require “outside” probabilities

of constituents, spans, or links

35

Hard Constraints on Valid Trees

Score of an edge e = θ ⋅ features(e) Standard algos valid parse with max total score





Thus, an edge may lose (or win) because of a consensus of other edges.

35

Hard Constraints on Valid Trees


36

Non-Projective Parses


The “projectivity” restriction.Do we really want it?

36

talk




I give a on bootstrappingtomorrowROOT ‘ll

36

talk




I give a on bootstrappingtomorrowROOT ‘ll

subtree rooted at “talk”is a discontiguous noun phrase

37


I give a on bootstrappingtalk tomorrowROOT ‘ll

occasional non-projectivity in English

37


ista meam norit gloria canitiemROOT



frequent non-projectivity in Latin, etc.

37




That glory may-know my going-gray (i.e., it shall last till I go gray)



37




thatNOM myACC may-know gloryNOM going-grayACC

That glory may-know my going-gray (i.e., it shall last till I go gray)



38

Finding highest-scoring non-projective tree Consider the sentence “John saw Mary” (left). The Chu-Liu-Edmonds algorithm finds the maximum-

weight spanning tree (right) – may be non-projective. Can be found in time O(n2).

9

10

30

20

30 0

11

3

9

root

John

saw

Mary

10

30

30

root

John

saw

Mary

slide thanks to Dragomir Radev

Every node selects best parentIf cycles, contract them and repeat

39

Summing over all non-projective treesFinding highest-scoring non-projective tree Consider the sentence “John saw Mary” (left). The Chu-Liu-Edmonds algorithm finds the maximum-

weight spanning tree (right) – may be non-projective. Can be found in time O(n2).

How about total weight Z of all trees? How about outside probabilities or gradients? Can be found in time O(n3) by matrix determinants and

inverses (Smith & Smith, 2007).

slide thanks to Dragomir Radev

40

Graph Theory to the Rescue!

Tutte’s Matrix-Tree Theorem (1948)

The determinant of the Kirchoff (aka Laplacian) adjacency matrix of directed graph G without row and column r is equal to the sum of scores of all directed spanning trees of G rooted at node r.

40




Exactly the Z we need!

40




Exactly the Z we need!

O(n3) time!

41

Building the Kirchoff (Laplacian) Matrix

• Negate edge scores• Sum columns

(children)• Strike root row/col.• Take determinant

41

Building the Kirchoff (Laplacian) Matrix

• Negate edge scores• Sum columns

(children)• Strike root row/col.• Take determinant

N.B.: This allows multiple children of root, but see Koo et al. 2007.

42

Why Should This Work?Chu-Liu-Edmonds analogy:Every node selects best parentIf cycles, contract and recur

Clear for 1x1 matrix; use induction

Undirected case; special root cases for directed

log-linear models with structured outputs (continued)dasmith/inlp2009/lect15-cs... ·...

Documents