statistical nlp spring 2011
DESCRIPTION
Statistical NLP Spring 2011. Lecture 17: Parsing III. Dan Klein – UC Berkeley. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A A. Treebank PCFGs. [Charniak 96]. Use PCFGs for broad coverage parsing - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/1.jpg)
Statistical NLPSpring 2011
Lecture 17: Parsing IIIDan Klein – UC Berkeley
![Page 2: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/2.jpg)
Treebank PCFGs
Use PCFGs for broad coverage parsing Can take a grammar right off the trees (doesn’t work well):
ROOT S 1
S NP VP . 1
NP PRP 1
VP VBD ADJP 1
…..
Model F1
Baseline 72.0
[Charniak 96]
![Page 3: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/3.jpg)
Conditional Independence?
Not every NP expansion can fill every NP slot A grammar with symbols like “NP” won’t be context-free Statistically, conditional independence too strong
![Page 4: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/4.jpg)
The Game of Designing a Grammar
Annotation refines base treebank symbols to improve statistical fit of the grammar Structural annotation
![Page 5: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/5.jpg)
A Fully Annotated (Unlex) Tree
![Page 6: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/6.jpg)
Some Test Set Results
Beats “first generation” lexicalized parsers. Lots of room to improve – more complex models next.
Parser LP LR F1 CB 0 CB
Magerman 95 84.9 84.6 84.7 1.26 56.6
Collins 96 86.3 85.8 86.0 1.14 59.9
Unlexicalized 86.9 85.7 86.3 1.10 60.3
Charniak 97 87.4 87.5 87.4 1.00 62.1
Collins 99 88.7 88.6 88.6 0.90 67.1
![Page 7: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/7.jpg)
Annotation refines base treebank symbols to improve statistical fit of the grammar Structural annotation [Johnson ’98, Klein and
Manning 03] Head lexicalization [Collins ’99, Charniak ’00]
The Game of Designing a Grammar
![Page 8: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/8.jpg)
Problems with PCFGs
If we do no annotation, these trees differ only in one rule: VP VP PP NP NP PP
Parse will go one way or the other, regardless of words We addressed this in one way with unlexicalized grammars (how?) Lexicalization allows us to be sensitive to specific words
![Page 9: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/9.jpg)
Problems with PCFGs
What’s different between basic PCFG scores here? What (lexical) correlations need to be scored?
![Page 10: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/10.jpg)
Lexicalized Trees
Add “headwords” to each phrasal node Syntactic vs. semantic
heads Headship not in (most)
treebanks Usually use head rules,
e.g.: NP:
Take leftmost NP Take rightmost N* Take rightmost JJ Take right child
VP: Take leftmost VB* Take leftmost VP Take left child
![Page 11: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/11.jpg)
Lexicalized PCFGs? Problem: we now have to estimate probabilities like
Never going to get these atomically off of a treebank
Solution: break up derivation into smaller steps
![Page 12: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/12.jpg)
Lexical Derivation Steps A derivation of a local tree [Collins 99]
Choose a head tag and word
Choose a complement bag
Generate children (incl. adjuncts)
Recursively derive children
![Page 13: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/13.jpg)
Lexicalized CKY
bestScore(X,i,j,h)
if (j = i+1)
return tagScore(X,s[i])
else
return
max max score(X[h]->Y[h] Z[h’]) *
bestScore(Y,i,k,h) *
bestScore(Z,k,j,h’)
max score(X[h]->Y[h’] Z[h]) *
bestScore(Y,i,k,h’) *
bestScore(Z,k,j,h)
Y[h] Z[h’]
X[h]
i h k h’ j
k,h’,X->YZ
(VP->VBD )[saw] NP[her]
(VP->VBD...NP )[saw]
k,h’,X->YZ
![Page 14: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/14.jpg)
Quartic Parsing Turns out, you can do (a little) better [Eisner 99]
Gives an O(n4) algorithm Still prohibitive in practice if not pruned
Y[h] Z[h’]
X[h]
i h k h’ j
Y[h] Z
X[h]
i h k j
![Page 15: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/15.jpg)
Pruning with Beams
The Collins parser prunes with per-cell beams [Collins 99] Essentially, run the O(n5) CKY Remember only a few hypotheses for
each span <i,j>. If we keep K hypotheses at each
span, then we do at most O(nK2) work per span (why?)
Keeps things more or less cubic
Also: certain spans are forbidden entirely on the basis of punctuation (crucial for speed)
Y[h] Z[h’]
X[h]
i h k h’ j
![Page 16: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/16.jpg)
Pruning with a PCFG
The Charniak parser prunes using a two-pass approach [Charniak 97+] First, parse with the base grammar For each X:[i,j] calculate P(X|i,j,s)
This isn’t trivial, and there are clever speed ups Second, do the full O(n5) CKY
Skip any X :[i,j] which had low (say, < 0.0001) posterior Avoids almost all work in the second phase!
Charniak et al 06: can use more passes Petrov et al 07: can use many more passes
![Page 17: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/17.jpg)
Prune?
For each chart item X[i,j], compute posterior probability:
… QP NP VP …
coarse:
refined:
E.g. consider the span 5 to 12:
< threshold
![Page 18: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/18.jpg)
Pruning with A*
You can also speed up the search without sacrificing optimality
For agenda-based parsers: Can select which items to
process first Can do with any “figure of
merit” [Charniak 98] If your figure-of-merit is a
valid A* heuristic, no loss of optimiality [Klein and Manning 03]
X
n0 i j
![Page 19: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/19.jpg)
Projection-Based A*
Factory payrolls fell in Sept.
NP PP
VP
S
Factory payrolls fell in Sept.
payrolls in
fell
fellFactory payrolls fell in Sept.
NP:payrolls PP:in
VP:fell
S:fellSYNTACTIC SEMANTIC
![Page 20: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/20.jpg)
A* Speedup
Total time dominated by calculation of A* tables in each projection… O(n3)
0102030
405060
0 5 10 15 20 25 30 35 40
Length
Tim
e (
sec)
Combined Phase
Dependency Phase
PCFG Phase
![Page 21: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/21.jpg)
Results
Some results Collins 99 – 88.6 F1 (generative lexical) Charniak and Johnson 05 – 89.7 / 91.3 F1 (generative
lexical / reranked) Petrov et al 06 – 90.7 F1 (generative unlexical) McClosky et al 06 – 92.1 F1 (gen + rerank + self-
train)
However Bilexical counts rarely make a difference (why?) Gildea 01 – Removing bilexical counts costs < 0.5 F1
![Page 22: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/22.jpg)
Annotation refines base treebank symbols to improve statistical fit of the grammar Structural annotation Head lexicalization Automatic clustering?
The Game of Designing a Grammar
![Page 23: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/23.jpg)
Latent Variable Grammars
Parse Tree Sentence Parameters
...
Derivations
![Page 24: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/24.jpg)
Forward
Learning Latent Annotations
EM algorithm:
X1
X2X7X4
X5 X6X3
He was right
.
Brackets are known Base categories are known Only induce subcategories
Just like Forward-Backward for HMMs. Backward
![Page 25: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/25.jpg)
Refinement of the DT tag
DT
DT-1 DT-2 DT-3 DT-4
![Page 26: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/26.jpg)
Hierarchical refinement
![Page 27: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/27.jpg)
Hierarchical Estimation Results
74
76
78
80
82
84
86
88
90
100 300 500 700 900 1100 1300 1500 1700
Total Number of grammar symbols
Par
sing
acc
urac
y (F
1)
Model F1
Flat Training 87.3
Hierarchical Training 88.4
![Page 28: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/28.jpg)
Refinement of the , tag Splitting all categories equally is wasteful:
![Page 29: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/29.jpg)
Adaptive Splitting
Want to split complex categories more Idea: split everything, roll back splits which
were least useful
![Page 30: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/30.jpg)
Adaptive Splitting Results
Model F1
Previous 88.4
With 50% Merging 89.5
![Page 31: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/31.jpg)
0
5
10
15
20
25
30
35
40
NP
VP PP
AD
VP S
AD
JP
SB
AR QP
WH
NP
PR
N
NX
SIN
V
PR
T
WH
PP
SQ
CO
NJP
FR
AG
NA
C
UC
P
WH
AD
VP
INT
J
SB
AR
Q
RR
C
WH
AD
JP X
RO
OT
LST
Number of Phrasal Subcategories
![Page 32: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/32.jpg)
Number of Lexical Subcategories
0
10
20
30
40
50
60
70
NN
P JJ
NN
SN
NV
BN
RB
VB
GV
BV
BD
CD IN
VB
ZV
BP
DT
NN
PS
CC
JJR
JJS :
PR
PP
RP
$M
DR
BR
WP
PO
SP
DT
WR
B-L
RB
- .E
XW
P$
WD
T-R
RB
- ''F
WR
BS
TO $
UH , ``
SY
MR
PLS #
![Page 33: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/33.jpg)
Learned Splits
Proper Nouns (NNP):
Personal pronouns (PRP):
NNP-14 Oct. Nov. Sept.
NNP-12 John Robert James
NNP-2 J. E. L.
NNP-1 Bush Noriega Peters
NNP-15 New San Wall
NNP-3 York Francisco Street
PRP-0 It He I
PRP-1 it he they
PRP-2 it them him
![Page 34: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/34.jpg)
Relative adverbs (RBR):
Cardinal Numbers (CD):
RBR-0 further lower higher
RBR-1 more less More
RBR-2 earlier Earlier later
CD-7 one two Three
CD-4 1989 1990 1988
CD-11 million billion trillion
CD-0 1 50 100
CD-3 1 30 31
CD-9 78 58 34
Learned Splits
![Page 35: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/35.jpg)
Coarse-to-Fine Inference Example: PP attachment
?????????
![Page 36: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/36.jpg)
Hierarchical Pruning
… QP NP VP …coarse:
split in two: … QP1
QP2
NP1 NP2 VP1 VP2 …
… QP1
QP1
QP3
QP4
NP1 NP2 NP3 NP4 VP1 VP2 VP3 VP4 …split in four:
split in eight: … … … … … … … … … … … … … … … … …
![Page 37: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/37.jpg)
Bracket Posteriors
![Page 38: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/38.jpg)
1621 min111 min35 min
15 min(no search error)
![Page 39: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/39.jpg)
Final Results (Accuracy)
≤ 40 words
F1
all
F1
EN
G
Charniak&Johnson ‘05 (generative) 90.1 89.6
Split / Merge 90.6 90.1
GE
R
Dubey ‘05 76.3 -
Split / Merge 80.8 80.1
CH
N
Chiang et al. ‘02 80.0 76.6
Split / Merge 86.3 83.4
Still higher numbers from reranking / self-training methods
![Page 40: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/40.jpg)
Parse Reranking
Assume the number of parses is very small We can represent each parse T as an arbitrary feature vector (T)
Typically, all local rules are features Also non-local features, like how right-branching the overall tree is [Charniak and Johnson 05] gives a rich set of features
![Page 41: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/41.jpg)
1-Best Parsing
![Page 42: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/42.jpg)
K-Best Parsing[Huang and Chiang 05, Pauls, Klein, Quirk 10]
![Page 43: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/43.jpg)
K-Best Parsing
![Page 44: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/44.jpg)
Dependency Parsing
Lexicalized parsers can be seen as producing dependency trees
Each local binary tree corresponds to an attachment in the dependency graph
questioned
lawyer witness
the the
![Page 45: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/45.jpg)
Dependency Parsing
Pure dependency parsing is only cubic [Eisner 99]
Some work on non-projective dependencies Common in, e.g. Czech parsing Can do with MST algorithms [McDonald and Pereira 05]
Y[h] Z[h’]
X[h]
i h k h’ j
h h’
h
h k h’
![Page 46: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/46.jpg)
Shift-Reduce Parsers
Another way to derive a tree:
Parsing No useful dynamic programming search Can still use beam search [Ratnaparkhi 97]
![Page 47: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/47.jpg)
Data-oriented parsing:
Rewrite large (possibly lexicalized) subtrees in a single step
Formally, a tree-insertion grammar Derivational ambiguity whether subtrees were generated
atomically or compositionally Most probable parse is NP-complete
![Page 48: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/48.jpg)
TIG: Insertion
![Page 49: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/49.jpg)
Tree-adjoining grammars
Start with local trees Can insert structure
with adjunction operators
Mildly context-sensitive
Models long-distance dependencies naturally
… as well as other weird stuff that CFGs don’t capture well (e.g. cross-serial dependencies)
![Page 50: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/50.jpg)
TAG: Long Distance
![Page 51: Statistical NLP Spring 2011](https://reader035.vdocument.in/reader035/viewer/2022062309/5681580c550346895dc57a14/html5/thumbnails/51.jpg)
CCG Parsing
Combinatory Categorial Grammar Fully (mono-)
lexicalized grammar
Categories encode argument sequences
Very closely related to the lambda calculus (more later)
Can have spurious ambiguities (why?)