natural language generation, non- metric methods ...smaskey/cs6998-0412/slides/... · weight the...
TRANSCRIPT
![Page 1: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/1.jpg)
1
Natural Language Generation, Non-
Metric Methods, Probabilistic
Context Free Grammar, Parsing
Algorithms, NLP Tools
Sameer Maskey
Week 4, Sept 26, 2012
*animation slides on parsing obtained from Prof Raymond Mooney
![Page 2: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/2.jpg)
2
Topics for Today
� Non-metric Methods
� Probabilistic Context Free Grammar
� Parsing Algorithms
� CKY Parsing
� Writing your Grammar and Parser
� Weighted Finite State Transducers
� Using WFST in NLP and Speech processing tasks
![Page 3: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/3.jpg)
3
Announcement
� Proposal Due tonight (11:59pm) – Graded� 5% of the project grade
� Email me the proposal with the title
� “Project Proposal : Statistical NLP for the Web”
� Use the following format if appropriate
1. Abstract/Summary2. Introduction and Related Work3. Data4. NLP/ML Algorithms5. System Description (end-to-end)5. Conclusion
� Homework 1 due October 4th (11:59pm) Thursday� Start early
![Page 4: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/4.jpg)
4
K-Means in Words
� Parameters to estimate for K classes
� Let us assume we can model this data
� with mixture of two Gaussians
� Start with 2 Gaussians (initialize mu values)
� Compute distance of each point to the mu of 2 Gaussians and assign it to the closest Gaussian (class label (Ck))
� Use the assigned points to recompute mu for 2 Gaussians
Hockey
Baseball
![Page 5: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/5.jpg)
5
� Optimize for each n separately by choosing for k that
gives minimum
� Assign each data point to the cluster that is the closest
� Hard decision to cluster assignment
rnk
||xn − rnk||2
rnk = 1 if k = argminj ||xn − µj ||2
= 0 otherwise
rnkµk
Step 1� Minimize J with respect to
� Keep fixed
![Page 6: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/6.jpg)
6
� J is quadratic in . Minimize by setting derivative w.rt. to
zero
� Take all the points assigned to cluster K and re-estimate the
mean for cluster K
� Minimize J with respect to
� Keep fixedrnk
µkStep 2
µk µk
µk =
∑n rnkxn∑n rnk
![Page 7: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/7.jpg)
7
Explaining Expectation Maximization
� EM is like fuzzy K-means
� Parameters to estimate for K classes
� Let us assume we can model this data
with mixture of two Gaussians (K=2)
� Start with 2 Gaussians (initialize mu and sigma values)
� Compute distance of each point to the mu of 2 Gaussians and assign it a soft
class label (Ck)
� Use the assigned points to recompute mu and sigma for 2 Gaussians; but
weight the updates with soft labels
Hockey
Baseball
Expectation
Maximization
![Page 8: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/8.jpg)
8
Estimating Parameters
� E-Step
γ(znk) = E(znk|xn) = p(zk = 1|xn)
![Page 9: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/9.jpg)
9
Estimating Parameters
� M-step
� Iterate until convergence of log likelihood
π′k =Nk
N
log p(X|π, µ,∑) =
∑N
n=1 log (∑k
k=1N (x|µk,∑k))
µ′k =1Nk
∑N
n=1 γ(znk)xn
∑′k =
1Nk
∑N
n=1 γ(znk)(xn − µ′k)(xn − µ
′k)T
γ(znk) =πkN (xn|µk,
∑k)
∑Kj=1
πjN (xn|µj ,∑
j)
where Nk =∑N
n=1 γ(znk)
![Page 10: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/10.jpg)
10
Hierarchical Clustering Algorithm
� Step 1
� Assign each data point to its own cluster
� Step 2
� Compute similarity between clusters
� Step 3
� Merge two most similar cluster to form one less cluster
![Page 11: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/11.jpg)
11
Human-Machine Dialog
![Page 12: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/12.jpg)
12
Human-Machine DialogMachine may need to
generate text tocommunicate with Humans
![Page 13: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/13.jpg)
13
Natural Language Generation
� For machines to communicate with humans they need to know how to generate valid meaningful text
� Validity
� Morphologically
� Syntactically
� Semantically
� How about discourse?
![Page 14: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/14.jpg)
14
Natural Language Generation (NLG)
� Text generation used in various NLP tasks
� Summarization
� Machine translation
� Question Answering
� Dialog System
� Based on data and tasks, generation methods vary widely
� Text to Text Generation
� Database to Text Generation
� Speech to Text Generation
� Concept to Text Generation
� Text Generators? :
� http://www.elsewhere.org/pomo/
� http://pdos.csail.mit.edu/scigen/
![Page 15: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/15.jpg)
15
NLG
� McDonald (1987)
� Natural language generation is the process of deliberately
constructing a natural language text in order to meet
specified communicative goals.
� Dale (1997):
� Natural language generation is the subfield of artificial
intelligence and computational linguistics that is concerned
with the construction of computer systems that can produce
understandable texts in... human languages from some
underlying non-linguistic representation of information.
![Page 16: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/16.jpg)
16
Dialog System
Figure Source :Natural Language Generation in Dialog System [Rambow, et. al]
![Page 17: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/17.jpg)
17
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
![Page 18: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/18.jpg)
18
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
Content/Text Planner : Break down of overall high-level communicative goal into individual atomic goals
![Page 19: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/19.jpg)
19
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
Sentence Planner : Finding abstract linguistic representations that will help in relating each atomic communicative goals
![Page 20: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/20.jpg)
20
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
Realizer : Convert abstract linguistic representation tosurface form (text) – syntax, word order, agreement
(use language dependent grammar to produce valid surface form)Validity : morphologically, syntactically, semantically
![Page 21: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/21.jpg)
21
Text to Words : Bag of Words Model
� We produced multinomial vectors from a given piece of text
In order to understand a book …
read
meaningfull
itsoriginal
passageunderstand
the
until
you
![Page 22: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/22.jpg)
22
Words to Text
read
meaningfull
itsoriginal
passageunderstand
the
until
you
?
![Page 23: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/23.jpg)
23
Human Sentence Generation Performance
read
meaningfull
itsoriginal
passageunderstand
the
until
you
read meaningfullitsoriginal passage understandthe until you
![Page 24: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/24.jpg)
24
Human Performance
� Humans backtrack to disambiguate
� Many points of disambiguation
� Frequency matters
� Generated sentence by human
� Syntactically sound
� Semantically sound
� Do we start think of semantics first or syntax first?
read the original passage until you understand its full meaning
![Page 25: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/25.jpg)
25
Syntax and Semantics
(S (VP read
(NP the original passage)
(SBAR until
(S (NP you)
(VP understand
(NP its full meaning))))))
Syntax
-study of combinatorics of basic units of language
-how to combine words together?
Semantics
-study of meaning
-what do grouped words mean together?
read the original passage until you understand its full meaning
Meaning of grouped words
“read the original passage”
vs
“read the passage”
![Page 26: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/26.jpg)
26
Putting Words Together
� Combinatorial problem created when we try to put words together is huge
� Try producing all possible word combination of our previous sentence of length 10� Total combinations : 10^10 = 1 billion sentences� Sent1 : “read the the the … passage” is unlikely� Sent2 : “read the passage …” is more likely
� How can we come up with scores that are higher for Sent1 than Sent2� Don’t allow to group words like “the the the”
� Make such construction invalid
� Invalidity as defined by a set of rules that govern the language� Such rules define the grammar� For mathematical modeling easier to use “Context Free
Grammar”
![Page 27: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/27.jpg)
27
Non-Metric Methods
� Can we use previously learned ML algorithms for NLG� Yes
� Why is it difficult?� Combination problem� Notion of metric or distance is limited
� What is the mean of distribution of all possible sentence combination of length 10?
� Distance between � “What is Apple?” - “What is Vodafone?”
� “What is Apple?” – “What is Orange?”
� “What is Apple?” – “What is a fruit?”
� “What is Apple?” – “What is a rock?”
� “What is Apple?” – “What is the?” (?)
� No clear notion of similarity
� From vector of real numbers to list of attributes
![Page 28: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/28.jpg)
28
Non-Metric Methods
� Decision Trees
� Rule Based Methods
� Grammar based Methods
� Finite State Transducers
![Page 29: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/29.jpg)
29
Grammar Based Methods
� Regular Grammar
� Can be represented by Finite State Automata
� Context Free Grammar
� Allows only 1 symbol on LHS
� Can apply the rule without caring about what is the context (left and right symbols)
� Well suited to describe recursive syntax
� Context Sensitive Grammar
� Allows more than 1 symbol on LHS
� aZb � aKb can only be applied to non-terminal Z only in the context of a and b
� Unrestricted Grammar
� E.g. natural language
![Page 30: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/30.jpg)
30
Context Free Grammars (CFG)
� N a set of non-terminal symbols (or variables)
� Σ a set of terminal symbols (disjoint from N)
� R a set of productions or rules of the form A→β, where A is a non-terminal and β is a string of symbols from (Σ∪ N)*
� S, a designated non-terminal called the start
symbol
*animation starting this one is provided by Prof. Raymond Mooney
![Page 31: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/31.jpg)
31
Simple CFG for ATIS English
S → NP VP
S → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → VP PP
PP → Prep NP
Det→ the | a | that | this
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | he | she | me
Proper-Noun → Houston | NWA
Aux → does
Prep → from | to | on | near | through
Grammar Lexicon
![Page 32: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/32.jpg)
32
Sentence Generation
� Sentences are generated by recursively rewriting the start
symbol using the productions until only terminals symbols
remain.
S
VP
Verb NP
Det Nominal
Nominal PP
book
Prep NP
through
Houston
Proper-Noun
the
flight
Noun
Derivation
or
Parse Tree
![Page 33: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/33.jpg)
33
Parsing
� Given a string of terminals and a CFG, determine if the string can be generated by the CFG.
� Also return a parse tree for the string
� Also return all possible parse trees for the string
� Must search space of derivations for one that derives the given string.
� Top-Down Parsing: Start searching space of derivations for the start symbol.
� Bottom-up Parsing: Start search space of reverse deivationsfrom the terminal symbols in the string.
![Page 34: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/34.jpg)
34
Parsing Example
S
VP
Verb NP
book Det Nominal
that Noun
flight
book that flight
![Page 35: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/35.jpg)
35
Top Down Parsing
S
VP
Verb NP
![Page 36: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/36.jpg)
36
Top Down Parsing
S
VP
Verb NP
book
![Page 37: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/37.jpg)
37
Top Down Parsing
S
VP
Verb NP
book Pronoun
![Page 38: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/38.jpg)
38
Top Down Parsing
S
VP
Verb NP
book Det Nominal
![Page 39: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/39.jpg)
39
Top Down Parsing
S
VP
Verb NP
book Det Nominal
that
![Page 40: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/40.jpg)
40
Top Down Parsing
S
VP
Verb NP
book Det Nominal
that Noun
![Page 41: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/41.jpg)
41
Top Down Parsing
S
VP
Verb NP
book Det Nominal
that Noun
flight
![Page 42: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/42.jpg)
42
Simple CFG for ATIS English
S → NP VP
S → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → VP PP
PP → Prep NP
Det→ the | a | that | this
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | he | she | me
Proper-Noun → Houston | NWA
Aux → does
Prep → from | to | on | near | through
Grammar Lexicon
![Page 43: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/43.jpg)
43
Bottom Up Parsing
43
book that flight
![Page 44: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/44.jpg)
44
Det
Bottom Up Parsing
44
book that
Verb
VP
S
NP
Nominal
flight
Noun
![Page 45: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/45.jpg)
45
Det
Bottom Up Parsing
45
book that
Verb
VP
S
XNP
Nominal
flight
Noun
![Page 46: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/46.jpg)
46
Bottom Up Parsing
46
book that
Verb
VP
VP
PP
Det
NP
Nominal
flight
Noun
![Page 47: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/47.jpg)
47
Bottom Up Parsing
47
book that
Verb
VP
VP
PP
Det
NP
Nominal
flight
Noun
X
![Page 48: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/48.jpg)
48
Bottom Up Parsing
48
book that
Verb
VP
Det
NP
Nominal
flight
Noun
NP
![Page 49: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/49.jpg)
49
Bottom Up Parsing
49
book that
Verb
VP
Det
NP
Nominal
flight
Noun
![Page 50: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/50.jpg)
50
Bottom Up Parsing
50
book that
Verb
VP
Det
NP
Nominal
flight
Noun
S
![Page 51: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/51.jpg)
51
Top Down vs. Bottom Up
� Top down never explores options that will not lead to a full
parse, but can explore many options that never connect to the
actual sentence.
� Bottom up never explores options that do not connect to the
actual sentence but can explore options that can never lead to
a full parse.
� Relative amounts of wasted search depend on how much the
grammar branches in each direction.
51
![Page 52: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/52.jpg)
52
Dynamic Programming Parsing
� To avoid extensive repeated work, must cache intermediate results, i.e. completed phrases.
� Caching critical to obtaining a polynomial time parsing (recognition) algorithm for CFGs.
� Dynamic programming algorithms based on both
top-down and bottom-up search can achieve O(n3) recognition time where n is the length of the input
string.
52
![Page 53: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/53.jpg)
53
Dynamic Programming Parsing
Methods� CKY (Cocke-Kasami-Younger) algorithm based on
bottom-up parsing and requires first normalizing the grammar.
� Earley parser is based on top-down parsing and does not require normalizing grammar but is more
complex.
� More generally, chart parsers retain completed
phrases in a chart and can combine top-down and bottom-up search.
53
![Page 54: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/54.jpg)
54
CKY
� First grammar must be converted to Chomsky normal form (CNF) in which productions must have either exactly 2 non-terminal symbols on the RHS or
1 terminal symbol (lexicon rules).
� Parse bottom-up storing phrases formed from all
substrings in a triangular table (chart).
54
![Page 55: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/55.jpg)
55
ATIS English Grammar Conversion
S → NP VP
S → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → VP PP
PP → Prep NP
Original Grammar Chomsky Normal Form
S → NP VP
S → X1 VP
X1 → Aux NP
S → book | include | prefer
S → Verb NP
S → VP PP
NP → I | he | she | me
NP → Houston | NWA
NP → Det Nominal
Nominal → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → book | include | prefer
VP → Verb NP
VP → VP PP
PP → Prep NP
![Page 56: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/56.jpg)
56
CKY Parser
56
Book the flight through Houston
i=
0
1
2
3
4
j= 1 2 3 4 5
Cell[i,j]
contains all
constituents
(non-terminals)
covering words
i +1 through j
![Page 57: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/57.jpg)
57
CKY Parser
57
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
![Page 58: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/58.jpg)
58
CKY Parser
58
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
![Page 59: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/59.jpg)
59
CKY Parser
59
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
![Page 60: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/60.jpg)
60
CKY Parser
60
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
![Page 61: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/61.jpg)
61
CKY Parser
61
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
![Page 62: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/62.jpg)
62
CKY Parser
62
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
![Page 63: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/63.jpg)
63
CKY Parser
63
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
![Page 64: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/64.jpg)
64
CKY Parser
64
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
![Page 65: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/65.jpg)
65
CKY Parser
65
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
VP
![Page 66: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/66.jpg)
66
CKY Parser
66
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
S
VP
![Page 67: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/67.jpg)
67
CKY Parser
67
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
VPS
VP
![Page 68: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/68.jpg)
68
CKY Parser
68
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
VPS
VP
S
![Page 69: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/69.jpg)
69
CKY Parser
69
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
VPS
VP
SParse
Tree
#1
![Page 70: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/70.jpg)
70
CKY Parser
70
Book the flight through Houston
S, VP, Verb,
Nominal,
Noun
Det
Nominal,
Noun
None
NP
VP
S
Prep
None
None
None
NP
ProperNoun
PP
Nominal
NP
VPS
VP
SParse
Tree
#2
![Page 71: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/71.jpg)
71
Complexity of CKY (recognition)
� There are (n(n+1)/2) = O(n2) cells
� Filling each cell requires looking at every possible
split point between the two non-terminals needed to introduce a new phrase.
� There are O(n) possible split points.
� Total time complexity is O(n3)
71
![Page 72: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/72.jpg)
7272
Probabilistic Context Free Grammar
(PCFG)
� A PCFG is a probabilistic version of a CFG where each production has a probability.
� Probabilities of all productions rewriting a given non-terminal must add to 1, defining a distribution for each non-terminal.
� String generation is now probabilistic where production probabilities are used to non-
deterministically select a production for rewriting a given non-terminal.
![Page 73: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/73.jpg)
73
Simple PCFG for ATIS English
S → NP VP
S → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → VP PP
PP → Prep NP
Grammar
0.8
0.1
0.1
0.2
0.2
0.6
0.3
0.2
0.5
0.2
0.5
0.3
1.0
Prob
+
+
+
+
1.0
1.0
1.0
1.0
Det → the | a | that | this
0.6 0.2 0.1 0.1
Noun → book | flight | meal | money
0.1 0.5 0.2 0.2
Verb → book | include | prefer
0.5 0.2 0.3
Pronoun → I | he | she | me
0.5 0.1 0.1 0.3
Proper-Noun → Houston | NWA
0.8 0.2
Aux → does
1.0
Prep → from | to | on | near | through
0.25 0.25 0.1 0.2 0.2
Lexicon
![Page 74: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/74.jpg)
7474
Sentence Probability
� Assume productions for each node are chosen independently.
� Probability of derivation is the product of the probabilities of its productions.
P(D1) = 0.1 x 0.5 x 0.5 x 0.6 x 0.6 x
0.5 x 0.3 x 1.0 x 0.2 x 0.2 x
0.5 x 0.8= 0.0000216
D1S
VP
Verb NP
Det Nominal
Nominal PP
book
Prep NP
through
Houston
Proper-Noun
the
flight
Noun
0.5
0.50.6
0.6 0.5
1.0
0.2
0.3
0.5 0.2
0.8
0.1
![Page 75: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/75.jpg)
75
Syntactic Disambiguation
� Resolve ambiguity by picking most probable parse tree.
7575
D2
VP
Verb NP
Det Nominalbook
Prep NP
through
Houston
Proper-Noun
the
flight
Noun
0.5
0.50.6
0.61.0
0.20.3
0.5 0.2
0.8
S
VP
0.1
PP
0.3
P(D2) = 0.1 x 0.3 x 0.5 x 0.6 x 0.5 x
0.6 x 0.3 x 1.0 x 0.5 x 0.2 x
0.2 x 0.8= 0.00001296
![Page 76: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/76.jpg)
76
Sentence Probability
� Probability of a sentence is the sum of the probabilities of all of its derivations.
76
P(“book the flight through Houston”) =
P(D1) + P(D2) = 0.0000216 + 0.00001296
= 0.00003456
![Page 77: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/77.jpg)
7777
Three Useful PCFG Tasks
� Observation likelihood: To classify and order sentences.
� Most likely derivation: To determine the most likely parse tree for a sentence.
� Maximum likelihood training: To train a PCFG to fit empirical training data.
![Page 78: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/78.jpg)
78
Probabilistic CKY
� CKY can be modified for PCFG parsing by including in each cell a probability for each non-terminal.
� Cell[i,j] must retain the most probable derivation of each constituent (non-terminal) covering words i +1 through j together with its associated probability.
� When transforming the grammar to CNF, must set production probabilities to preserve the probability of derivations.
![Page 79: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/79.jpg)
79
Probabilistic Grammar Conversion
S → NP VPS → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det NominalNominal → Noun
Nominal → Nominal NounNominal → Nominal PPVP → Verb
VP → Verb NPVP → VP PPPP → Prep NP
Original Grammar Chomsky Normal Form
S → NP VPS → X1 VPX1 → Aux NPS → book | include | prefer
0.01 0.004 0.006S → Verb NPS → VP PPNP → I | he | she | me
0.1 0.02 0.02 0.06NP → Houston | NWA
0.16 .04NP → Det NominalNominal → book | flight | meal | money
0.03 0.15 0.06 0.06Nominal → Nominal NounNominal → Nominal PPVP → book | include | prefer
0.1 0.04 0.06VP → Verb NPVP → VP PPPP → Prep NP
0.80.1
0.1
0.2
0.2
0.60.3
0.20.50.2
0.50.31.0
0.80.11.0
0.050.03
0.6
0.20.5
0.50.31.0
![Page 80: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/80.jpg)
80
Probabilistic CKY Parser
80
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
![Page 81: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/81.jpg)
81
Probabilistic CKY Parser
81
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
![Page 82: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/82.jpg)
82
Probabilistic CKY Parser
82
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
![Page 83: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/83.jpg)
83
Probabilistic CKY Parser
83
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
![Page 84: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/84.jpg)
84
Probabilistic CKY Parser
84
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
![Page 85: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/85.jpg)
85
Probabilistic CKY Parser
85
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
![Page 86: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/86.jpg)
86
Probabilistic CKY Parser
86
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
![Page 87: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/87.jpg)
87
Probabilistic CKY Parser
87
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
S:.05*.5*
.000864
=.0000216
![Page 88: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/88.jpg)
88
Probabilistic CKY Parser
88
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
S:.0000216
S:.03*.0135*
.032
=.00001296
![Page 89: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/89.jpg)
89
Probabilistic CKY Parser
89
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
S:.0000216Pick most probable
parse, i.e. take max to
combine probabilities
of multiple derivations
of each constituent in
each cell.
![Page 90: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/90.jpg)
9090
PCFG: Observation Likelihood
� There is an analog to Forward algorithm for HMMs called the Inside algorithm for efficiently determining how likely a string is to be produced by a PCFG.
� Can use a PCFG as a language model to choose between alternative sentences for speech recognition or machine translation.
The dog big barked.
The big dog barked
O1
O2
![Page 91: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/91.jpg)
91
Inside Algorithm
� Use CKY probabilistic parsing algorithm but combine probabilities of multiple derivations of any constituent using addition instead of max.
91
![Page 92: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/92.jpg)
9292
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
S:.0000216
S:..00001296
Probabilistic CKY Parser
for Inside Computation
![Page 93: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/93.jpg)
9393
Book the flight through Houston
S :.01, VP:.1,
Verb:.5
Nominal:.03
Noun:.1
Det:.6
Nominal:.15
Noun:.5
None
NP:.6*.6*.15
=.054
VP:.5*.5*.054
=.0135
S:.05*.5*.054
=.00135
None
None
None
Prep:.2
NP:.16
PropNoun:.8
PP:1.0*.2*.16
=.032
Nominal:
.5*.15*.032
=.0024
NP:.6*.6*
.0024
=.000864
+.0000216
=.00003456
S: .00001296 Sum probabilities
of multiple derivations
of each constituent in
each cell.
Probabilistic CKY Parser
for Inside Computation
![Page 94: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/94.jpg)
9494
PCFG: Supervised Training
� If parse trees are provided for training sentences, a grammar
and its parameters can be can all be estimated directly from
counts accumulated from the tree-bank (with appropriate smoothing).
.
.
.
Tree Bank
Supervised
PCFG
Training
S →NP VP
S → VP
NP → Det A N
NP → NP PP
NP → PropN
A → ε
A → Adj A
PP → Prep NP
VP → V NP
VP → VP PP
0.9
0.1
0.5
0.3
0.2
0.6
0.4
1.0
0.7
0.3
English
S
NP VP
John V NP PP
put the dog in the pen
S
NP VP
John V NP PP
put the dog in the pen
![Page 95: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/95.jpg)
95
Estimating Production Probabilities
� Set of production rules can be taken directly from the set of rewrites in the treebank.
� Parameters can be directly estimated from frequency counts in the treebank.
95
)count(
)count(
)count(
)count()|(
α
βα
γα
βααβα
γ
→=
→
→=→∑
P
![Page 96: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/96.jpg)
9696
PCFG: Maximum Likelihood Training
� Given a set of sentences, induce a grammar that maximizes the probability that this data was generated from this grammar.
� Assume the number of non-terminals in the grammar is specified.
� Only need to have an unannotated set of sequences generated from the model. Does not need correct parse trees for these sentences. In this sense, it is unsupervised.
![Page 97: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/97.jpg)
9797
PCFG: Maximum Likelihood Training
John ate the apple
A dog bit Mary
Mary hit the dog
John gave Mary the cat.
.
.
.
Training Sentences
PCFG
Training
S →NP VP
S → VP
NP → Det A N
NP → NP PP
NP → PropN
A → ε
A → Adj A
PP → Prep NP
VP → V NP
VP → VP PP
0.9
0.1
0.5
0.3
0.2
0.6
0.4
1.0
0.7
0.3
English
![Page 98: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/98.jpg)
98
Write Your Own CFG
� Palindromes
� We want to construct a grammar that creates
palindromes
� aabbaa, aababaa
Non-Terminal = Z
Terminals = (a, b, e)
Start Symbol = SRules : Set R : S � Z
Z � aZaZ � bZb
Z � a
Z � bZ � e
We need G = (N, T, S, R)
S
aSa
aaSaa
aabSbaa
aababaa
![Page 99: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/99.jpg)
99
Write Your Own Probabilistic CFG
� Weighted Palindromes
� We want to construct a grammar that creates
palindromes that has more ‘a’ symbols
Non-Terminal = Z
Terminals = (a, b, e)
Start Symbol = SRules : Set R : S � Z
Z � aZaZ � bZb
Z � a
Z � bZ � e
We need G = (N, T, S, R)
S
aSa
aaSaa
aaaSaaa
aaaabaaaa
10.3
0.15
0.40.1
0.05Rule
Probabilities
![Page 100: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/100.jpg)
100
Write Your Own Probabilistic CFG# Rules for creating full sentences.
1 ROOT S .
1 ROOT S !
1 ROOT is it true that S ? # mixing terminals and nonterminals is ok.
# The basic grammar rules. Here's what the abbreviations stand for:
# S = sentence
# NP = noun phrase
# VP = verb phrase
# PP = prepositional phrase
# Det = determiner (sometimes called "article")
# Prep = preposition
# Adj = adjective
1 S NP VP
1 VP Verb NP
1 NP Det Noun
1 NP NP PP
1 PP Prep NP
1 Noun Adj Noun
Example from Jason Eisner and Noah Smith’s paper
![Page 101: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/101.jpg)
101
Write Your Own Probabilistic CFG# Vocabulary. Your program can see that "ate" is a terminal
# symbol because there exists no rule for rewriting it.
# Any symbol that can rewrite as a terminal (or a string of
# terminals, like "chief of staff") is called a "preterminal." Notice
# that a preterminal is a special kind of nonterminal.
1 Verb ate
1 Verb wanted
1 Verb kissed
1 Verb understood
1 Verb pickled
1 Det the
1 Det a
1 Det every
1 Noun president
1 Noun sandwich
1 Noun pickle
1 Noun chief of staff
1 Noun floor
![Page 102: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/102.jpg)
102
Write Your Sentence Generator
� Possible sentence that can be generated?
� the president ate every sandwich !
� the president understood the chief of staff .
� Can these sentences be generated?
� president understood the chief of staff .
� the chief of staff pickled the chief of staff !
� Make the grammar generate more questions?
![Page 103: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/103.jpg)
103
Human-Machine Dialog
![Page 104: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/104.jpg)
104
Simple Grammar for Simple Dialog
System
� What kind of answers should the Dialog system be able to generate?
� Ok. I will send your message.
� I have set an alarm for 4 a.m.
� Here’s the contact information.
� Your flight is at 6 p.m.
� Grammar design may be governed by the domain
![Page 105: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/105.jpg)
105
Writing Grammar for Simple Dialog
System.
ROOT S .
S NP VP
VP Verb NP
NP Det Noun
NP NP PP
PP Prep NP
Noun Adj Noun
Verb send
Verb set
Verb contact
Noun message
Noun alarm
Noun flight
.
.
.
![Page 106: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/106.jpg)
106
Rule Probabilities
� Count the rewrite rules from Penn
Treebank corpus
)count(
)count(
)count(
)count()|(
α
βα
γα
βααβα
γ
→=
→
→=→∑
P
.
ROOT S .
S NP VP
VP Verb NP
NP Det Noun
NP NP PP
PP Prep NP
Noun Adj Noun
Verb send
Verb set
Verb contact
Noun message
Noun alarm
Noun flight
.
.
.
![Page 107: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/107.jpg)
107
Rule Probabilities
� Count the rewrite rules from Penn
Treebank corpus
)count(
)count(
)count(
)count()|(
α
βα
γα
βααβα
γ
→=
→
→=→∑
P
.
ROOT S .
S NP VP
VP Verb NP
NP Det Noun
NP NP PP
PP Prep NP
Noun Adj Noun
Verb send
Verb set
Verb contact
Noun message
Noun alarm
Noun flight
.
.
.
� Verb � send [25]
� Verb � set [12]
� Verb � contact [13]
� 25/(25+12+13) for Verb�Send
=0.5
0.5
Rewrite count
![Page 108: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/108.jpg)
108
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
Realizer : Convert abstract linguistic representation tosurface form (text) – syntax, word order, agreement
(use language dependent grammar to produce valid surface form)Validity : morphologically, syntactically, semantically
![Page 109: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/109.jpg)
109
NLG Components
Content/Text
Planner
Sentence
PlannerRealizer
Sentence Planner : Finding abstract linguistic representations that will help in relating each atomic communicative goals
![Page 110: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/110.jpg)
110
Similarity
� While clustering documents we are essentially finding ‘similar’ documents
� How we compute similarity makes a difference in the performance of clustering algorithm
� Some similarity metrics
� Euclidean distance
� Cross Entropy
� Cosine Similarity
� Which similarity metric to use?
![Page 111: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/111.jpg)
111
Similarity for Words
� Edit distance
� Insertion, deletion, substitution
� Dynamic programming algorithm
� Longest Common Subsequence
� Bigram overlap of characters
� Similarity based on meaning
� WordNet synonyms
� Similarity based on collocation
![Page 112: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/112.jpg)
112
Similarity of Text : Surface, Syntax and
Semantics� Cosine Similarity
� Binary Vectors
� Multinomial Vectors
� Edit distance
� Insertion, deletion, substitution
� Semantic similarity
� Look beyond surface forms
� WordNet, semantic classes
� Syntactic similarity
� Syntactic structure
� Many ways to look at similarity and choice of the metric is important for the type of clustering algorithm we are using
![Page 113: Natural Language Generation, Non- Metric Methods ...smaskey/CS6998-0412/slides/... · weight the updates with soft labels Hockey Baseball Expectation Maximization. 8 ... communicative](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7a529afeedad09370707c2/html5/thumbnails/113.jpg)
113
NLP/ML Tools
� Weka
� Stanford NLP Tools
� Parsers, taggers, chunkers, NE recognizer
� Ratnaparkhi’s NE Tagger
� NLTK
� OpenNLP