lecture 10 nltk pos tagging part 3 topics taggers rule based taggers probabilistic taggers...

Lecture 10 NLTK POS Tagging Part 3

Lecture 10 NLTK POS Tagging Part 3

Topics Topics Taggers Rule Based Taggers Probabilistic Taggers Transformation Based Taggers - Brill Supervised learning

Readings: Chapter 5.4-?Readings: Chapter 5.4-?

February 18, 2013

CSCE 771 Natural Language Processing

– 2 – CSCE 771 Spring 2011

OverviewOverviewLast TimeLast Time

Overview of POS Tags

TodayToday Part of Speech Tagging Parts of Speech Rule Based taggers Stochastic taggers Transformational taggers

ReadingsReadings Chapter 5.4-5.?

– 3 – CSCE 771 Spring 2011

brown_lrnd_tagged = brown_lrnd_tagged = brown.tagged_words(categories='learned', brown.tagged_words(categories='learned', simplify_tags=True)simplify_tags=True)

tags = [b[1] for (a, b) in tags = [b[1] for (a, b) in nltk.ibigrams(brown_lrnd_tagged) if a[0] == 'often']nltk.ibigrams(brown_lrnd_tagged) if a[0] == 'often']

fd = nltk.FreqDist(tags)fd = nltk.FreqDist(tags)

print fd.tabulate()print fd.tabulate()

VN V VD ADJ DET ADV P , CNJ . TO VBZ VG WHVN V VD ADJ DET ADV P , CNJ . TO VBZ VG WH

15 12 8 5 5 4 4 3 3 1 1 1 1 115 12 8 5 5 4 4 3 3 1 1 1 1 1

– 4 – CSCE 771 Spring 2011

highly ambiguous wordshighly ambiguous words

>>> brown_news_tagged = >>> brown_news_tagged = brown.tagged_words(categories='news', brown.tagged_words(categories='news', simplify_tags=True)simplify_tags=True)

>>> data = nltk.ConditionalFreqDist((word.lower(), tag) ... >>> data = nltk.ConditionalFreqDist((word.lower(), tag) ... for (word, tag) in brown_news_tagged) for (word, tag) in brown_news_tagged)

>>> for word in data.conditions(): >>> for word in data.conditions():

... if len(data[word]) > 3: ... if len(data[word]) > 3:

... tags = data[word].keys() ... tags = data[word].keys()

... print word, ' '.join(tags) ... print word, ' '.join(tags)

... ...

best ADJ ADV NP V best ADJ ADV NP V

better ADJ ADV V DETbetter ADJ ADV V DET

……..

– 5 – CSCE 771 Spring 2011

Tag PackageTag Package

http://nltk.org/api/nltk.tag.html#module-nltk.taghttp://nltk.org/api/nltk.tag.html#module-nltk.tag

– 6 – CSCE 771 Spring 2011

Python's Dictionary Methods: Python's Dictionary Methods:

..

– 7 – CSCE 771 Spring 2011

5.4   Automatic Tagging5.4   Automatic Tagging

Training setTraining set

Test setTest set

### setup### setup

import nltk, re, pprintimport nltk, re, pprint

from nltk.corpus import brownfrom nltk.corpus import brown

brown_tagged_sents = brown_tagged_sents = brown.tagged_sents(categories='news')brown.tagged_sents(categories='news')

brown_sents = brown.sents(categories='news')brown_sents = brown.sents(categories='news')

– 8 – CSCE 771 Spring 2011

Default.tagger NNDefault.tagger NNtags = [tag for (word, tag) in tags = [tag for (word, tag) in

brown.tagged_words(categories='news')]brown.tagged_words(categories='news')]

print nltk.FreqDist(tags).max()print nltk.FreqDist(tags).max()

raw = 'I do not like green eggs and ham, I …Sam I am!'raw = 'I do not like green eggs and ham, I …Sam I am!'

tokens = nltk.word_tokenize(raw)tokens = nltk.word_tokenize(raw)

default_tagger = nltk.DefaultTagger('NN')default_tagger = nltk.DefaultTagger('NN')

print default_tagger.tag(tokens)print default_tagger.tag(tokens)

[('I', 'NN'), ('do', 'NN'), ('not', 'NN'), ('like', 'NN'), …[('I', 'NN'), ('do', 'NN'), ('not', 'NN'), ('like', 'NN'), …

print default_tagger.evaluate(brown_tagged_sents)print default_tagger.evaluate(brown_tagged_sents)

0.1308948425720.130894842572

– 9 – CSCE 771 Spring 2011

Tagger2: regexp_taggerTagger2: regexp_tagger

patterns = [patterns = [

(r'.*ing$', 'VBG'), (r'.*ing$', 'VBG'), # gerunds# gerunds

(r'.*ed$', 'VBD'), (r'.*ed$', 'VBD'), # simple past# simple past

(r'.*es$', 'VBZ'), (r'.*es$', 'VBZ'), # 3rd singular present# 3rd singular present

(r'.*ould$', 'MD'), (r'.*ould$', 'MD'), # modals# modals

(r'.*\'s$', 'NN$'), (r'.*\'s$', 'NN$'), # possessive nouns# possessive nouns

(r'.*s$', 'NNS'), (r'.*s$', 'NNS'), # plural nouns# plural nouns

(r'^-?[0-9]+(.[0-9]+)?$', 'CD'), # cardinal numbers(r'^-?[0-9]+(.[0-9]+)?$', 'CD'), # cardinal numbers

(r'.*', 'NN') (r'.*', 'NN') # nouns (default)# nouns (default)

]]

regexp_tagger = nltk.RegexpTagger(patterns)regexp_tagger = nltk.RegexpTagger(patterns)

– 10 – CSCE 771 Spring 2011

Evaluate regexp_taggerEvaluate regexp_tagger

regexp_tagger = nltk.RegexpTagger(patterns)regexp_tagger = nltk.RegexpTagger(patterns)

print regexp_tagger.tag(brown_sents[3])print regexp_tagger.tag(brown_sents[3])

[('``', 'NN'), ('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), …[('``', 'NN'), ('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), …

print regexp_tagger.evaluate(brown_tagged_sents)print regexp_tagger.evaluate(brown_tagged_sents)

0.2032639178950.203263917895

– 11 – CSCE 771 Spring 2011

Unigram Tagger: 100 Most Freq tagUnigram Tagger: 100 Most Freq tag

fd = nltk.FreqDist(brown.words(categories='news'))fd = nltk.FreqDist(brown.words(categories='news'))cfd = cfd =

nltk.ConditionalFreqDist(brown.tagged_words(categories='news'))nltk.ConditionalFreqDist(brown.tagged_words(categories='news'))

most_freq_words = fd.keys()[:100]most_freq_words = fd.keys()[:100]

likely_tags = dict((word, cfd[word].max()) for word in likely_tags = dict((word, cfd[word].max()) for word in most_freq_words)most_freq_words)

baseline_tagger = nltk.UnigramTagger(model=likely_tags)baseline_tagger = nltk.UnigramTagger(model=likely_tags)

print baseline_tagger.evaluate(brown_tagged_sents)print baseline_tagger.evaluate(brown_tagged_sents)

0.4557849513690.455784951369

– 12 – CSCE 771 Spring 2011

Likely_tags; Backoff to NNLikely_tags; Backoff to NN

sent = brown.sents(categories='news')[3]sent = brown.sents(categories='news')[3]

baseline_tagger.tag(sent)baseline_tagger.tag(sent)('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), ('handful', 'NN'), ('of', 'NN'),('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), ('handful', 'NN'), ('of', 'NN'),

baseline_tagger = baseline_tagger = nltk.UnigramTagger(model=likely_tags,nltk.UnigramTagger(model=likely_tags,

backoff=nltk.DefaultTagger('NN'))backoff=nltk.DefaultTagger('NN'))

print baseline_tagger.tag(sent)print baseline_tagger.tag(sent)'Only', 'NN'), ('a', 'AT'), ('relative', 'NN'), ('handful', 'NN'), ('of', 'IN'),'Only', 'NN'), ('a', 'AT'), ('relative', 'NN'), ('handful', 'NN'), ('of', 'IN'),

print baseline_tagger.evaluate(brown_tagged_sents)print baseline_tagger.evaluate(brown_tagged_sents)

0.5817769556660.581776955666

– 13 – CSCE 771 Spring 2011

Performance of Easy TaggersPerformance of Easy Taggers

.. Tagger Performance Comment

NN tagger 0.13

Regexp tagger 0.20

100 Most Freq tag 0.46

Likely_tags; Backoff to NN

0.58

– 14 – CSCE 771 Spring 2011

def performance(cfd, wordlist):def performance(cfd, wordlist):

lt = dict((word, cfd[word].max()) for word in wordlist)lt = dict((word, cfd[word].max()) for word in wordlist)

baseline_tagger = nltk.UnigramTagger(model=lt, baseline_tagger = nltk.UnigramTagger(model=lt, backoff=nltk.DefaultTagger('NN'))backoff=nltk.DefaultTagger('NN'))

return return baseline_tagger.evaluate(brown.tagged_sents(categbaseline_tagger.evaluate(brown.tagged_sents(categories='news'))ories='news'))

– 15 – CSCE 771 Spring 2011

DisplayDisplaydef display(): def display():

import pylabimport pylab

words_by_freq = words_by_freq = list(nltk.FreqDist(brown.words(categories='news')))list(nltk.FreqDist(brown.words(categories='news')))

cfd = cfd = nltk.ConditionalFreqDist(brown.tagged_words(categnltk.ConditionalFreqDist(brown.tagged_words(categories='news'))ories='news'))

sizes = 2 ** pylab.arange(15)sizes = 2 ** pylab.arange(15)

perfs = [performance(cfd, words_by_freq[:size]) for perfs = [performance(cfd, words_by_freq[:size]) for size in sizes]size in sizes]

pylab.plot(sizes, perfs, '-bo')pylab.plot(sizes, perfs, '-bo')

pylab.title('Lookup Tagger Perf. vs Model Size')pylab.title('Lookup Tagger Perf. vs Model Size')

pylab.xlabel('Model Size')pylab.xlabel('Model Size')

pylab.ylabel('Performance')pylab.ylabel('Performance')

pylab.show()pylab.show()

– 16 – CSCE 771 Spring 2011

Error !?Error !?

Traceback (most recent call last):Traceback (most recent call last):

File File "C:/Users/mmm/Documents/Courses/771/Python771/"C:/Users/mmm/Documents/Courses/771/Python771/ch05.4.py", line 70, in <module>ch05.4.py", line 70, in <module>

import pylabimport pylab

ImportError: No module named pylabImportError: No module named pylab

google (download pylab) google (download pylab) scipy ?? scipy ??

– 17 – CSCE 771 Spring 2011

5.5 N-gram Tagging5.5 N-gram Tagging

from nltk.corpus import brownfrom nltk.corpus import brown

brown_tagged_sents = brown_tagged_sents = brown.tagged_sents(categories='news')brown.tagged_sents(categories='news')

brown_sents = brown.sents(categories='news')brown_sents = brown.sents(categories='news')

unigram_tagger = unigram_tagger = nltk.UnigramTagger(brown_tagged_sents)nltk.UnigramTagger(brown_tagged_sents)

print unigram_tagger.tag(brown_sents[2007])print unigram_tagger.tag(brown_sents[2007])

[('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'), ('are', [('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'), ('are', 'BER'), ('of', 'IN'),'BER'), ('of', 'IN'),

print unigram_tagger.evaluate(brown_tagged_sents)print unigram_tagger.evaluate(brown_tagged_sents)

0.9349006503970.934900650397

– 18 – CSCE 771 Spring 2011

Dividing into Training/Test SetsDividing into Training/Test Sets

size = int(len(brown_tagged_sents) * 0.9)size = int(len(brown_tagged_sents) * 0.9)

print sizeprint size

41604160

train_sents = brown_tagged_sents[:size]train_sents = brown_tagged_sents[:size]

test_sents = brown_tagged_sents[size:]test_sents = brown_tagged_sents[size:]

unigram_tagger = nltk.UnigramTagger(train_sents)unigram_tagger = nltk.UnigramTagger(train_sents)

print unigram_tagger.evaluate(test_sents)print unigram_tagger.evaluate(test_sents)

0.8110236220470.811023622047

– 19 – CSCE 771 Spring 2011

bigram_tagger 1rst try -- bigram_tagger 1rst try --

bigram_tagger = nltk.BigramTagger(train_sents)bigram_tagger = nltk.BigramTagger(train_sents)

print "bigram_tagger.tag-2007", print "bigram_tagger.tag-2007", bigram_tagger.tag(brown_sents[2007])bigram_tagger.tag(brown_sents[2007])

bigram_tagger.tag-2007 [('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), bigram_tagger.tag-2007 [('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'), ('are', 'BER')('apartments', 'NNS'), ('are', 'BER')

unseen_sent = brown_sents[4203]unseen_sent = brown_sents[4203]

print "bigram_tagger.tag-4203", print "bigram_tagger.tag-4203", bigram_tagger.tag(unseen_sent)bigram_tagger.tag(unseen_sent)

bigram_tagger.tag-4203 [('The', 'AT'), ('is', 'BEZ'), ('13.5', bigram_tagger.tag-4203 [('The', 'AT'), ('is', 'BEZ'), ('13.5', None), ('million', None), (',', None), ('divided', None), None), ('million', None), (',', None), ('divided', None),

print bigram_tagger.evaluate(test_sents)print bigram_tagger.evaluate(test_sents)

0.102162862554 ---not too good0.102162862554 ---not too good

– 20 – CSCE 771 Spring 2011

Backoff bigramunigram NNBackoff bigramunigram NN

t0 = nltk.DefaultTagger('NN')t0 = nltk.DefaultTagger('NN')

t1 = nltk.UnigramTagger(train_sents, backoff=t0)t1 = nltk.UnigramTagger(train_sents, backoff=t0)

t2 = nltk.BigramTagger(train_sents, backoff=t1)t2 = nltk.BigramTagger(train_sents, backoff=t1)

print t2.evaluate(test_sents)print t2.evaluate(test_sents)

0.8447124489190.844712448919

– 21 – CSCE 771 Spring 2011

Your turn: tribiuni NNYour turn: tribiuni NN

– 22 – CSCE 771 Spring 2011

Tagging Unknown WordsTagging Unknown Words

Our approach to tagging unknown words still uses backoff to a Our approach to tagging unknown words still uses backoff to a regular-expression tagger or a default tagger. These are regular-expression tagger or a default tagger. These are unable to make use of context. Thus, if our tagger unable to make use of context. Thus, if our tagger encountered the word blog, not seen during training, it encountered the word blog, not seen during training, it would assign it the same tag, regardless of whether this would assign it the same tag, regardless of whether this word appeared in the context the blog or to blog. How can word appeared in the context the blog or to blog. How can we do better with these unknown words, or out-of-we do better with these unknown words, or out-of-vocabulary items?vocabulary items?

A useful method to tag unknown words based on context is to A useful method to tag unknown words based on context is to limit the vocabulary of a tagger to the most frequent n limit the vocabulary of a tagger to the most frequent n words, and to replace every other word with a special word words, and to replace every other word with a special word UNK using the method shown in UNK using the method shown in 5.3. During training, a . During training, a unigram tagger will probably learn that UNK is usually a unigram tagger will probably learn that UNK is usually a noun. However, the n-gram taggers will detect contexts in noun. However, the n-gram taggers will detect contexts in which it has some other tag. For example, if the preceding which it has some other tag. For example, if the preceding word is to (tagged TO), then UNK will probably be tagged as word is to (tagged TO), then UNK will probably be tagged as a verb.a verb.

– 23 – CSCE 771 Spring 2011

Serialization = pickleSerialization = pickle

SavingSaving

Object serializationObject serialization

from cPickle import dumpfrom cPickle import dump

output=open('t2.pkl', 'wb')output=open('t2.pkl', 'wb')

dump(t2, output, -1)dump(t2, output, -1)

output.close()output.close()

LoadingLoading

from cPickle import loadfrom cPickle import load

input = open('t2.pkl', 'rb')input = open('t2.pkl', 'rb')

tagger = load(input)tagger = load(input)

input.close()input.close()

– 24 – CSCE 771 Spring 2011

Performance LimitationsPerformance Limitations

– 25 – CSCE 771 Spring 2011

text = """The board's action shows what free enterprisetext = """The board's action shows what free enterprise

is up against in our complex maze of regulatory is up against in our complex maze of regulatory laws ."""laws ."""

tokens = text.split()tokens = text.split()

tagger.tag(tokens)tagger.tag(tokens)

cfd = nltk.ConditionalFreqDist(cfd = nltk.ConditionalFreqDist(

((x[1], y[1], z[0]), z[1])((x[1], y[1], z[0]), z[1])

for sent in brown_tagged_sentsfor sent in brown_tagged_sents

for x, y, z in nltk.trigrams(sent))for x, y, z in nltk.trigrams(sent))

ambiguous_contexts = [c for c in cfd.conditions() if ambiguous_contexts = [c for c in cfd.conditions() if len(cfd[c]) > 1]len(cfd[c]) > 1]

print sum(cfd[c].N() for c in ambiguous_contexts) / cfd.N()print sum(cfd[c].N() for c in ambiguous_contexts) / cfd.N()

– 26 – CSCE 771 Spring 2011

Confusion MatrixConfusion Matrix

test_tags = [tag for sent in test_tags = [tag for sent in brown.sents(categories='editorial')brown.sents(categories='editorial')

for (word, tag) in t2.tag(sent)]for (word, tag) in t2.tag(sent)]

gold_tags = [tag for (word, tag) in gold_tags = [tag for (word, tag) in brown.tagged_words(categories='editorial')]brown.tagged_words(categories='editorial')]

print nltk.ConfusionMatrix(gold_tags, test_tags)print nltk.ConfusionMatrix(gold_tags, test_tags)

overwhelming outputoverwhelming output

– 27 – CSCE 771 Spring 2011

– 28 – CSCE 771 Spring 2011

nltk.tag.brill.demo()nltk.tag.brill.demo()

Loading tagged data... Loading tagged data...

Done loading.Done loading.

Training unigram tagger:Training unigram tagger:

[accuracy: 0.832151][accuracy: 0.832151]

Training bigram tagger:Training bigram tagger:

[accuracy: 0.837930][accuracy: 0.837930]

Training Brill tagger on 1600 sentences...Training Brill tagger on 1600 sentences...

Finding initial useful rules...Finding initial useful rules...

Found 9757 useful rules.Found 9757 useful rules.

– 29 – CSCE 771 Spring 2011

S F r O | Score = Fixed - BrokenS F r O | Score = Fixed - Broken

c i o t | R Fixed = num tags changed incorrect -> correctc i o t | R Fixed = num tags changed incorrect -> correct

o x k h | u Broken = num tags changed correct -> incorrecto x k h | u Broken = num tags changed correct -> incorrect

r e e e | l Other = num tags changed incorrect -> incorrectr e e e | l Other = num tags changed incorrect -> incorrect

e d n r | ee d n r | e

------------------+-------------------------------------------------------------------------+-------------------------------------------------------

11 15 4 0 | WDT -> IN if the tag of words i+1...i+2 is 'DT'11 15 4 0 | WDT -> IN if the tag of words i+1...i+2 is 'DT'

10 12 2 0 | IN -> RB if the text of the following word is10 12 2 0 | IN -> RB if the text of the following word is

| 'well'| 'well'

9 9 0 0 | WDT -> IN if the tag of the preceding word is9 9 0 0 | WDT -> IN if the tag of the preceding word is

| 'NN', and the tag of the following word is 'NNP'| 'NN', and the tag of the following word is 'NNP'

7 9 2 0 | RBR -> JJR if the tag of words i+1...i+2 is 'NNS'7 9 2 0 | RBR -> JJR if the tag of words i+1...i+2 is 'NNS'

7 10 3 0 | WDT -> IN if the tag of words i+1...i+2 is 'NNS'7 10 3 0 | WDT -> IN if the tag of words i+1...i+2 is 'NNS'

– 30 – CSCE 771 Spring 2011

5 5 0 0 | WDT -> IN if the tag of the preceding word is5 5 0 0 | WDT -> IN if the tag of the preceding word is

| 'NN', and the tag of the following word is 'PRP'| 'NN', and the tag of the following word is 'PRP'

4 4 0 1 | WDT -> IN if the tag of words i+1...i+3 is 'VBG'4 4 0 1 | WDT -> IN if the tag of words i+1...i+3 is 'VBG'

3 3 0 0 | RB -> IN if the tag of the preceding word is 'NN',3 3 0 0 | RB -> IN if the tag of the preceding word is 'NN',

| and the tag of the following word is 'DT'| and the tag of the following word is 'DT'

3 3 0 0 | RBR -> JJR if the tag of the following word is3 3 0 0 | RBR -> JJR if the tag of the following word is

| 'NN'| 'NN'

3 3 0 0 | VBP -> VB if the tag of words i-3...i-1 is 'MD'3 3 0 0 | VBP -> VB if the tag of words i-3...i-1 is 'MD'

3 3 0 0 | NNS -> NN if the text of the preceding word is3 3 0 0 | NNS -> NN if the text of the preceding word is

| 'one'| 'one'

3 3 0 0 | RP -> RB if the text of words i-3...i-1 is 'were'3 3 0 0 | RP -> RB if the text of words i-3...i-1 is 'were'

3 3 0 0 | VBP -> VB if the text of words i-2...i-1 is "n't"3 3 0 0 | VBP -> VB if the text of words i-2...i-1 is "n't"

Brill accuracy: 0.839156Brill accuracy: 0.839156

Done; rules and errors saved to rules.yaml and errors.out.Done; rules and errors saved to rules.yaml and errors.out.

– 31 – CSCE 771 Spring 2011