part-of-speech tagging for historical english · penn corpora of historical english modern briesh...
TRANSCRIPT
Part-of-SpeechTaggingforHistoricalEnglish
YiYangandJacobEisensteinGeorgiaTech
[MuralidharanandHearst,2011&2012]
‣ DigitalhumaniEesresearch
‣ HowdoestheportrayalofmenandwomendifferinShakespeare’splays?
‣ What’sthelanguageusepaMernsinNorthAmericanslavenarraEves?
[MuralidharanandHearst,2011&2012]
‣ NLPcanhelp!
‣ DigitalhumaniEesresearch
‣ HowdoestheportrayalofmenandwomendifferinShakespeare’splays?
‣ What’sthelanguageusepaMernsinNorthAmericanslavenarraEves?
[MuralidharanandHearst,2011&2012]
‣ NLPcanhelp!
‣ DigitalhumaniEesresearch
‣ HowdoestheportrayalofmenandwomendifferinShakespeare’splays?
‣ What’sthelanguageusepaMernsinNorthAmericanslavenarraEves?
‣ OnlyifNLPworksforhistoricaltexts…
EarlyModernEnglish
Heesaidnobodyhadsaidanythingagtmee.
[HenryOxinden,1660]
EarlyModernEnglish
Heesaidnobodyhadsaidanythingagtmee.
‣SpellingvariaEon
He againstHe me
[HenryOxinden,1660]
StanfordPOSTagger
Heesaidnobodyhadsaidanythingagtmee.
‣SpellingvariaEon
Stanford:
StanfordPOSTagger
Heesaidnobodyhadsaidanythingagtmee.X X X
‣SpellingvariaEon
Stanford:Gold:
TransferLossforPOSTagging
0
5
10
15
20
25
3.0
Errorrate
ModernEnglish
[Raysonetal.,2007]
TransferLossforPOSTagging
0
5
10
15
20
25
18.0
3.0
Errorrate
ModernEnglish
EarlyModernEnglish
[Raysonetal.,2007]
Approaches
‣ SpellingnormalizaEon }‣ Mapfromhistoricalspellingstocontemporaryforms.
Raysonetal.(2007)Scheibleetal.(2011)Bollmann(2011)
Approaches
‣ DomainadaptaEon(thiswork)
‣ SpellingnormalizaEon }‣ Mapfromhistoricalspellingstocontemporaryforms.
‣ BuildrobustNLPsystemswithrepresentaEonlearning.
Raysonetal.(2007)Scheibleetal.(2011)Bollmann(2011)
}Yang&Eisenstein(2014)Yang&Eisenstein(2015)
SpellingNormalizaEon
[VARD;BaronandRayson,2008]
Original:Heesaidnobodyhadsaidanythingagtmee.
Normalized:Heesaidnobodyhadsaidanythingagedme.
SpellingNormalizaEon
‣CorrectnormalizaEon
[VARD;BaronandRayson,2008]
Original:Heesaidnobodyhadsaidanythingagtmee.
Normalized:Heesaidnobodyhadsaidanythingagedme.
X
SpellingNormalizaEon
‣CorrectnormalizaEon
[VARD;BaronandRayson,2008]
Original:Heesaidnobodyhadsaidanythingagtmee.
Normalized:Heesaidnobodyhadsaidanythingagedme.
‣IncorrectnormalizaEon
X X
against
SpellingNormalizaEon
‣CorrectnormalizaEon
[VARD;BaronandRayson,2008]
Original:Heesaidnobodyhadsaidanythingagtmee.
Normalized:Heesaidnobodyhadsaidanythingagedme.
‣IncorrectnormalizaEon‣FalsenegaEve
X X
against
X
He
SpellingNormalizaEon
[VARD;BaronandRayson,2008]
Normalized:Heesaidnobodyhadsaidanythingagedme.
X XX
Stanford:Gold:
SpellingNormalizaEon
[VARD;BaronandRayson,2008]
Normalized:Heesaidnobodyhadsaidanythingagedme.
X XX
Stanford:Gold:
XX
RepresentaEonLearning
Heesaidnobodyhadsaidanythingagtmee.
RepresentaEonLearning
Heesaidnobodyhadsaidanythingagtmee.
RepresentaEonLearning
Heesaidnobodyhadsaidanythingagtmee.
RepresentaEonLearning
Heesaidnobodyhadsaidanythingagtmee.
Hee
saidwascametold…
} HeI
We…
saidwascametold…
}IVOOV Context Context
Model
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
u2
Outputembeddings
Inputembeddings
v1
v3
v4
p(ft|f2) / exp
�u2
>vt
�
FeatureEmbeddings
[FEMA;YangandEisenstein,2015]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
u2
Outputembeddings
Inputembeddings
v1
v3
v4
p(ft|f2) / exp
�u2
>vt
�
` =TX
t 6=2
logp(ft|f2)
WordEmbeddings
[word2vec;Mikolovetal.,2013]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
heesaidnobodyhad…
}words
1
2
3
4
1
2
3
4
‣ Wordembeddings
‣ Featureembeddings
WordEmbeddings
[word2vec;Mikolovetal.,2013]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
heesaidnobodyhad…
}words
1
2
3
4
1
2
3
4
‣ Wordembeddings
‣ Featureembeddings
‣ GenericrepresentaEons
WordEmbeddings
[word2vec;Mikolovetal.,2013]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
heesaidnobodyhad…
}words
1
2
3
4
1
2
3
4
‣ Wordembeddings
‣ Featureembeddings
‣ GenericrepresentaEons
‣ Task-specificrepresentaEons
WordEmbeddings
[word2vec;Mikolovetal.,2013]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
heesaidnobodyhad…
}words
1
2
3
4
1
2
3
4
‣ Wordembeddings
‣ Featureembeddings
‣ GenericrepresentaEons
‣ Task-specificrepresentaEons
‣ Wordco-occurrences
WordEmbeddings
[word2vec;Mikolovetal.,2013]
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
heesaidnobodyhad…
}words
1
2
3
4
1
2
3
4
‣ Wordembeddings
‣ Featureembeddings
‣ GenericrepresentaEons
‣ Task-specificrepresentaEons
‣ Wordco-occurrences
‣ Featureco-occurrences
LearningfromMulEpleDomains
[FEMA;YangandEisenstein,2015]
‣PreviousworkonunsuperviseddomainadaptaEoninvolvesintwodomains.
LearningfromMulEpleDomains
[FEMA;YangandEisenstein,2015]
‣PreviousworkonunsuperviseddomainadaptaEoninvolvesintwodomains.‣UnsupervisedmulE-domainadaptaEon
LearningfromMulEpleDomains
[FEMA;YangandEisenstein,2015]
‣PreviousworkonunsuperviseddomainadaptaEoninvolvesintwodomains.‣UnsupervisedmulE-domainadaptaEon
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
DomainAMributes: Genre Epoch
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
leMers 1600+
DomainAMributes: Genre Epoch
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
leMers 1600+
DomainAMributes:
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
Genre Epoch
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
leMers 1600+
DomainAMributes:
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
= +(shared) (leMers)
+(1600+)
Genre Epoch
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.
leMers 1600+
DomainAMributes:
CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
= +(shared) (leMers)
+(1600+)
Genre Epoch
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
(shared) (leMers)= + +
(1600+)
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
(shared) (leMers)
= + +
(1600+)
u2 = h(shared)2 + h(letters)
2 + h(1600+)2
= + +
MulEpleFeatureEmbeddings
[FEMA;YangandEisenstein,2015]
Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said
Prefix1=hSuffix1=e
…
}features
1
2
3
4
(shared) (leMers) (1600+)
u2 = h(shared)2 + h(letters)
2 + h(1600+)2
= + +
p(ft|f2) / exp
�u2
>vt
�
Experiments
PennCorporaofHistoricalEnglishModernBriEshEnglish(MBE)
1840-1914
1770-1839
1700-1769
0 110,000 220,000 330,000 440,000
343,024
427,424
322,255
#oftokens
EarlyModernEnglish(EME)
1640-1710
1570-1639
1500-1569
0 177,500 355,000 532,500 710,000
640,255
706,587
614,315
#oftokens
[KrochandTaylor,2000;Krochetal.,2004]
TagsetMappings
‣PennCorporaofHistoricalEnglish(PCHE)tagset:83tags‣PennTreebank(PTB)tagset:45tags
[MoonandBaldridge,2007]
TagsetMappings
‣PennCorporaofHistoricalEnglish(PCHE)tagset:83tags‣PennTreebank(PTB)tagset:45tags
[MoonandBaldridge,2007]
ADJ
PCHE PTB
JJADVALSO RB
VB VBVBI… …
Systems
‣Supportvectormachine(SVM)tagger‣SixteenbasicfeaturetemplatesbyRatnaparkhi(1996)
Systems
‣Supportvectormachine(SVM)tagger
‣RepresentaEonlearningmethods
‣SixteenbasicfeaturetemplatesbyRatnaparkhi(1996)
‣Structuralcorrespondencelearning(SCL)‣Brownclustering‣word2vecembeddings‣MulEplefeatureembeddings(FEMA)
[Blitzeretal.,2006;Brownetal.,1992;Mikolovetal.,2013]
TemporalAdaptaEonModernBriEshEnglish(MBE)
1840-1914
1770-1839
1700-1769
0 110,000 220,000 330,000 440,000
343,024
427,424
322,255
#oftokens
EarlyModernEnglish(EME)
1640-1710
1570-1639
1500-1569
0 177,500 355,000 532,500 710,000
640,255
706,587
614,315
#oftokens
Train Train
Test1 Test1
Test2 Test2
0
1.2
2.4
3.6
4.8
6
4.6
Averageerrorrate
Baseline SCL Brown word2vecFEMA
Results:ModernBriEshEnglish
0
1.2
2.4
3.6
4.8
6
4.44.24.34.6
Averageerrorrate
Baseline SCL Brown word2vecFEMA
Results:ModernBriEshEnglish
0
1.2
2.4
3.6
4.8
6
3.74.44.24.3
4.6
Averageerrorrate
Baseline SCL Brown word2vecFEMA
Results:ModernBriEshEnglish
(-0.9)
(Ourmethod)
0
2.2
4.4
6.6
8.8
11
9.4
BaselineSCL Brown word2vec
FEMA
Averageerrorrate
Results:EarlyModernEnglish
0
2.2
4.4
6.6
8.8
11
8.38.08.29.4
BaselineSCL Brown word2vec
FEMA
Averageerrorrate
Results:EarlyModernEnglish
0
2.2
4.4
6.6
8.8
11
6.6
8.38.08.29.4
BaselineSCL Brown word2vec
FEMA
Averageerrorrate
Results:EarlyModernEnglish
(-2.8)
(Ourmethod)
AdaptaEonfromPTB
PennTreebank
ModernBriEshEnglish
EarlyModernEnglish
0 500,000 1,000,000 1,500,000 2,000,000
1,961,157
1,092,703
969,905
#oftokens
Train
Test1
Test2
AdaptaEonfromPTB
StandardevaluaEonscenarioforEnglishPOStagging.
AdaptaEonfromPTB
StandardevaluaEonscenarioforEnglishPOStagging.
‣Lowresourcelanguages‣Specificgenres,styles,orepochs
InsufficientdataannotaEonforhistoricaltexts.
0
4.6
9.2
13.8
18.4
23
18.9
Errorrate
Baseline SCL Brown word2vec FEMA
Results:ModernBriEshEnglish
0
4.6
9.2
13.8
18.4
23
18.318.418.418.9
Baseline SCL Brown word2vec FEMAErrorrate
Results:ModernBriEshEnglish
0
4.6
9.2
13.8
18.4
23
17.518.318.418.418.9
Baseline SCL Brown word2vec FEMAErrorrate
Results:ModernBriEshEnglish
(-1.4)
(Ourmethod)
0
6
12
18
24
30
25.9
Baseline SCL Brown word2vec FEMAErrorrate
Results:EarlyModernEnglish
0
6
12
18
24
30
24.224.024.125.9
Baseline SCL Brown word2vec FEMAErrorrate
Results:EarlyModernEnglish
Results:EarlyModernEnglish
0
6
12
18
24
30
22.124.224.024.1
25.9
Baseline SCL Brown word2vec FEMAErrorrate
(-3.8)(Ourmethod)
0
6
12
18
24
30
22.125.9
Baseline FEMA+VARD
FEMAErrorrate
NormalizaEonvs.RepresentaEonLearning
(-3.8) (-2.6)(-4.9)
RepresentaEonlearning
(-3.8)
FEMA
(-3.8)22.1
0
6
12
18
24
30
23.322.125.9
Baseline VARD FEMA+VARD
FEMAErrorrate
NormalizaEonvs.RepresentaEonLearning
(-3.8) (-2.6)(-4.9)
RepresentaEonlearning
SpellingnormalizaEon
0
6
12
18
24
30
21.023.322.1
25.9
Baseline VARD FEMA+VARD
FEMAErrorrate
NormalizaEonvs.RepresentaEonLearning
(-3.8) (-2.6)(-4.9)
RepresentaEonlearning+
normalizaEon
RepresentaEonlearning
SpellingnormalizaEon
token annotaEonsinPCHE annotaEonsinPTB
,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)
.(period) ,(comma;12.3%).(period;87.7%) .(period)
to TO(54.6%)IN(44.3%) TO
all/any/every JJ DT
ErrorAnalysis
‣AnnotaEoninconsistenciesandtagsetmismatches
token annotaEonsinPCHE annotaEonsinPTB
,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)
.(period) ,(comma;12.3%).(period;87.7%) .(period)
to TO(54.6%)IN(44.3%) TO
all/any/every JJ DT
ErrorAnalysis
‣AnnotaEoninconsistenciesandtagsetmismatches
token annotaEonsinPCHE annotaEonsinPTB
,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)
.(period) ,(comma;12.3%).(period;87.7%) .(period)
to TO(54.6%)IN(44.3%) TO
all/any/every JJ DT
ErrorAnalysis
‣AnnotaEoninconsistenciesandtagsetmismatches
ErrorAnalysis
token annotaEonsinPCHE annotaEonsinPTB
,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)
.(period) ,(comma;12.3%).(period;87.7%) .(period)
to TO(54.6%)IN(44.3%) TO
all/any/every JJ(quanEfier) DT
‣AnnotaEoninconsistenciesandtagsetmismatches
Conclusions
Conclusions
‣ FeatureembeddingsoutperformwordembeddingsbyexploiEngtask-specificinformaEoninfeaturetemplates.
Conclusions
‣ RepresentaEonlearningandspellingnormalizaEonarecomplementaryforimprovingtaggingperformance.
‣ FeatureembeddingsoutperformwordembeddingsbyexploiEngtask-specificinformaEoninfeaturetemplates.
Conclusions
‣ RepresentaEonlearningandspellingnormalizaEonarecomplementaryforimprovingtaggingperformance.
‣ TagsetmismatchesmakeithardtoevaluatemodernPOStaggersforhistoricalEnglish.
‣ FeatureembeddingsoutperformwordembeddingsbyexploiEngtask-specificinformaEoninfeaturetemplates.