Phra
se T
agse
t Map
ping
for
Fren
ch a
nd E
nglis
hTr
eeba
nks a
nd It
s App
licat
ion
in M
achi
ne T
rans
latio
nEv
alua
tion
25th
Inte
rnat
iona
l Con
fere
nce,
GSC
L 20
13
Aar
on L
.-F. H
an, D
erek
F. W
ong,
Lid
ia S
. Cha
o, L
iang
ye H
e, S
huo
Li,
and
Ling
Zhu
Se
ptem
ber 2
5th
-27t
h, 2
013,
Dar
mst
adt,
Ger
man
y
Natu
ral L
angu
age
Proc
essin
g &
Por
tugu
ese-
Chin
ese
Mach
ine
Tran
slatio
n La
bora
tory
Dep
artm
ent o
f Com
pute
r and
Info
rmat
ion
Sci
ence
Uni
vers
ity o
f Mac
au
●B
ackg
roun
d of
lang
uage
Tre
eban
k●
Mot
ivat
ion
●D
esig
ned
phra
se ta
gset
map
ping
●A
pplic
atio
n in
MT
eval
uatio
n1.
Man
ual e
valu
atio
ns2.
Trad
ition
al a
utom
atic
MT
eval
uatio
n m
etho
ds3.
Des
igne
d un
supe
rvis
ed M
T ev
alua
tion
4.E
valu
atin
g th
e ev
alua
tion
met
hod
5.E
xper
imen
ts6.
Ope
n so
urce
cod
e●
Dis
cuss
ion
●Fu
rthe
r in
form
atio
n
Contents
1. B
ackg
roun
d of
lang
uage
Tre
eban
k
•To
pro
mot
e th
e de
velo
pmen
t of s
ynta
ctic
ana
lysi
s•
Man
y la
ngua
ge tr
eeba
nks a
re d
evel
oped
–En
glis
h Pe
nn T
reeb
ank
(Mar
cus e
t al.,
199
3; M
itche
ll et
al.,
1994
)–
Ger
man
Neg
ra T
reeb
ank
(Sku
t et a
l., 1
997)
–Fr
ench
Tre
eban
k (A
beill
é et
al.,
200
3)–
Chi
nese
Sin
ica
Tree
bank
(Che
n et
al.,
200
3)–
Etc.
1. B
ackg
roun
d of
lang
uage
Tre
eban
k
•Pr
oble
ms
–D
iffer
ent t
reeb
anks
use
thei
r ow
n sy
ntac
tic ta
gset
s–
The
num
ber o
f tag
s ran
ging
from
tens
(e.g
. Eng
lish
Penn
Tree
bank
) to
hund
reds
(e.g
. Chi
nese
Sin
ica
Tree
bank
)–
Inco
nven
ient
whe
n un
derta
king
the
mul
tilin
gual
or c
ross
-lin
gual
rese
arch
2. M
otiv
atio
n
•To
brid
ge th
e ga
p be
twee
n th
ese
treeb
anks
and
faci
litat
e fu
ture
rese
arch
–E.
g. th
e un
supe
rvise
d in
duct
ion
of sy
ntac
tic st
ruct
ure
•Pe
trov
et a
l. (2
012)
dev
elop
a u
nive
rsal
PO
S ta
gset
•H
ow a
bout
the
phra
se le
vel t
ags?
•Th
e di
sacc
ord
prob
lem
in th
e ph
rase
leve
l tag
sre
mai
ns u
nsol
ved
–Le
t’s tr
y to
solv
e it
3. D
esig
ned
phra
se ta
gset
map
ping
•Te
ntat
ive
desig
n of
phr
ase
tags
et m
appi
ng–
On
Engl
ish P
enn
Tree
bank
I, II
& F
renc
h Tr
eeba
nk•
9 un
iver
sal p
hras
al c
ateg
orie
s cov
erin
g–
14 p
hras
e ta
gs in
Eng
lish
Penn
Tre
eban
k I
–26
phr
ase
tags
in E
nglis
h Pe
nn T
reeb
ank
II–
14 p
hras
e ta
gs in
Fre
nch
Tree
bank
3. D
esig
ned
phra
se ta
gset
map
ping
Tabl
e 1:
phr
ase
tags
et m
appi
ng fo
r Fre
nch
and
Eng
lish
treeb
anks
3. D
esig
ned
phra
se ta
gset
map
ping
•U
nive
rsal
phr
asal
cat
egor
ies:
NP
(nou
n ph
rase
), V
P(v
erb
phra
se),
AJP
(adj
ectiv
e ph
rase
), AV
P (a
dver
bial
phra
se),
PP (p
repo
sitio
nal p
hras
e), S
(sub
/-sen
tenc
e),
CON
JP (c
onju
nctio
n ph
rase
), CO
P (c
oord
inat
edph
rse)
, X (o
ther
phr
ases
or u
nkno
wn)
•N
P co
verin
g–
Fren
ch ta
gs: N
P–
Engl
ish ta
gs: N
P, N
AC
(the
scop
e of
cer
tain
pre
nom
inal
mod
ifier
s with
in a
n N
P), N
X (w
ithin
cer
tain
com
plex
NPs
to m
ark
the
head
of N
P), W
HN
P (w
h-no
un p
hras
e), Q
P(q
uant
ifier
phr
ase)
3. D
esig
ned
phra
se ta
gset
map
ping
•V
P co
verin
g–
Fren
ch ta
gs: V
N (v
erba
l nuc
leus
), V
P (in
finiti
ves a
ndno
nfini
te c
laus
es)
–En
glis
h ta
gs: V
P (v
erb
phra
se)
•A
JP c
over
ing
–Fr
ench
tags
: AP
(adj
ectiv
al p
hras
e)–
Engl
ish
tags
: AD
JP (a
djec
tive
phra
se),
WH
AD
JP (w
h-ad
ject
ive
phra
se)
3. D
esig
ned
phra
se ta
gset
map
ping
•A
VP
cove
ring
–Fr
ench
tags
: AdP
(adv
erbi
al p
hras
es)
–En
glis
h ta
gs: A
DV
P (a
dver
b ph
rase
), W
HAV
P (w
h-ad
verb
phra
se),
PRT
(par
ticle
)•
PP c
over
ing
–Fr
ench
tags
: PP
–En
glis
h ta
gs: P
P, W
HPP
(wh-
prop
ositi
onal
phr
ase
phra
se)
3. D
esig
ned
phra
se ta
gset
map
ping
•S
cove
ring
–Fr
ench
tags
: SEN
T (s
ente
nce)
, S (fi
nite
cla
use)
–En
glish
tags
: S (s
impl
e de
clar
ativ
e cl
ause
), SB
AR
(cla
use
intro
duce
d by
a su
bord
inat
ing
conj
unct
ion)
, SBA
RQ (d
irect
ques
tion
intro
duce
d by
a w
h-ph
rase
), SI
NV
(dec
lara
tive
sent
ence
with
subj
ect-a
ux in
vers
ion)
, SQ
(sub
-con
stitu
ent
of S
BARQ
), PR
N (p
aren
thet
ical
), FR
AG
(fra
gmen
t), R
RC(re
duce
d re
lativ
e cl
ause
).•
CON
JP c
over
ing
–Fr
ench
tags
: N/A
–En
glish
tags
: CO
NJP
3. D
esig
ned
phra
se ta
gset
map
ping
•CO
P co
verin
g–
Fren
ch ta
gs: C
OO
RD (c
oord
inat
ed p
hras
e)–
Engl
ish ta
gs: U
CP (c
oord
inat
ed p
hras
es b
elon
ging
todi
ffere
nt c
ateg
orie
s)•
X c
over
ing
–Fr
ench
tags
: unk
now
n–
Engl
ish ta
gs: X
(unk
now
n or
unc
erta
in),
INTJ
(inte
rject
ion)
, LST
(list
mar
ker)
4. A
pplic
atio
n in
4.
App
licat
ion
in M
achi
ne T
rans
latio
n M
achi
ne T
rans
latio
n ev
alua
tion
eval
uatio
n
4.1
Man
ual e
valu
atio
ns
•R
apid
dev
elop
men
t of M
achi
ne T
rans
latio
ns–
MT
bega
n as
ear
ly a
s in
the
1950
s (W
eave
r, 19
55)
–B
ig p
rogr
ess s
cien
ce th
e 19
90s d
ue to
the
deve
lopm
ent o
fco
mpu
ters
(sto
rage
cap
acity
and
com
puta
tiona
l pow
er) a
ndth
e en
larg
ed b
iling
ual c
orpo
ra (M
arin
o et
al.
2006
)•
Diffi
culti
es o
f MT
eval
uatio
n–
lang
uage
var
iabi
lity
resu
lts in
no
sing
le c
orre
ct tr
ansl
atio
n–
the
natu
ral l
angu
ages
are
hig
hly
ambi
guou
s and
diff
eren
tla
ngua
ges d
o no
t alw
ays e
xpre
ss th
e sa
me
cont
ent i
n th
esa
me
way
(Arn
old,
200
3)
4.1
Man
ual e
valu
atio
ns
•Tr
aditi
onal
man
ual e
valu
atio
n cr
iteria
:–
inte
lligi
bilit
y (m
easu
ring
how
und
erst
anda
ble
the
sent
ence
is)
–fid
elity
(mea
surin
g ho
w m
uch
info
rmat
ion
the
trans
late
dse
nten
ce re
tain
s as c
ompa
red
to th
e or
igin
al) b
y th
eA
utom
atic
Lan
guag
e Pr
oces
sing
Adv
isor
y C
omm
ittee
(ALP
AC
) aro
und
1966
(Car
roll,
196
6)–
adeq
uacy
(sim
ilar a
s fide
lity)
, flue
ncy
(whe
ther
the
sent
ence
is w
ell-f
orm
ed a
nd fl
uent
) and
com
preh
ensio
n(im
prov
ed in
telli
gibi
lity)
by
Def
ense
Adv
ance
d R
esea
rch
Proj
ects
Age
ncy
(DA
RPA
) of U
S (W
hite
et a
l., 1
994)
4.1
Man
ual e
valu
atio
ns
•Pr
oble
ms o
f m
anu
al e
valu
atio
ns :
–Ti
me-
cons
umin
g–
Expe
nsiv
e–
Unr
epea
tabl
e–
Low
agr
eem
ent (
Cal
lison
-Bur
ch, e
t al.,
201
1)
4.2
Trad
ition
al a
utom
atic
MT
eval
uatio
ns
•M
easu
ring
the
simila
rity
of a
utom
atic
tran
slatio
n an
dre
fere
nce
trans
latio
n–
Aut
omat
ic tr
ansla
tion
(or h
ypot
hesis
tran
slatio
n, ta
rget
trans
latio
n): b
y au
tom
atic
MT
syste
m–
Refe
renc
e tra
nsla
tion:
by
prof
essio
nal t
rans
lato
rs–
Sour
ce la
ngua
ge a
nd so
urce
doc
umen
t: no
t use
d•
Trad
ition
al a
utom
atic
eva
luat
ion:
–BL
EU: n
-gra
m p
reci
sions
(Pap
inen
i et a
l., 2
002)
–TE
R: e
dit d
istan
ces (
Snov
er e
t al.,
200
6)–
MET
EOR:
pre
cisio
n an
d re
call
(Ban
erje
e an
d La
vie,
200
5)
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•Pr
oble
ms i
n su
perv
ised
MT
eval
uatio
n–
Ref
eren
ce tr
ansl
atio
ns a
re e
xpen
sive
–R
efer
ence
tran
slat
ions
are
not
ava
ilabl
e is
som
e ca
ses
•C
ould
we
get r
id o
f the
refe
renc
e tra
nsla
tion?
–U
nsup
ervi
sed
MT
eval
uatio
n m
etho
d–
Extra
ct in
form
atio
n fr
om so
urce
and
targ
et la
ngua
ge–
How
to u
se th
e de
sign
ed u
nive
rsal
phr
ase
tags
et?
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•A
ssum
e th
at th
e tra
nsla
ted
sent
ence
shou
ld h
ave
asi
mila
r se
t of p
hras
e ca
tego
ries
with
the
sour
cese
nten
ce.
–Th
is d
esig
n is
insp
ired
by th
e sy
nony
mou
s rel
atio
n be
twee
nso
urce
and
targ
et se
nten
ce.
•Tw
o se
nten
ces t
hat h
ave
sim
ilar s
et o
f phr
ases
may
talk
abo
ut d
iffer
ent t
hing
s.–
How
ever
, thi
s eva
luat
ion
appr
oach
is n
ot d
esig
ned
for
gene
ral c
ircum
stan
ce–
Ass
ume
that
the
targ
eted
sent
ence
s are
inde
ed th
etra
nsla
ted
sent
ence
s fro
m th
e so
urce
doc
umen
t
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•Fi
rst,
we
pa
rse
the
sour
ce a
nd ta
rget
lang
uage
sre
spec
tivel
y•
Then
we
extr
act
the
phr
ase
set f
rom
the
sour
ce a
ndta
rget
sent
ence
s•
Third
, we
conv
ert t
he p
hras
es in
to th
e de
velo
ped
univ
ersa
l phr
ase
cate
gorie
s•
Last
, we
mea
sure
the
sim
ilarit
y of
sour
ce a
nd ta
rget
lang
uage
on
the
univ
ersa
l phr
ase
sequ
ence
s
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
Figu
re 1
: the
par
sed
Fren
ch a
nd E
nglis
h se
nten
ce
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
Figu
re 2
: con
vert
the
extra
cted
phr
ase
into
uni
vers
al p
hras
e ta
gs
The
leve
l of e
xtra
cted
phr
ase
tags
: jus
t the
upp
er le
vel o
f PO
S ta
gs, b
otto
m-u
p
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•W
hat i
s the
sim
ilarit
y m
etric
we
empl
oyed
?•
Des
igne
d si
mila
rity
met
ric: H
PPR
–N
1 g
ram
pos
ition
ord
er d
iffer
ence
pen
alty
–W
eigh
ted
N2
gram
pre
cisi
on–
Wei
ghte
d N
3 gr
am re
call
–W
eigh
ted
geom
etric
mea
n in
n-g
ram
pre
cisi
on &
reca
ll–
Wei
ghte
d ha
rmon
ic m
ean
to c
ombi
ne su
b-fa
ctor
s–
The
para
met
ers a
re tu
nabl
e ac
cord
ing
to d
iffer
ent l
angu
age
pairs
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
Figu
re 3
: N1
gram
tag
alig
nmen
t alg
orith
m
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
•
4.3
Des
igne
d un
supe
rvise
d M
T ev
alua
tion
Figu
re 5
: big
ram
chu
nk m
atch
ing
exam
ple
4.4
Eval
uatin
g th
e ev
alua
tion
met
hod
•
4.5
Expe
rim
ents
•C
orp
us fr
om W
MT
–W
orks
hop
of st
atis
tical
mac
hine
tran
slat
ion
–SI
GM
T, A
CL’
S sp
ecia
l int
eres
t gro
up o
f mac
hine
trans
latio
n•
Trai
ning
dat
a (W
MT1
1), t
une
the
para
met
ers
–3,
003
sent
ence
s for
eac
h do
cum
ent
–18
aut
omat
ic F
renc
h-to
-Eng
lish
MT
syst
ems
•Te
stin
g da
ta (W
MT1
2)–
3, 0
03 se
nten
ces f
or e
ach
docu
men
t–
15 a
utom
atic
Fre
nch-
to-E
nglis
h M
T sy
stem
s
4.5
Expe
rim
ents
•Tr
aini
ng, t
une
the
pa
ram
eter
s–
N1,
N2
and
N3
are
tune
d as
2, 3
and
3 d
ue to
the
fact
that
the
4-gr
am c
hunk
mat
ch u
sual
ly re
sults
in 0
scor
e.–
Tune
d va
lues
of f
acto
r wei
ghts
are
show
n in
tabl
e
Tabl
e 2:
tune
d pa
ram
eter
val
ues
4.5
Expe
rim
ents
•C
ompa
rison
s with
:–
BLE
U, m
easu
re th
e cl
osen
ess o
f the
hyp
othe
sis a
ndre
fere
nce
trans
latio
ns, n
-gra
m p
reci
sion
–TE
R, m
easu
re th
e ed
iting
dis
tanc
e of
hyp
othe
sis t
ore
fere
nce
trans
latio
ns
4.5
Expe
rim
ents
Tabl
e 3:
trai
ning
(dev
elop
men
t) sc
ores
on
WM
T11
corp
us
Tabl
e 4:
test
ing
scor
es o
n W
MT1
2 co
rpus
4.5
Expe
rim
ents
Tabl
e 5:
cor
rela
tion
scor
e in
tro (C
ohen
, 198
8)
●Th
e ex
perim
ent r
esul
ts o
n th
e de
velo
pmen
t and
test
ing
corp
ora
show
that
HP
PR
with
out u
sing
ref
eren
ce tr
ansl
atio
ns h
as y
ield
ed p
rom
isin
g co
rrel
atio
nsc
ores
(0.6
3 an
d 0.
59 re
spec
tivel
y).
●Th
ere
is s
till p
oten
tial t
o im
prov
e th
e pe
rform
ance
s of
all
the
thre
e m
etric
s,ev
en t
houg
h th
at t
he c
orre
latio
n sc
ores
whi
ch a
re h
ighe
r th
an 0
.5 a
real
read
y co
nsid
ered
as
stro
ng c
orre
latio
n as
sho
wn
in T
able
5.
4.6
Ope
n so
urce
cod
e
•Ph
rase
Tag
set M
appi
ng fo
r Fre
nch
and
Engl
ish
Tree
bank
san
d Its
App
licat
ion
in M
achi
ne T
rans
latio
n Ev
alua
tion
–A
aron
L.-F
. Han
, Der
ek F
. Won
g, L
idia
S. C
hao,
Lia
ngye
He,
Shu
o Li
, and
Lin
g Zh
u. G
SCL
2013
, Dar
mst
adt,
Ger
man
y. L
NC
S Vo
l. 81
05, p
p. 1
19-1
31, V
olum
e Ed
itors
:Ir
yna
Gur
evyc
h, C
hris
Bie
man
n an
d To
rste
n Ze
sch.
•O
pen
sour
ce to
ol fo
r phr
ase
tags
et m
appi
ng a
nd H
PPR
simila
rity
mea
surin
g al
gorit
hms:
http
s://g
ithub
.com
/aar
onlif
engh
an/a
aron
-pro
ject
-hpp
r
5. D
iscus
sion
•Fa
cilit
ate
futu
re re
sear
ch in
mul
tilin
gual
or c
ross
-lin
gual
lite
ratu
re, t
his p
aper
des
igns
a p
hras
e ta
gsm
appi
ng b
etw
een
the
Fren
ch T
reeb
ank
and
the
Engl
ish P
enn
Tree
bank
usin
g 9
phra
se c
ateg
orie
s.•
One
of t
he p
oten
tial a
pplic
atio
ns o
f the
des
igne
dun
iver
sal p
hras
e ta
gset
is sh
own
in th
e un
supe
rvise
dM
T ev
alua
tion
task
in th
e ex
perim
ent s
ectio
n.
5. D
iscus
sion
•Th
ere
are
still
som
e lim
itatio
ns in
this
wor
k to
be
addr
esse
d in
the
futu
re.
–Th
e de
sign
ed u
nive
rsal
phr
ase
cate
gorie
s may
not
be
able
to co
ver a
ll th
e ph
rase
tags
of o
ther
lang
uage
tree
bank
s,so
this
tags
et c
ould
be
expa
nded
whe
n ne
cess
ary.
–Th
e de
sign
ed H
PPR
form
ula c
onta
ins t
he n
-gra
m fa
ctor
sof
pos
ition
diff
eren
ce, p
reci
sion
and
reca
ll, w
hich
may
not
be su
ffici
ent o
r sui
tabl
e fo
r som
e of
the
othe
r lan
guag
epa
irs, s
o di
ffere
nt m
easu
ring
fact
ors s
houl
d be
adde
d or
switc
hed
whe
n fa
cing
new
task
s.
5. D
iscus
sion
•A
ctua
lly sp
eaki
ng, t
he d
esig
ned
mod
els a
re v
ery
rela
ted
to th
e si
mila
rity
mea
surin
g. W
here
we
have
empl
oyed
them
is in
the
MT
eval
uatio
n. T
hese
wor
ksm
ay b
e fu
rther
dev
elop
ed in
to o
ther
lite
ratu
re:
–in
form
atio
n re
triev
al–
ques
tion
and
answ
erin
g–
Sear
chin
g–
text
ana
lysi
s–
etc.
6. F
urth
er in
form
atio
n
•O
ngoi
ng a
nd fu
rther
wor
ks:
–Th
e co
mbi
natio
n of
tran
slatio
n an
d ev
alua
tion,
tuni
ng th
etra
nsla
tion
mod
el u
sing
eval
uatio
n m
etric
s–
Eval
uatio
n m
odel
s fro
m th
e pe
rspe
ctiv
e of
sem
antic
s–
The
furth
er e
xplo
ratio
ns o
f uns
uper
vise
d ev
alua
tion
mod
els,
extra
ctin
g ot
her f
eatu
res f
rom
sour
ce a
nd ta
rget
lang
uage
s•
Aar
on o
pen
sour
ce to
ols:
http
s://g
ithub
.com
/aar
onlif
engh
an•
Aar
on n
etw
ork
Hom
e: h
ttp://
ww
w.lin
kedi
n.co
m/in
/aar
onha
n
Phra
se T
agse
t Map
ping
for
Fren
ch a
nd E
nglis
hTr
eeba
nks a
nd It
s App
licat
ion
in M
achi
neTr
ansla
tion
Eval
uatio
nG
SCL
2013
, Dar
mst
adt,
Ger
man
y
Aar
on L
.-F. H
anem
ail:
hanl
ifeng
aaro
n AT
gm
ail D
OT
com
Natu
ral L
angu
age
Proc
essin
g &
Por
tugu
ese-
Chin
ese
Mach
ine
Tran
slatio
n La
bora
tory
Dep
artm
ent o
f Com
pute
r and
Info
rmat
ion
Sci
ence
Uni
vers
ity o
f Mac
au
Q a
nd A
