Bilingual Word Representations with Monolingual Quality in Mind

Minh-Thang Luong, Hieu Pham, Christopher D. Manning

Proceedings of NAACL-HLT 2015 Workshop

AHC-Lab

M1 Hiroyuki Fudaba

1

What are Word Representations?

Vectors representing words

• One-hot word representations

• Distributed word representations [Bengio et al. 2003]

0, 0, 0, … , 0, 1, 0, 0, 0, … , 0

1.1, 0.5, −3.2, 0.5, … , 0.4

2

Distributed Word Representations

• Vectors representing words’ syntactic / semantic features

3

2 different languages in 1 vector space

4

Why do we need bilingual word representations?

• Crosslingual document classification

5

Apple Inc. Google

apple banana

companies

fruits

アップル株式会社

りんご

Which is more appropriate?

How to do 2-in-1

• Mapping

• Learning with Joint model

6

𝑦 = 𝑊𝑥dog

cat

犬

猫

cat猫

dog犬

Problem of previous work

Perform poorly on monolingual tasks

Why?

tradeoff between bilingual tasks’ performance and monolinguals’

7

Paper’s approach

Substitute words to predict surroundings

8

Which one to substitute?

1. No alignment (BiSkip-MonoAlign)

2. Align before substitution (BiSkip-UnsupAlign)

I have a dog .

私は 犬を 飼って います ．

9

Which one to substitute?

1. No alignment (BiSkip-MonoAlign)

2. Align before substitution (BiSkip-UnsupAlign)

I have a dog .

私は 犬を 飼って います ．

10

Bilingual Skipgram Model

11

犬

is

my

,

Delicious

Try to predict“is my , Delicious” from “犬”

Evaluation: word similarity

• Measures semantic quality of the word vectors monolingually

e.g.

tiger cat

computer keyboard internet

12

Evaluation: CLDC

Train with language A’s vector, and predict documents with language B

13

Document classifier (perceptron)

Result

14

Conclusion and future work

What this paper say

• Substituting words make better bilingual word representations

Future work

• Pivoting to improve performance

15

references

• [Bengio et al. 2003] A Neural Probabilistic Language Model

• [Xiaochuan et al. 2011] Cross Lingual Text Classification by Mining Multilingual Topics from Wikipedia

16