CS224N Research Highlight

A Simple but Tough-to-beat Baseline for Sentence Embeddings

Sanjeev Arora, Yingyu Liang, Tengyu MaPrinceton University

Presenter: Danqi Chen

In submission to ICLR 2017

linguistics =

2

Word Sentence ?

0

BBBBBBBBBB@

0.2860.792�0.177�0.1070.109�0.5420.3490.271

1

CCCCCCCCCCA

linguistics =

2

Word Sentence ?

Natural language processing is fun. =

0

BBBBBBBBBB@

�0.1321.1290.8270.110�0.5270.1560.349�0.286

1

CCCCCCCCCCA

0

BBBBBBBBBB@

0.2860.792�0.177�0.1070.109�0.5420.3490.271

1

CCCCCCCCCCA

3

Sentence embedding

• Compute sentence similarity using the inner product:

S1: Mexico wishes to guarantee citizen’s safety.

S2: Mexico wishes to avoid more violence.Score: 4 (/5)

S1: Iranians Vote in Presidential Election.

S2: Keita Wins Mali Presidential Election. Score: 0.4 (/5)

4

Sentence embedding

Natural language processing is fun. =

0

BBBBBBBBBB@

�0.1321.1290.8270.110�0.5270.1560.349�0.286

1

CCCCCCCCCCA

• Use as features for sentence classification (e.g., sentiment analysis):

• Bag-of-words (BoW) v(“natural language processing”) =

1/3 (v(“natural”) + v(“language”) + v(“processing”))

5

From Bag-of-words to Complex Models…

• Bag-of-words (BoW) v(“natural language processing”) =

1/3 (v(“natural”) + v(“language”) + v(“processing”))

5

From Bag-of-words to Complex Models…

• Recurrent neural networks, recursive neural networks, convolutional neural networks..

• A VERY SIMPLE unsupervised method

• weighted Bag-of-words + remove some special direction

6

This paper

• A VERY SIMPLE unsupervised method

• weighted Bag-of-words + remove some special direction

6

This paper

• Step 1:

• A VERY SIMPLE unsupervised method

• weighted Bag-of-words + remove some special direction

6

This paper

• Step 1:

• Step 2:

7

A Probabilistic Interpretation

       

         

 

   

 

commondiscourse,oftenrelatedtosyntax

8

Resultssentence similarity

sentence classification

8

Results

Thanks!

sentence similarity

sentence classification