encoding logic rules in sentiment ... - the last...
TRANSCRIPT
![Page 1: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/1.jpg)
Encoding Logic Rules in Sentiment Classification
Kalpesh Krishna* UMass Amherst
Preethi Jyothi IIT Bombay
Mohit Iyyer UMass Amherst
* Work done at IIT Bombay
![Page 2: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/2.jpg)
Sentiment Classification
![Page 3: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/3.jpg)
Sentiment Classification
Classify a sentence as positive or negative
![Page 4: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/4.jpg)
Sentiment Classification
Classify a sentence as positive or negative
this movie has a great story
![Page 5: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/5.jpg)
Sentiment Classification
Classify a sentence as positive or negative
this movie has a great story
Sentiment = Positive
![Page 6: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/6.jpg)
Not Always Easy!
this movie has a great story
Solution :- Lexicons, Bag of Words
![Page 7: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/7.jpg)
Not Always Easy!
this movie has a great story
Solution :- Lexicons, Bag of Words
Easy!
![Page 8: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/8.jpg)
Not Always Easy!
this movie has a great story
Solution :- Lexicons, Bag of Words
Easy!
Contrastive - this movie is funny, but horribly directed
Negation - this is not a movie worth waiting for
![Page 9: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/9.jpg)
Not Always Easy!
this movie has a great story
Solution :- Lexicons, Bag of Words
Easy!
Contrastive - this movie is funny, but horribly directed
Negation - this is not a movie worth waiting for
Much Harder!
![Page 10: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/10.jpg)
Logic Rules
this movie is funny, but horribly directed A-but-B
![Page 11: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/11.jpg)
Logic Rules
this movie is funny, but horribly directed A-but-B
sentiment(A-but-B) = sentiment(B)
![Page 12: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/12.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
![Page 13: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/13.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit
![Page 14: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/14.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
![Page 15: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/15.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
![Page 16: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/16.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit?
![Page 17: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/17.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
![Page 18: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/18.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
Contribution #2 ELMo embeddings
learn logic rules without explicit supervision
![Page 19: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/19.jpg)
Neural Nets + Logic Rules?Method Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
Contribution #2 ELMo embeddings
learn logic rules without explicit supervision
Contribution #3 Robustness of explicit / implicit methods to varying
annotator agreement in A-but-B sentences
![Page 20: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/20.jpg)
Digression: Reproducibility
![Page 21: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/21.jpg)
Digression: Reproducibility
Small benchmark datasets (SST, MR, CR)
Significant variation in performance every run (due to random initialization / GPU parallelization)
![Page 22: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/22.jpg)
Digression: Reproducibility
Small benchmark datasets (SST, MR, CR)
Significant variation in performance every run (due to random initialization / GPU parallelization)
Solution :- Average performance over a large number of random seeds (Reimers and Gurevych 2017)
![Page 23: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/23.jpg)
Large Variation (100 seeds)
![Page 24: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/24.jpg)
Large Variation (100 seeds)
![Page 25: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/25.jpg)
Large Variation (100 seeds)
![Page 26: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/26.jpg)
OutlineMethod Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
Contribution #2 ELMo embeddings
learn logic rules without explicit supervision
Contribution #3 Robustness of explicit / implicit methods to varying
annotator agreement in A-but-B sentences
![Page 27: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/27.jpg)
Model in Hu et al. 2016Expectation-Maximization style algorithm
E: Projection (Ganchev et al. 2010)
M: Distillation (Hinton et al. 2014)
![Page 28: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/28.jpg)
E: Projection (Ganchev et al. 2010)
![Page 29: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/29.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
![Page 30: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/30.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
negative = 0.34 positive = 0.66
![Page 31: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/31.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
negative = 0.34 positive = 0.66
project
![Page 32: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/32.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
negative = 0.34 positive = 0.66
negative = 0.77 positive = 0.23
project
![Page 33: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/33.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
negative = 0.34 positive = 0.66
negative = 0.77 positive = 0.23
project
projection is a convex optimization problem
![Page 34: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/34.jpg)
E: Projection (Ganchev et al. 2010)
this movie is funny, but horribly directed
negative = 0.34 positive = 0.66
negative = 0.77 positive = 0.23
project
new distribution consistent with logic rulesprojection is a convex optimization problem
![Page 35: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/35.jpg)
M: Distillation (Hinton et al. 2014)
train model with projected distribution as soft-label
![Page 36: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/36.jpg)
Hu et al. 2016 algorithmE: ProjectionM: Distillation
forall minibatch (x,y) {p = forward(x)q = project(p)theta += grad-update(p, q, y)
}
![Page 37: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/37.jpg)
Conclusions in Hu et al. 2016
1) Distilled model better than single projection
2) Distilled neural network has significant gain on SST2 as it learns A-but-B rule
![Page 38: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/38.jpg)
Our Conclusions1) Distilled model better than single projection
2) Distilled neural network has significant gain on SST2 as it learns A-but-B rule
1) A single projection is a good way to explicitly encode logic rules
2) Distilled neural nets aren't learning logic rules
![Page 39: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/39.jpg)
Distillation is ineffective
0
0.75
1.5
2.25
3
Distillation Single Projection Distill + Single Project
Reported Averaged
![Page 40: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/40.jpg)
Single Projection good!
0
0.75
1.5
2.25
3
Distillation Single Projection Distill + Single Project
Reported Averaged
![Page 41: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/41.jpg)
0
0.75
1.5
2.25
3
Distillation Single Projection Distill + Single Project
Reported Averaged
Single Projection sufficient!
![Page 42: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/42.jpg)
0
0.75
1.5
2.25
3
Distillation Single Projection Distill + Single Project
Reported Averaged
Single Projection sufficient!
![Page 43: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/43.jpg)
Consistent Trend on A-but-B
Reported N / A
Averaged Gain %
Distillation = 1.9% Single Projection = 9.3%
Distillation + Projection = 8.9%
Again, a single projection at test time is sufficient!
![Page 44: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/44.jpg)
Our Conclusions1) Distilled model better than single projection
2) Distilled neural network has significant gain on SST2 as it learns A-but-B rule
1) A single projection is a good way to explicitly encode logic rules
2) Distilled neural nets aren't learning logic rules
![Page 45: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/45.jpg)
OutlineMethod Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
Contribution #2 ELMo embeddings
learn logic rules without explicit supervision
Contribution #3 Robustness of explicit / implicit methods to varying
annotator agreement in A-but-B sentences
![Page 46: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/46.jpg)
ELMo RepresentationsEmbeddings from Language Models
![Page 47: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/47.jpg)
ELMo RepresentationsEmbeddings from Language Models
large language model trained on the 1 Billion Words dataset
![Page 48: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/48.jpg)
ELMo RepresentationsEmbeddings from Language Models
large language model trained on the 1 Billion Words dataset
learnt representations used for downstream task
![Page 49: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/49.jpg)
ELMo RepresentationsEmbeddings from Language Models
large language model trained on the 1 Billion Words dataset
learnt representations used for downstream task
Unlike word2vec, these embeddings are contextual
![Page 50: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/50.jpg)
ELMo Results (100 seeds)
![Page 51: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/51.jpg)
ELMo Results (100 seeds)Model SST2 A-but-B A-but-B +
negation
CNN (Baseline) 86.0 78.7 80.1
CNN + ELMo 88.9 86.5 87.2
Gain % 2.9 7.8 7.1
![Page 52: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/52.jpg)
ELMo Results (100 seeds)
Significant improvement, even after averaging!
Model SST2 A-but-B A-but-B + negation
CNN (Baseline) 86.0 78.7 80.1
CNN + ELMo 88.9 86.5 87.2
Gain % 2.9 7.8 7.1
![Page 53: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/53.jpg)
60% of the improvement is on A-but-B sentences and negations
Is ELMo Learning Logic Rules?Model SST2 A-but-B A-but-B +
negation
CNN (Baseline) 86.0 78.7 80.1
CNN + ELMo 88.9 86.5 87.2
Gain % 2.9 7.8 7.1
![Page 54: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/54.jpg)
60% of the improvement is on A-but-B sentences and negations
(Only 24.5% of corpus is A-but-B / negations)
Is ELMo Learning Logic Rules?Model SST2 A-but-B A-but-B +
negation
CNN (Baseline) 86.0 78.7 80.1
CNN + ELMo 88.9 86.5 87.2
Gain % 2.9 7.8 7.1
![Page 55: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/55.jpg)
ELMo + Explicit (Projection)Model SST2 A-but-B
ELMo 88.9 86.5
ELMo + project 89.0 87.2
Gain % 0.1 0.7
![Page 56: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/56.jpg)
Test-time projection is ineffective for ELMo
ELMo + Explicit (Projection)Model SST2 A-but-B
ELMo 88.9 86.5
ELMo + project 89.0 87.2
Gain % 0.1 0.7
![Page 57: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/57.jpg)
Test-time projection is ineffective for ELMo
ELMo + Explicit (Projection)
Distance between ELMo distribution and projected distribution is 0.13 (vs 0.26 distillation, 0.27 baseline)
Model SST2 A-but-B
ELMo 88.9 86.5
ELMo + project 89.0 87.2
Gain % 0.1 0.7
![Page 58: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/58.jpg)
Clustering ELMo Vectors
Cosine similarity between every pair of words
![Page 59: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/59.jpg)
Clustering ELMo Vectors
Cosine similarity between every pair of words
Contrastive (A-but-B) there are slow and repetitive parts, but it has
just enough spice to keep it interesting
![Page 60: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/60.jpg)
there are slow and repetitive parts, but it has just enough spice to keep it interesting
![Page 61: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/61.jpg)
word2vec ELMo
there are slow and repetitive parts, but it has just enough spice to keep it interesting
![Page 62: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/62.jpg)
word2vec ELMoClustering for A part and B part in A-but-B
sentences for ELMo embeddings
there are slow and repetitive parts, but it has just enough spice to keep it interesting
![Page 63: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/63.jpg)
word2vec ELMoClustering for A part and B part in A-but-B
sentences for ELMo embeddings
there are slow and repetitive parts, but it has just enough spice to keep it interesting
![Page 64: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/64.jpg)
ELMo Representations learn the scope of a contrastive conjunction!
word2vec ELMo
![Page 65: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/65.jpg)
OutlineMethod Previous Work Our Contributions
Explicit Hu et al. (ACL 2016)
Contribution #1 replication study finds incorrect conclusions
Implicit? Peters et al. (NAACL 2018)
Contribution #2 ELMo embeddings
learn logic rules without explicit supervision
Contribution #3 Robustness of explicit / implicit methods to varying
annotator agreement in A-but-B sentences
![Page 66: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/66.jpg)
Sentiment is Ambiguous!
beautiful film, but those who have read the book will be disappointed
![Page 67: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/67.jpg)
Sentiment is Ambiguous!
nine crowd-workers label each A-but-B sentence as positive / negative / neutral
beautiful film, but those who have read the book will be disappointed
![Page 68: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/68.jpg)
Sentiment is Ambiguous!
nine crowd-workers label each A-but-B sentence as positive / negative / neutral
we test our models on subsets of varying agreement
beautiful film, but those who have read the book will be disappointed
![Page 69: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/69.jpg)
Consistent trends on all levels of agreement
![Page 70: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/70.jpg)
Projection degrades accuracy on high agreement sentences!
![Page 71: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/71.jpg)
Key Takeaways• Carefully perform sentiment classification research
• variation across runs - average across several seeds
• ambiguous sentences - benchmark on subsets of varying annotator agreement
• ELMo embeddings implicitly learn logic rules for sentiment classification
Code + Data github.com/martiansideofthemoon/logic-rules-sentiment
![Page 72: Encoding Logic Rules in Sentiment ... - The Last Martianmartiansideofthemoon.github.io/assets/emnlp-2018.pdf · this movie is funny, but horribly directed negative = 0.34 positive](https://reader034.vdocument.in/reader034/viewer/2022042911/5f41d2751e08550923520764/html5/thumbnails/72.jpg)
Key Takeaways• Carefully perform sentiment classification research
• variation across runs - average across several seeds
• ambiguous sentences - benchmark on subsets of varying annotator agreement
• ELMo embeddings implicitly learn logic rules for sentiment classification
Thank You!
Code + Data github.com/martiansideofthemoon/logic-rules-sentiment