translations between modalities learning robust joint ... · 10 learning joint representations: 2...
TRANSCRIPT
1
Presenter: Hai PhamHai Pham*, Paul Pu Liang*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
Found in Translation:Learning Robust Joint Representations by Cyclic
Translations Between Modalities
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
2
Progress of Artificial Intelligence
Multimedia Content Intelligent Personal Assistants
Robots andVirtual Agents
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
3
Multimodal Language Modalities
➢ Gestures
➢ Body language
➢ Eye contact
➢ Facial expressions
Language Visual
Acoustic
➢ Lexicon
➢ Syntax
➢ Pragmatics
➢ Prosody
➢ Vocal expressions
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
4
Multimodal Language Modalities
➢ Gestures
➢ Body language
➢ Eye contact
➢ Facial expressions
Language Visual
Acoustic
➢ Lexicon
➢ Syntax
➢ Pragmatics
➢ Prosody
➢ Vocal expressions
➢ Anger➢ Disgust➢ Fear➢ Happiness➢ Sadness➢ Surprise
Emotion
Personality➢ Confidence➢ Persuasion➢ Passion
Sentiment➢ Positive➢ Negative
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
5
Challenge 1: Intra-modal Interactions
“This movie is great”
Smile
Intr
a-m
od
al
Speaker’s behaviors Sentiment Intensity
time
time
Head nod
a) Temporal sequences
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
6
Challenge 2: Cross-modal Interactions
“This movie is great”
Smile
Cro
ss-m
od
al
Speaker’s behaviors Sentiment Intensity
Loud voice
time
a) Multiple co-occurring interactionsb) Different weighted combinations
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
7
Learning Joint Representations: 2 modalities
Traditional MethodsVisual Modality
Today was a great day!Language Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
8
Learning Joint Representations: 2 modalities
Traditional MethodsVisual Modality
Today was a great day!Language Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Joint Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
9
Learning Joint Representations: 2 modalities
Traditional MethodsVisual Modality
Today was a great day!Language Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Joint Representation
Sentiment Prediction
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
10
Learning Joint Representations: 2 modalities
Traditional MethodsVisual Modality
Today was a great day!Language Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Joint Representation
Sentiment Prediction
Both modalities required at test time!Sensitive to missing/noisy visual modality.
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
11
Learning Robust Joint Representations: 2 modalities
Bimodal Cyclic TranslationsJoint Representation Visual Modality
Today was a great day!Language Modality forward forward
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
12
Learning Robust Joint Representations: 2 modalities
Bimodal Cyclic TranslationsJoint Representation Visual Modality
Today was a great day!Language Modality forward
backward
forward
backward
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
13
Learning Robust Joint Representations: 2 modalities
Bimodal Cyclic TranslationsJoint Representation Visual Modality
Today was a great day!Language Modality forward
backward
forward
backward
Sentiment Prediction
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
14
Learning Robust Joint Representations: 2 modalities
Bimodal Cyclic TranslationsJoint Representation Visual Modality
Today was a great day!Language Modality forward
backward
forward
backward
Sentiment Prediction
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Only language modality required at test time!
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
15
Learning Robust Joint Representations: 3 modalities
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Visual Modality
Today was a great day!Language Modality
Trimodal Cyclic Translations
Acoustic Modality
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
16
Learning Robust Joint Representations: 3 modalities
Visual Modality
Today was a great day!Language Modality forward
Trimodal Cyclic Translations
backward
forward
backward
Acoustic Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
17
Learning Robust Joint Representations: 3 modalities
Joint Representation
Visual Modality
Today was a great day!Language Modality forward
Trimodal Cyclic Translations
backward
forward
backward
Acoustic Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
18
Learning Robust Joint Representations: 3 modalities
Joint Representation
Visual Modality
Today was a great day!Language Modality forward
Trimodal Cyclic Translations
backward
forward
backward
Sentiment Prediction
Acoustic Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
19
Learning Robust Joint Representations: 3 modalities
Joint Representation
Visual Modality
Today was a great day!Language Modality forward
Trimodal Cyclic Translations
backward
forward
backward
Sentiment Prediction
Acoustic Modality
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Only language modality required at test time!
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
20
Cyclic Translations
[Zhu*, Park*, Isola, and Efros, ICCV 2017]
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
21
Multimodal Cyclic Translation Network
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Source
Target
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
22
Multimodal Cyclic Translation Network
Source
Target
ForwardTranslation
1
Encoder RNN
Hai Pham Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesFound in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
23
Multimodal Cyclic Translation Network
Source
Target
ForwardTranslation
1
Encoder RNN
Embedded Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
24
Multimodal Cyclic Translation Network
Source
Target Forward
Translation
ForwardTranslation
1
2
Encoder RNN
Decoder RNN
Embedded Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
25
Multimodal Cyclic Translation Network
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
1
2
Encoder RNN
Decoder RNN
Embedded Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
26
Multimodal Cyclic Translation Network
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
1
23
Encoder RNN
Decoder RNN
Embedded Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
27
Multimodal Cyclic Translation Network
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
1
23
4
Encoder RNN
Decoder RNN
Embedded Representation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
28
Multimodal Cyclic Translation Network
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment Prediction
1
23
4
5
Encoder RNN
Decoder RNN
Embedded Representation Prediction RNN
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
29
Multimodal Cyclic Translation Network
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment
Sentiment Prediction
1
23
4
5
Encoder RNN
Decoder RNN
Embedded Representation Prediction RNN
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
30
Coupled Translation-Prediction Objective
➢ Forward translation loss
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment Prediction
1
23
4
5
Encoder RNN
Decoder RNN
Embedded Representation Prediction RNN
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
31
Coupled Translation-Prediction Objective
➢ Forward translation loss
➢ Cycle consistent loss Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment Prediction
1
23
4
5
Encoder RNN
Decoder RNN
Embedded Representation Prediction RNN
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
32
Coupled Translation-Prediction Objective
➢ Forward translation loss
➢ Cycle consistent loss
➢ Prediction loss
Source
Target
Seq2Seq
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment Prediction
1
23
4
5
Encoder RNN
Decoder RNN
Embedded Representation Prediction RNN
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
33
Hierarchical Multimodal Cyclic Translation Network
Source
Target1
Target2
34
Hierarchical Multimodal Cyclic Translation Network
Source
ForwardTranslation
ForwardTranslation
Target1
1
2
Encoder RNN
Decoder RNN
Embedded Representation
Target2
35
Hierarchical Multimodal Cyclic Translation Network
Source
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Target1
1
23
4
Encoder RNN
Decoder RNN
Embedded Representation
Target2
36
Hierarchical Multimodal Cyclic Translation Network
Source
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Seq2Seq 1
ForwardTranslation
Seq2Seq 2
Target2
Target1
1
23
4
5
Encoder RNN
Encoder RNN
Decoder RNN
Embedded Representation
Embedded Representation
37
Hierarchical Multimodal Cyclic Translation Network
Source
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Seq2Seq 1
ForwardTranslation
Seq2Seq 2
Target2
Target1
ForwardTranslation
1
23
4
5
6
Encoder RNN
Encoder RNN
Decoder RNN
Decoder RNN
Embedded Representation
Embedded Representation
38
Hierarchical Multimodal Cyclic Translation Network
Source
ForwardTranslation
ForwardTranslation
Backward Translation
Backward Translation
Sentiment
Seq2Seq 1
ForwardTranslation
Seq2Seq 2
Target2
SentimentPredictionTarget1
ForwardTranslation
1
23
4
5
6
7
Prediction RNN
Encoder RNN
Encoder RNN
Decoder RNN
Decoder RNN
Embedded Representation
Embedded Representation
39
1. Non-temporal models: SVM (Cortes and Vapnik, 1995), DF (Nojavanasghari et al., 2016)
2. Early fusion: EF-LSTM (Hochreiter and Schmidhuber, 1997), EF-RHN (Zilly et al., 2016)
3. Late fusion: LMF (Liu et al., 2018), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017)
4. Multi-view learning: MV-LSTM (Rajagopalan et al., 2016)
5. Memory-based models: MARN, MFN (Zadeh et al., 2018)
6. Multi-stage model: RMFN (Liang et al., 2018)
Baseline Models
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
40
MCTN: Only language modality required at test time!
State-of-the-art Results: CMU-MOSI
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
41
State-of-the-art Results: ICT-MMMO and YouTube
MCTN: Only language modality required at test time!
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
42
Bimodal Variations
Enc
Dec
Enc
next input
Enc
Dec
Enc
Dec
MCTN Bimodal
[ ]Dec
Step 1 Step 2 Seq2Seq 1 Seq2Seq 2
concat
Simple Bimodal
No-Cycle Bimodal
Double Bimodal
Test: use of cyclic translations, modality ordering, and hierarchical structure
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
43
Bimodal Variations Results
1. Use cyclic translations2. Use language as source modality3. Share parameters in seq2seq models
44
Trimodal Variations
Enc
Dec
EncEnc
Dec
Simple Trimodal
Double Trimodal
MCTN Trimodal
[ ]Enc
Dec
Concat Trimodal
Dec
concat
Enc
Paired Trimodal
[ ]concat
Dec2Dec1
Test: use of cyclic translations, modality ordering, and hierarchical structure
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
45
Trimodal Variations Results
1. Use hierarchical translations2. Use cyclic translations3. Use language as source modality4. Share parameters in seq2seq models
46
Adding More Modalities
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
47
Adding More Modalities
Bimodal MCTN without
cyclic translation
Trimodal MCTN with
cyclic translation
Bimodal MCTN with
cyclic translation
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
48
Thank you for your attention!
Email: [email protected]: @hai_t_pham
Email: [email protected]: @pliang279
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Code: https://github.com/hainow/MCTN/
Email: [email protected]: @Tom_Manzini