1

Found in Translation:Learning Robust Joint Representations by

Cyclic Translations Between Modalities

Paul Pu LiangJoint work with: Hai Pham, Thomas Manzini, Louis-Philippe Morency, Barnabas Poczos

2

Robust Representation Learning

“Thisisunbelievable!”

Speaker’sbehaviors

Loud

lang

uage

visu

alac

oust

ic

multimodalrepresentation

predictionnoisy/missing

at test time

3

Learning Joint Representations: 2 modalities

Traditional MethodsVisual Modality

Today was a great day!Language Modality

Joint Representation

Prediction

Both modalities required at test time!Sensitive to missing/noisy visual modality.

4

Learning Robust Joint Representations: 2 modalities

Bimodal Cyclic TranslationsJoint Representation Visual Modality

Today was a great day!Language Modality forward

backward

forward

backward

Prediction

Only language modality required at test time!Robust to noisy/missing visual modalities.