Multimodal Machine Learning

Main GoalDefine a common taxonomy for multimodal machine

learning and provide an overview of research in this area

Introduction: Preliminary Terms

Modality: the way in which something happens or is experienced

Multimodal machine learning (MML): building models that process and relate information from multiple modalities

History of MMLAudio-Visual Speech Recognition (AVSR)

Multimedia content indexing and retrieval

Multimodal interaction

Media Description

● McGurk effect● Visual information

improved performance when the speech signal was noisy

● Searching visual and multimodal content directly

● Understanding human multimodal behaviors (facial expressions, speech, etc.) during social interactions

● Image captioning● Challenging

problem to evaluate

Five Main Challenges of MML1. Representation – representing and summarizing multimodal data

2. Translation – mapping from one modality to another (e.g., image captioning)

3. Alignment – identifying the corresponding elements between modalities (e.g., recipe steps to the correct video frame)

4. Fusion – joining information from multiple modalities to predict (e.g., using lip motion and speech to predict spoken words)

5. Co-learning – transferring knowledge between modalities, their representation, and their predictive models

These challenges need to be tackled for the field to progress.

RepresentationMultimodal representation: a representation of data using information from multiple entities (an image, word/sentence, audio sample, etc.)

We need to represent multimodal data in a meaningful way to have good models.

This is challenging because multimodal data are heterogeneous.

Joint

Representation

Coordinated

Example constraints: minimize cosine similarity, maximize correlation

Joint RepresentationMostly used when multimodal data is present during training and inference

Methods:

● Simple concatenation● Neural networks● Probabilistic graphical models● Sequential representation

Neural networks are often pre-trained using an autoencoder on unsupervised data.

Coordinated RepresentationSimilarity Models Structured Coordinated Space Models

Enforce similarity between representations by minimizing the distance between modalities in the coordinated space

Enforce additional constraints between modalities

Example: cross-modal hashing. Additional constraints are:

● N-dimensional Hamming space● The same object from different modalities has

to have a similar hash code● Similarity-preserving

“dog”

Translation: Mapping from one modality to another (e.g., image captioning)

Example-based

Use a dictionary to translate between modalities

Generative

Construct a model that translates between modalities

Example-Based TranslationCombination-Based Retrieval-Based

Combines retrievals from the dictionary in a meaningful way to create a better translation

Rules are often hand-crafted or heuristic.

Use retrieved translation without modification

Problem: Often requires an extra processing step (e.g., re-ranking of retrieved translations) – similarity in the unimodal space does not always mean a good translation

Solution: Use an intermediate semantic space for similarity comparison.Performs better because the space reflects both modalities and allows for bi-directional translation.Requires manual construction or learning of the space, which needs large training dictionaries.

Generative TranslationConstructing models that perform multimodal translation on a unimodal source

Grammar-Based Encoder-Decoder Continuous Generation

Detect high-level concepts from source and generate a target using a pre-defined grammar

More likely to generate logically correct targets

Formulaic translations, need complex pipelines for concept detection

Example: video description of who did what to whom and where and how

Encode the source modality into a latent representation, then decode that representation into the target modality (one pass)

Encoders: RNNs, DBNs, CNNsDecoders: RNNs, LSTMs

May be memorizing the dataRequire lots of data for training

Generate target modality at every timestep based on a stream of source modality inputs

HMMs, RNNs, encoder-decoders

Requires the ability to understand the source and generate the target

Translation Evaluation: A Major ChallengeThere are often multiple correct translations.

Evaluation methods

● Human evaluation – impractical and biased● BLEU, ROUGE, Meteor, CIDEr – low correlation to human judgment, require

a high number of reference translations● Retrieval – better reflects human judgments

○ Rank the available captions and assess if the correct captions get a high rank

● Visual question-answering for image captioning – ambiguity in questions and answers, question bias

Alignment“Finding relationships and correspondences between sub-components of instances from two or more modalities.”

Examples:

● Given an image and caption, find the areas of the image corresponding to the caption.

● Given a movie, align it to the book chapters it was based on.

Explicit Alignment (unsupervised and supervised)Unsupervised: no direct alignment labels. Supervised: direct alignment labels.

Most approaches inspired from work on statistical machine translation and genome sequences.

If there is no similarity metric between modalities, canonical correlation analysis (CCA) is used to map the modalities to a shared space.

CCA finds the linear combinations of data that maximizes their correlation

Example applications:

● Spoken words ↔ visual objects in images● Movie shots and scenes ↔ screenplay● Recipes ↔ cooking videos● Speakers ↔ videos● Sentences ↔ video frames● Image regions ↔ phrases● Speakers in audio ↔ locations in video● Objects in 3D scenes ↔ nouns in text

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

Implicit AlignmentUsed as an intermediate step for another task

Does not rely on supervised alignment examples

Data is latently aligned during model training

Useful for speech recognition, machine translation, media description, visual question-answering

Example: alignment of words and image regions before performing image retrieval based on text descriptions

Difficulties in alignment:

● Few datasets with explicitly annotated alignments

● Difficult to design similarity metrics● May exist 0, 1, or many correct alignments

FusionEarly fusion - features integrated immediately (concatenation)

Late fusion - each modality makes an independent decision (averaging, voting schemes, weighted combinations, other ensemble techniques)

Hybrid fusion - exploits advantages of both

Fusion TechniquesMultiple kernel learning (MKL):

● An extension of kernel support vector machines

● Kernels function as similarity functions between data

● Modality-specific kernels allows for better fusion

MKL Application: performing musical artist similarity ranking from acoustic, semantic, and social view data.

(McFee et al., Learning Multi-modal Similarity)

Neural networks (RNN/LSTM) can learn the multimodal representation and fusion component end-to-end. They achieve good performance but require large datasets and are less interpretable.

LSTM Applications:

● Audio-visual emotion classification● Neural image captioning

Co-learningModeling a resource poor modality by exploiting a resource rich modality.

Used to address lack of annotated data, noisy data, and unreliable labels.

Can generate more labeled data, but also can lead to overfitting.

Co-learning examplesTransfer learning application: using text to improve visual representations for image classification by coordinating CNN features with word2vec features

Conceptual grounding: learning meanings/concepts based on vision, sound, or smell (not just on language)

Zero-short learning (ZSL): recognizing a class without having seen a labeled example of it

ZSL Example: using an intermediate semantic space to predict unseen words people are thinking about from fMRI data

Zero-Shot Learning with Semantic Output Codes

Grounding Semantics in Olfactory Perception

“This work opens up interesting possibilities in analyzing smell and even taste. It could be applied in a variety of settings beyond semantic similarity, from chemical information retrieval to metaphor interpretation to cognitive modelling. A speculative blue-sky application based on this, and other multi-modal models, would be an NLG application describing a wine based on its chemical composition, and perhaps other information such as its color and country of origin.”

Paper CritiqueThis paper is very thorough in its survey of MML challenges and what researchers have done to approach them.

MML is central to the advancement of AI; thus, this area must be studied in order to make progress.

Future research directions include any MML projects that make headway in the five challenge areas.

Questions?