Methods for Understanding How

Deep Neural Networks Work

Dr. Wojciech Samek

Head of Machine Learning Group

Fraunhofer Heinrich Hertz Institute

Vision Industry and Technology Forum 6th Sep 2017, Hamburg, Germany

Unbeatable AI Systems

AlphaGo beats Go

human champ

Deep Net outperforms humans

in image classification

Deep Net beats human at

recognizing traffic signs

DeepStack beats

professional poker players

Computer out-plays

humans in "doom"

Autonomous search-and-rescue

drones outperform humans

IBM's Watson destroys

humans in jeopardy

Elon Musk (2017): “AI will be able to beat humans at

EVERYTHING by 2030.”

Unbeatable AI Systems

You may heard such promises before

H. A. Simon (1965): "Machines will be capable, within

twenty years, of doing any work a man can do."

M. Minsky (1970): "In from three to eight years we will

have a machine with the general intelligence of an

average human being."

Computing power

Very large deep neural networks Information (implicit)

Solve task

Huge volumes of data

Past promises did not become reality, but today …

650,000 neurons, 60 million parameters

Deep Neural Networks

Why are these models so successful ?

- “end-to-end” training

- feature hierarchy

- distributed / reusable representation

Deep Neural Networks: “End-to-End” Training

“Slippery road”

Traditional Approaches

input feature extraction classification output

Deep Learning

“Slippery road”

input feature extraction + classification output

(Source: Yann LeCun

Deep Neural Networks: Feature Hierarchy

Low level

features

Mid level

features

High level

featuresClassifier

“Car”

Hierarchical information processing in the brain

Deep Neural Networks: Feature Hierarchy

(Source: Simon Thorpe)

Deep Neural Networks: Reusable Representation

Features extracted from “car” images can be used for

other tasks / classification of other objects.

objects with

roundish shape

Very large deep neural networks Information (implicit)

What did the neural network learn ?

How does it solve the problem ?

Can we extract human interpretable information ?

Do we understand the AI ?

Can we trust these “end-to-end” trained black box algorithms ?

Black-Box Systems

(More information: https://neil.fraser.name/writing/tank/)

Already in the 1980s neural networks have been used for

classification tasks, e.g., identify tanks in the forest.

Training Dataset:

- 100 photos of tanks hiding behind trees

- 100 photos of trees with no tanks.

First results were promising, but did

network solve the task correctly ?

Understand “weaknesses” of classifier

Detect biases / bring in human intuition.

Learn from the learning machine

“I've never seen a human play this move.” (Fan Hui)

Wrong decisions can be harmful and costly.

Verify that system works as expected

Interpretability in the sciences

The “why” often more important than the prediction.

Compliance to legislation

“right to explanation”, retain human decision …

We need to “open” Black-Box Systems

“rooster”

We developed a general method to explain

individual classification decisions.

Main idea:

Opening the Black-Box

Layer-wise Relevance Propagation (LRP)(Bach et al., PLOS ONE, 2015)

“measure how much each pixel

contributes to the overall prediction”

Classification

cat

rooster

dog

Opening the Black-Box

Explanation

cat

rooster

dog

=

Initialization

Idea: Backpropagate “relevances”

Opening the Black-Box

Opening the Black-Box

Explanation

cat

rooster

dog

?

Intuition: Redistribute relevance proportionally

- a neuron gets more relevance if it’s more activated

- more relevance flows over strong connections

Opening the Black-Box

Explanation

cat

rooster

dog

Theoretical interpretation: Deep Taylor Decomposition(Montavon et al., Pattern Recognition, 2017)

[number]: explanation target class

red color: evidence for prediction

blue color: evidence against prediction

what speaks for / against

classification as “3”

what speaks for / against

classification as “9”

Opening the Black-Box

Opening the Black-Box

Application: Compare Classifiers

Two classifiers

- similar classification accuracy on horse class

- but do they solve the problem similarly ?

(Lapuschkin et al., IEEE CVPR, 2016)

Application: Compare Classifiers

Images from

PASCAL VOC

2007 dataset

Interpretability helps to

- understand biases / flaws in the data and weaknesses of the classifier

- verify that system works as expected

Application: Compare Classifiers

GoogleNet focuses on

faces of animal.

—> suppresses background noise

(Binder et al., ICML Visualization

Workshop, 2016)

Interpretability helps to

- compare and select models / architectures

Application: Measure Context Use

classifier

how important

is context ?

how important

is context ?

relevance outside bbox

relevance inside bbox

importance

of context=

(Lapuschkin et al., IEEE CVPR, 2016)

Image Fisher Vector DNN

boat

boat

Co

nte

xt u

se

airplane

airplane

sofachair chair sofa

Application: Measure Context Use

Interpretability helps to

- extract additional classification-related information

Application: Video Analysis

Motion vectors can be extracted

from the compressed video

-> allows very efficient analysis

Application: Video Analysis

Interpretability helps to

- extract additional classification-related information

- compare features

Other Applications

Identifying age-related features

(Arbabzadah et al., GCPR, 2016)

(Sturm et al., J Neuroscience Methods, 2016)

Brain-Computer Interfacing

Identifying relevant words in text

(Arras et al., PLOS ONE, 2017)

Detection of morphing attacks

(Seibold et al., IWDW, 2017)

In many problems interpretability as important as prediction

(trusting a black-box system may not be an option).

Use in practice

- verify predictions, detect biases and flaws, debug models

- compare and select architectures, understand and improve models

- extract additional information, perform further tasks

We have a powerful, mathematically well-founded method to explain

individual predictions of complex machine learning models.

More research needed on how to compare and evaluate all the different

aspects of interpretability.

Summary

Thank you for your attention

For more information, check out our tutorial paper:

Montavon et al. “Methods for Interpreting and Understanding Deep Neural Networks”

https://arxiv.org/abs/1706.07979