joint unsupervised learning of deep representations and...

Joint Unsupervised Learning of Deep Representations and Image Clusters Jianwei Yang, Devi Parikh, Dhruv Batra Virginia Tech IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview a) Meaningful clusters can provide supervisory signals to learn image representations. Intuitions Recurrent Framework Methodology Divide T timesteps into P partially unrolled periods. A forward pass and backward pass are implemented at each period. Goal Analysis and Visualizations https://github.com/jwyang/joint-unsupervised-learning MNIST-test Initial (1762) Middle (17) Final (10) COIL-20 Initial (421) Middle (42) Final (20) Cluster Labels CNN Parameters Agglomerative Clustering CNN Training (backprop) Cluster and learn deep representations for unlabeled images 1-nearest neighbor classification error for different methods on MNIST test set. PCA Autoencoder Parametric t-SNE Visualizing MNIST test set in 2D. The first three figures are copied from param. t-SNE paper. Ours Objective Function Quantitative Results Forward pass: merge clusters based on local structure (right), and conventional merging strategy (left). Proposed recurrent framework for unsupervised learning of deep representations and image clusters. Backward pass: learn representation to further reduce dissimilarity among samples in merged clusters. Test datasets and methods Information. Testing generalization of our learnt (unsupervised) representation to LFW face verification. Evaluation on CIFAR-10 classification Minimizing overall loss for T time steps: Loss in forward pass in period p (merge clusters): pass Loss in backward pass in period p (train CNN): pass Approximated weighted triplet-loss in backward pass: pass b) Good representations help to get meaningful clusters. e) Cluster and learn rep. iteratively & progressively d) Learn rep. first, and then cluster image based on that c) Cluster first, and then learn representations (rep.) Iterative optimization Our clustering performance vs. that of existing clustering approaches using raw image data. Clustering performance using our representation fed to existing clustering algorithms. Back-propagate loss Images The clusters are more accurate and representations are more discriminative from initial stage to final stage. Average 1.97% error reduction. Our approach can be potentially used as a visualization tool. The KNN purity is significantly improved compared with raw image data (see arrows). This explains the quantitative improvements of our method. , arg min (, | ) y Ly I arg min ( | ,) y Ly I arg min ( | ,) L yI Average 0.3% lower than supervised method 1.96% over K-means Average +21.5% on NMI +22.2% on AC +14.1% on NMI The stats below are average across all datasets. See paper for details Metric: Normalized Mutual Info. (NMI) Clustering Accuracy (AC) This triplet-loss can be optimized via backprop. Visualization of clusters and learned representations at different learning stages. KNN class purity of raw image data (left) and our learned representations (right). Cluster Image Learn Representation Average +25.7% on AC +6.43% on NMI; +12.76% on AC to best performance of existing approaches averaged over all datasets

Upload: others

Post on 19-Jun-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Joint Unsupervised Learning of Deep Representations and ...jyang375/Jianwei_Yang_files/cvpr16-jule... · Joint Unsupervised Learning of Deep Representations and Image Clusters Jianwei

Joint Unsupervised Learning of Deep Representations and Image ClustersJianwei Yang, Devi Parikh, Dhruv Batra

Virginia Tech

IEEE 2016 Conference on

Computer Vision and Pattern

Recognition

Overview

a) Meaningful clusters can provide supervisory signals to

learn image representations.

Intuitions

Recurrent Framework

Methodology

Divide T timesteps into P partially unrolled periods. A forward pass and backward pass

are implemented at each period.

Goal

Analysis and Visualizations

https://github.com/jwyang/joint-unsupervised-learning

MNIST-test

Initial (1762) Middle (17) Final (10)

COIL-20

Initial (421) Middle (42) Final (20)

Cluster Labels

CNN Parameters

Agglomerative Clustering

CNN Training (backprop)

Cluster and learn deep representations for unlabeled images

1-nearest neighbor classification error for different methods on MNIST test set.

PCA Autoencoder

Parametric t-SNE

Visualizing MNIST test set in 2D. The first three figures are copied from param. t-SNE paper.

Ours

Objective Function Quantitative Results

Forward pass: merge clusters based

on local structure (right), and

conventional merging strategy (left).

Proposed recurrent framework for unsupervised learning of

deep representations and image clusters.

Backward pass: learn representation

to further reduce dissimilarity among

samples in merged clusters.

Test datasets and methods Information.

Testing generalization of our learnt (unsupervised) representation to LFW face verification.

Evaluation on CIFAR-10 classification

Minimizing overall loss for T time steps:

Loss in forward pass in period p (merge clusters): pass

Loss in backward pass in period p (train CNN): pass

Approximated weighted triplet-loss in backward pass:

pass

b) Good representations help to get meaningful clusters.

e) Cluster and learn rep. iteratively & progressively

d) Learn rep. first, and then cluster image based on that

c) Cluster first, and then learn representations (rep.)

Iterative optimization

Our clustering performance vs. that of existing clustering approaches using raw image data.

Clustering performance using our representation fed to existing clustering algorithms.

Back-propagate loss

Images

The clusters are more accurate and representations are more discriminative

from initial stage to final stage.

Average 1.97% error reduction.

Our approach can be potentially

used as a visualization tool.

The KNN purity is significantly improved compared with raw image data

(see arrows). This explains the quantitative improvements of our method.

arg min ( , | )y

L y I

arg min ( | , )y

L y I

arg min ( | , )L y I

Average 0.3% lower than supervised method 1.96% over K-means

Average +21.5% on NMI

+22.2% on AC

+14.1% on NMIThe stats below are average across all datasets. See paper for details

Metric:

Normalized Mutual Info. (NMI)

Clustering Accuracy (AC)

This triplet-loss can be optimized via backprop.

Visualization of clusters and learned representations at different learning stages.

KNN class purity of raw image data (left) and our learned representations (right).

Cluster Image

Learn Representation

Average +25.7% on AC

+6.43% on NMI; +12.76% on AC to best performance

of existing approaches averaged over all datasets

UNSUPERVISED FEATURE LEARNING VIA SPARSE ...web.eecs.umich.edu/~honglak/thesis_final.pdfUNSUPERVISED FEATURE LEARNING VIA SPARSE HIERARCHICAL REPRESENTATIONS A DISSERTATION SUBMITTED

L15 how CNNs BrandonRohrer - GitHub Pages · Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh

1 Unsupervised learning of visual representations and their use in object & face recognition Gary Cottrell Chris Kanan Honghao Shan Lingyun Zhang Matthew

Unsupervised learning of hierarchical representations with ...web.eecs.umich.edu/~honglak/cacm2011-researchHighlights-convDBN.pdf · Hierarchical Representations with Convolutional

Unsupervised Learning of Dense Visual Representations · 2020. 12. 9. · Unsupervised Learning of Dense Visual Representations Pedro O. Pinheiro1, Amjad Almahairi, Ryan Y. Benmalek2,

Knowledgeable Independent Focused · Source: Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations, Honglak Lee, Roger Grosse, Rajesh

UNSUPERVISED LEARNING IN COMPUTER VISION · Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations. Proceedings of the 26th International

“Unsupervised Learning of Visual Representations by ...yjlee/teaching/ecs289g-fall2016/chenshan.pdfRepresentations by Solving ... - Problem: Self-supervised learning CNN - Solution:

Unsupervised Learning of Discriminative Attributes and ... · PDF fileUnsupervised Learning of Discriminative Attributes and Visual Representations Chen Huang1,2 Chen Change Loy1,3

Deep Learning of Representations for Unsupervised and Transfer Learning

SECOMO Documentation · Convolutional Deep Belief Networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26 th International Confefence on

Joint Unsupervised Learning of Deep Representations and ...jw2yang/Jianwei_Yang_files/cvpr16-jule... · Joint Unsupervised Learning of Deep Representations and Image Clusters Jianwei

UNSUPERVISED ROBOTIC SORTING TOWARDS AUTONOMOUS DECISION ... · encoders combined with ensemble clustering to generate feature representations suitable for clustering. As of today,

M-theory - LCSLlcsl.mit.edu/ldr-workshop/Slides/Poggio_LDR_MIT_112313.pdf · 2014-04-14 · M-theory: unsupervised learning of hierarchical invariant representations tomaso poggio

PVEs: Position-Velocity Encoders for Unsupervised Learning of … · 2017-07-25 · PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations

Unsupervised Learning of Latent ... - ppn.csail.mit.eduppn.csail.mit.edu/papers/ppn_uai.pdf · prediction network (PPN). Consisting of a per-ception module that extracts representations

HoloGAN: Unsupervised Learning of 3D Representations From ...storage.googleapis.com/theis.io/publications/... · learning of 3D representations from images, enabling direct manip-ulation

Unsupervised Learning of Dense Hierarchical … · Unsupervised Learning of Dense Hierarchical Appearance Representations ... the hierarchy, ... NBP, each vertex receives

Learning Physical Graph Representations from …Recently, machine learning researchers have developed deep generative models for unsupervised segmentation by capturing relations between

Challenging Common Assumptions in the Unsupervised Learning … · 2018-11-30 · Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations Francesco

Self-Supervised Learning of Face Representations for Video Face Clustering · 2020-05-21 · 17.05.19 FG_Best: Dattaet al.: Unsupervised learning of face representations. In FG. IEEE,

CONVOLUTIONAL DEEP BELIEF NETWORKSdasgupta/254-deep/emanuele.pdf · Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations probabilistic

Challenging Common Assumptions in the Unsupervised ... · ﬁrst theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without

Learning Semantic Representations for Unsupervised Domain ...proceedings.mlr.press/v80/xie18c/xie18c.pdfadaptation methods are based on generative adversarial net-works (GAN) (Goodfellow

Learning unsupervised feature representations for single ... · RESEARCH ARTICLE Learning unsupervised feature representations for single cell microscopy images with paired cell inpainting

Unsupervised Learning of Visual Representations by Solving ... · Representations by Solving Jigsaw Puzzles Mehdi Noroozi and Paolo Favaro Presented by : Ehsan Amiri . Ehsan Amiri

Unsupervised Sentence Embedding Using Document Structure ...Keywords: sentence embedding document structure out-of-vocabulary 1 Introduction Distributed representations of words and

Bildverarbeitung beim Laserstrahlschweißen · [2] Lee et al. - Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations [3] Springenberg

Unsupervised learning of hierarchical representations with

Convolutional*Deep*Belief*Networks*for*Scalable ...swadhin/reading_group/... · Convolutional*Deep*Belief*Networks*for*Scalable*Unsupervised*Learning*of*Hierarchical*Representations

Unsupervised Learning of Visual Representations using Videosnitish/depth_oral.pdf · This information is valuable because temporal ordering is closely related to semantic structure

Visualizing and Understanding Deep Texture Representationssmaji/papers/texture-cvpr16.pdf · Visualizing and Understanding Deep Texture Representations Tsung-Yu Lin University of

RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger

Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and

Unsupervised Learning Learning Unsupervised...Unsupervised Learning and Data Mining Learning Mining Unsupervised Learning and Data Mining Unsupervised Data Clustering Supervised Learning