1

Combining Generative and DiscriminativeRepresentation Learning for Lung CT Analysis with

Convolutional Restricted Boltzmann MachinesGijs van Tulder and Marleen de Bruijne

AbstractThe choice of features greatly influences the per-formance of a tissue classification system. Despite this, manysystems are built with standard, predefined filter banks thatare not optimized for that particular application. Representationlearning methods such as restricted Boltzmann machines mayoutperform these standard filter banks because they learn afeature description directly from the training data. Like manyother representation learning methods, restricted Boltzmannmachines are unsupervised and are trained with a generativelearning objective; this allows them to learn representationsfrom unlabeled data, but does not necessarily produce featuresthat are optimal for classification. In this paper we propose theconvolutional classification restricted Boltzmann machine, whichcombines a generative and a discriminative learning objective.This allows it to learn filters that are good both for describingthe training data and for classification. We present experimentswith feature learning for lung texture classification and airwaydetection in CT images. In both applications, a combinationof learning objectives outperformed purely discriminative orgenerative learning, increasing, for instance, the lung tissueclassification accuracy by 1 to 8 percentage points. This showsthat discriminative learning can help an otherwise unsupervisedfeature learner to learn filters that are optimized for classification.

Index TermsRepresentation learning, Restricted Boltzmannmachine, Deep learning, Machine learning, Segmentation, Patternrecognition and classification, Neural network, Lung, X-rayimaging and computed tomography.

I. INTRODUCTION

Most methods for automated image classification do notwork directly with image data, but first extract a higher-level description of useful features from the image. Thechoice of features determines a large part of the classificationperformance. Which features work well depends on the natureof the classification problem: for example, some problemsrequire features that preserve and extract scale differences,whereas other problems require features that are invariant tothose properties. Often, feature representations are based onstandard filter banks of common feature descriptors, such asGaussian derivatives that detect edges in the image. These

Copyright c 2016 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending a request to pubs-permissions@ieee.org.

This research is financed by the Netherlands Organization for ScientificResearch (NWO).

G. van Tulder and M. de Bruijne are with the Biomedical Imaging Group,Erasmus MC, Rotterdam, The Netherlands. M. de Bruijne is also with theDepartment of Computer Science, University of Copenhagen, Denmark.

Code used for the experiments is available as supplementary material andat http://vantulder.net/code/2016/tmi-ccrbm/.

predefined filter banks are not specifically optimized for aparticular problem or dataset.

As an alternative to such predefined feature sets, represen-tation learning or feature learning methods [1] learn a high-level representation directly from the training data. Becausethis representation is learned from the training data, it can beoptimized to give a better description of the data. Using thislearned representation as the input for a classification systemmight give a better classification performance than using ageneric set of features.

Most feature learning methods use unsupervised modelsthat are trained with unlabeled data. While this can be anadvantage because it makes it easier to create a large trainingset, it can also lead to suboptimal results for classification,because the features that these methods learn are not nec-essarily useful to discriminate between classes. Unsupervisedfeature learning tends to learn features that model the strongestvariations in the data, while classifiers need features thatdiscriminate between classes. If the variation between samplesfrom the same class is much stronger than the variationbetween classes, feature learning probably produces featuresthat capture primarily within-class variation. If those featuresdo not represent enough between-class variation, they mightgive a lower classification performance.

This issue of within-class variation is relevant for manyapplications, including medical image analysis. For example,in disease classification, the differences between patients areoften greater than the subtle differences between disease pat-terns. As a result, representation learners might learn featuresthat model these between-patient differences, rather than thosethat improve classification.

In this paper we study the restricted Boltzmann machine(RBM), a popular representation learning model, as a wayto learn features that are optimized for classification. Thestandard RBM does not include labels and is trained with anunsupervised, generative learning objective. The classificationRBM [2], an extension of the standard RBM, does includelabel information and can also be trained with a discriminativelearning objective. This discriminative learning objective opti-mizes the classification performance of the classification RBM.The generative and discriminative objectives can be combinedto learn discriminative features that represent the data and areuseful for classification.

We propose the convolutional classification RBM, whichcombines the classification RBM with the convolutional RBM,another extension of the standard RBM. The convolutional

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.The final version of record is available at http://dx.doi.org/10.1109/TMI.2016.2526687

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

http://vantulder.net/code/2016/tmi-ccrbm/

2

RBM [3][6] uses the convolutional weight-sharing patternfrom convolutional networks to learn small filters that areapplied to every position in a larger image. This weight sharingmakes learning more efficient and allows the RBM to modelsmall features that occur in multiple areas of an image, whichis useful for describing textures.

The ability to use both generative and discriminative learn-ing objectives distinguishes the classification RBM from manyother representation learning methods. Unsupervised modelssuch as the standard RBM are usually trained with only agenerative learning objective. Supervised representation learn-ing methods, such as convolutional neural networks [7], areusually trained with only a discriminative learning objective.The classification RBM can be trained with a generativeobjective, a discriminative objective, or a combination.

We present experiments on lung tissue classification andairway detection. For the lung tissue classification experimentswe used a dataset on interstitial lung diseases (ILD) [8]with CT images of 73 patients. Previously published tissue-classification experiments on this dataset used wavelets [9][12], local binary patterns [13], [14], bag-of-visual-words [15],[16], filter banks derived from the discrete Fourier transform[17], RBMs [18], [19] and convolutional networks [20].

We used RBMs to learn features for lung tissue classifica-tion. From the images, we first extracted 2D patches that weused to train RBMs with different mixtures of discriminativeand generative learning. Using the RBM-learned representa-tions, we trained and evaluated classifiers that classify eachpatch in one of the five tissue classes. We compared thoseresults with those of two standard filter banks.

We expected the effect of discriminative learning to becomeless important for larger representations (more hidden nodesin the RBM), because larger representations are more likely tocontain sufficient discriminative features even without explicitdiscriminative learning. To study this effect, we performedairway detection experiments on lung CT images from theDanish Lung Cancer Screening Trial (DLCST) [21]. We usednon-convolutional classification RBMs with different mixturesof discriminative and generative learning to learn featuresfor this dataset. The non-convolutional RBMs allowed us toexperiment with larger numbers of hidden nodes.

This paper extends our earlier workshop paper [22] in whichwe introduced the convolutional classification RBM and foundthat using a mixture of generative and discriminative learningobjectives can produce features that improve classificationresults. In this paper, we present the results of more extensiveexperiments that confirm these preliminary conclusions.

The rest of this paper is organized as follows. Section IIgives a brief overview of other relevant representation learningapproaches. Section III describes the RBM and its learningalgorithm. Section IV introduces the datasets and the ex-periments. Section V describes the results. We end with adiscussion and conclusion.

II. RELATED WORK

Representation learning methods have been used for tissueclassification in lung CT before. In experiments similar to

those presented in this paper and using the same ILD dataset,Li et al. [18] used RBMs to extract features. Whereas we useclassification RBMs with convolution to learn small filters,Li et al. trained standard (non-convolutional) RBMs on smallsubpatches extracted from the patch that is to be classified. Inlater work [19] on the same dataset, Li et al. reported that con-volutional neural networks gave a slightly better perf