Thoracic Disease Identification and Localization with Limited Supervision

Zhe Li1∗, Chong Wang3, Mei Han2∗, Yuan Xue3, Wei Wei3, Li-Jia Li3, Li Fei-Fei31Syracuse University, 2PingAn Technology, US Research Lab, 3Google Inc.

1zli89@syr.edu, 2ex-hanmei001@pingan.com.cn,3{chongw, yuanxue, wewei, lijiali, feifeili}@google.com

Abstract

Accurate identification and localization of abnormali-ties from radiology images play an integral part in clini-cal diagnosis and treatment planning. Building a highlyaccurate prediction model for these tasks usually requiresa large number of images manually annotated with labelsand finding sites of abnormalities. In reality, however, suchannotated data are expensive to acquire, especially the oneswith location annotations. We need methods that can workwell with only a small amount of location annotations. Toaddress this challenge, we present a unified approach thatsimultaneously performs disease identification and local-ization through the same underlying model for all images.We demonstrate that our approach can effectively leverageboth class information as well as limited location annota-tion, and significantly outperforms the comparative refer-ence baseline in both classification and localization tasks.

1. IntroductionAutomatic image analysis is becoming an increasingly

important technique to support clinical diagnosis and treat-ment planning. It is usually formulated as a classificationproblem where medical imaging abnormalities are identi-fied as different clinical conditions [27, 4, 28, 30, 36]. Inclinical practice, visual evidence that supports the classifi-cation result, such as spatial localization [2] or segmenta-tion [37, 40] of sites of abnormalities is an indispensablepart of clinical diagnosis which provides interpretation andinsights. Therefore, it is of vital importance that the imageanalysis method is able to provide both classification resultsand the associated visual evidence with high accuracy.

Figure 1 is an overview of our approach. We focus onchest X-ray image analysis. Our goal is to both classifythe clinical conditions and identify the abnormality loca-tions. A chest X-ray image might contain multiple sitesof abnormalities with monotonous and homogeneous imagefeatures. This often leads to the inaccurate classification ofclinical conditions. It is also difficult to identify the sites of

∗This work was done when Zhe Li and Mei Han were at Google.

Chest X-ray ImageLocalization for Cardiomegaly

Unified diagnosis network

Atelectasis: 0.7189 Cardiomegaly: 0.8573 Consolidation:0.0352 Edema:0.0219...... Pathology

Diagnosis

Figure 1. Overview of our chest X-ray image analysis network forthoracic disease diagnosis. The network reads chest X-ray imagesand produces prediction scores and localization for the diseases

abnormalities because of their variances in the size and lo-cation. For example, as shown in Figure 2, the presentationof “Atelectasis” (alveoli are deflated down) is usually lim-ited to local regions of a lung [11] but possible to appearanywhere on both sides of lungs; while “Cardiomegaly”(enlarged heart) always covers half of the chest and is al-ways around the heart.

The lack of large-scale datasets also stalls the advance-ment of automatic chest X-ray diagnosis. Wang et al.provides one of the largest publicly available chest x-raydatasets with disease labels1 along with a small subsetwith region-level annotations (bounding boxes) for evalua-tion [31]2. As we know, the localization annotation is muchmore informative than just a single disease label to improvethe model performance as demonstrated in [21]. However,getting detailed disease localization annotation can be diffi-cult and expensive. Thus, designing models that can workwell with only a small amount of localization annotation isa crucial step for the success of clinical applications.

In this paper, we present a unified approach that simul-taneously improves disease identification and localizationwith only a small amount of X-ray images containing dis-ease location information. Figure 1 demonstrates an exam-ple of the output of our model. Unlike the standard objectdetection task in computer vision, we do not strictly predictbounding boxes. Instead, we produce regions that indicatethe diseases, which aligns with the purpose of visualizing

1While abnormalities, findings, clinical conditions, and diseases havedistinct meanings in the medical domain, here we simply refer them asdiseases and disease labels for the focused discussion in computer vision.

2The method proposed in [31] did not use the bounding box informa-tion for localization training.

1

arX

iv:1

711.

0637

3v6

[cs

.CV

] 2

0 Ju

n 20

18

Cardiomegaly Atelectasis

Figure 2. Examples of chest X-ray images with the disease bound-ing box. The disease regions are annotated in the yellow boundingboxes by radiologists.

and interpreting the disease better. Firstly, we apply a CNNto the input image so that the model learns the informationof the entire image and implicitly encodes both the classand location information for the disease [24]. We then slicethe image into a patch grid to capture the local informationof the disease. For an image with bounding box annota-tion, the learning task becomes a fully supervised problemsince the disease label for each patch can be determined bythe overlap between the patch and the bounding box. Foran image with only a disease label, the task is formulatedas a multiple instance learning (MIL) problem [3]—at leastone patch in the image belongs to that disease. If there isno disease in the image, all patches have to be disease-free.In this way, we have unified the disease identification andlocalization into the same underlying prediction model butwith two different loss functions.

We evaluate the model on the aforementioned chest X-ray image dataset provided in [31]. Our quantitative resultsshow that the proposed model achieves significant accuracyimprovement over the published state-of-the-art on both dis-ease identification and localization, despite the limited num-ber of bounding box annotations of a very small subset ofthe data. In addition, our qualitative results reveal a strongcorrespondence between the radiologist’s annotations anddetected disease regions, which might produce further in-terpretation and insights of the diseases.

2. Related WorkObject detection. Following the R-CNN work [9], re-

cent progresses has focused on processing all regions withonly one shared CNN [12, 8], and on eliminating explicitregion proposal methods by directly predicting the bound-ing boxes. In [25], Ren et al. developed a region proposalnetwork (RPN) that regresses from anchors to regions ofinterest (ROIs). However, these approaches could not beeasily used for images without enough annotated boundingboxes. To make the network process images much faster,Redmon et al. proposed a grid-based object detection net-work, YOLO, where an image is partitioned into S×S gridcells, each of which is responsible to predict the coordinatesand confidence scores of B bounding boxes [24]. The clas-sification and bounding box prediction are formulated intoone loss function to learn jointly. A step forward, Liu et al.partitioned the image into multiple grids with different sizesproposing a multi-box detector overcoming the weaknessin YOLO and achieved better performance [22]. Similarly,

these approaches are not applicable for the images withoutbounding boxes annotation. Even so, we still adopt the ideaof handling an image as a group of grid cells and treat eachpatch as a classification target.

Medical disease diagnosis. Zhang et al. proposed adual-attention model using images and optional texts tomake accurate prediction [35]. In [36], Zhang et al. pro-posed an image-to-text model to establish a direct mappingfrom medical images to diagnostic reports. Both modelswere evaluated on a dataset of bladder cancer images andcorresponding diagnostic reports. Wang et al. took advan-tage of a large-scale chest X-ray dataset to formulate thedisease diagnosis problem as multi-label classification, us-ing class-specific image feature transformation [31]. Theyalso applied a thresholding method to the feature map visu-alization [34] for each class and derived the bounding boxfor each disease. Their qualitative results showed that themodel usually generated much larger bounding box thanthe ground-truth. Hwang et al. [17] proposed a self-transferlearning framework to learn localization from the globallypooled class-specific feature maps supervised by image la-bels. These works have the same essence with class activa-tion mapping [38] which handles natural images. The lo-cation annotation information was not directly formulatedinto the loss function in the none of these works. Featuremap pooling based localization did not effectively capturethe precise disease regions.

Multiple instance learning. In multiple instance learn-ing (MIL), an input is a labeled bag (e.g., an image) withmany instances (e.g., image patches) [3]. The label is as-signed at the bag level. Wu et al. assumed each imageas a dual-instance example, including its object proposalsand possible text annotations [32]. The framework achievedconvincing performance in vision tasks including classifica-tion and image annotation. In medical imaging domain, Yanet al. utilized a deep MIL framework for body part recogni-tion [33]. Hou et al. first trained a CNN on image patchesand then an image-level decision fusion model by patch-level prediction histograms to generate the image-level la-bels [15]. By ranking the patches and defining three typesof losses for different schemes, Zhu et al. proposed an end-to-end deep multi-instance network to achieve mass classi-fication for whole mammogram [39]. We are building anend-to-end unified model to make great use of both imagelevel labels and bounding box annotations effectively.

3. Model

Given images with disease labels and limited boundingbox information, we aim to design a unified model that si-multaneously produces disease identification and localiza-tion. We have formulated two tasks into the same under-lying prediction model so that 1) it can be jointly trainedend-to-end and 2) two tasks can be mutually beneficial. Theproposed architecture is summarized in Figure 3.

2

Image w/o bounding boxes

Image w/ bounding boxes

CNN

ResNet

Conv Features

ℎ′ × ×

ℎ × ×

Conv Features

× ×

Bilinear Interpolation

(a1,b2) (a2,b2)

(a1,b1) (a2,b1)

(a,b)

Max Pooling

5 81 634

72

69

54

76

87

87

9 99 9

7 7 8

Padding=0, Stride=1

(a)

Patch Slicing

(b)

× × ∗ × ×

Recognition Network

Patch Scores

Conv

Conv

Patch Features

(c)

Score

Label Prediction

Train

Annotated for k (Eq. 1 )

Non-annotated for k(Eq. 2 )

Test

Any image for k(Eq. 2 )

Pat

(d)

Infiltration

Formulation

......

Figure 3. Model overview. (a) The input image is firstly processed by a CNN. (b) The patch slicing layer resizes the convolutional featuresfrom the CNN using max-pooling or bilinear interpolation. (c) These regions are then passed to a fully-convolutional recognition network.(d) During training, we use multi-instance learning assumption to formulate two types of images; during testing, the model predicts bothlabels and class-specific localizations. The red frame represents the ground truth bounding box. The green cells represent patches withpositive labels, and brown is negative. Please note during training, for unannotated images, we assume there is at least one positive patchand the green cells shown in the figure are not deterministic.

3.1. Image modelConvolutional neural network. As shown in Fig-

ure 3(a), we use the residual neural network (ResNet) ar-chitecture [13] given its dominant performance in ILSVRCcompetitions [26]. Our framework can be easily extended toany other advanced CNN models. The recent version of pre-act-ResNet [14] is used (we call it ResNet-v2 interchange-ably in this paper). After removing the final classificationlayer and global pooling layer, an input image with shapeh×w× c produces a feature tensor with shape h′×w′× c′where h, w, and c are the height, width, and number ofchannels of the input image respectively while h′ = h

32 ,w′ = w

32 , c′ = 2048. The output of this network encodesthe images into a set of abstracted feature maps.

Patch slicing. Our model divides the input image intoP × P patch grid, and for each patch, we predict K binaryclass probabilities, where K is the number of possible dis-ease types. As the CNN gives c′ input feature maps withsize of h′ × w′, we down/up sample the input feature mapsto P×P through a patch slicing layer shown in Figure 3(b).Please note that P is an adjustable hyperparameter. In thisway, a node in the same spatial location across all the fea-ture maps corresponds to one patch of the input image. Weupsample the feature maps If their sizes are smaller thanexpected patch grid size. Otherwise, we downsample them.

Upsampling. We use a simple bilinear interpolation toupsample the feature maps to the desired patch grid size. Asinterpolation is, in essence, a fractionally stridden convolu-tion, it can be performed in-network for end-to-end learningand is fast and effective [23]. A deconvolution layer [34] isnot necessary to cope with this simple task.

Downsampling. The bilinear interpolation makes sensefor downsampling only if the scaling factor is close to 1. Weuse max-pooling to down sample the feature maps. In gen-

eral cases, the spatial size of the output volume is a functionof the input width/height (w), the filter (receptive field) size(f ), the stride (s), and the amount of zero padding used (p)on the border. The output width/height (o) can be obtainedby w−f+2p

s + 1. To simplify the architecture, we set p = 0and s = 1, so that f = w − o+ 1.

Fully convolutional recognition network. We follow[23] to use fully convolution layers as the recognition net-work. Its structure is shown in Figure 3(c). The c′ resizedfeature maps are firstly convolved by 3 × 3 filters into asmaller set of feature maps with c∗ channels, followed bybatch normalization [18] and rectified linear units (ReLU)[10]. Note that the batch normalization also regularizes themodel. We set c∗ = 512 to represent patch features. Theabstracted feature maps are then passed through a 1×1 con-volution layer to generate a set of P × P final predictionswith K channels. Each channel gives prediction scores forone class among all the patches, and the prediction for eachclass is normalized by a logistic function (sigmoid function)to [0, 1]. The final output of our network is the P × P ×Ktensor of predictions. The image-level label prediction foreach class in K is calculated across P × P scores, which isdescribed in Section 3.2.

3.2. Loss function

Multi-label classification. Multiple disease types can beoften identified in one chest X-ray image and these diseasetypes are not mutually exclusive. Therefore, we define abinary classifier for each class/disease type in our model.The binary classifier outputs the class probability. Note thatthe binary classifier is not applied to the entire image, butto all small patches. We will show how this can translate toimage-level labeling below.

Joint formulation of localization and classification.

3

Since we intend to build K binary classifiers, we will ex-emplify just one of them, for example, class k. Note thatK binary classifiers will use the same features and onlydiffer in their last logistic regression layers. The ith im-age xi is partitioned into a setM of patches equally, xi =[xi1, xi2, ..., xim], where m = |M| = P × P

Images with annotated bounding boxes. As shown inFigure 3(d), suppose an image is annotated with class k anda bounding box. We denote n be the number of patchescovered by the bounding box, where n < m. Let this set beN . Each patch in the set N as positive for class k and eachpatch outside the bounding box as negative. Note that if apatch is covered partially by the bounding box of class k, westill consider it a positive patch for class k. The boundingbox information is not lost. For the jth patch in ith image,let pkij be the foreground probability for class k. Since allpatches have their labels, the probability of an image beingpositive for class k is defined as,

p(yk|xi,bboxki ) =∏

j∈N pkij ·

∏j∈M\N (1− pkij), (1)

where yk is the kth network output denoting whether animage is a positive example of class k. For example, for aclass other than k, this image is treated as the negative sam-ple without a bounding box. We define a patch as positive toclass k when it is overlapped with a ground-truth box, andnegative otherwise.

Images without annotated bounding boxes. If the ith im-age is labeled as class k without any bounding box, weknow that there must be at least one patch classified as kto make this image a positive example of class k. There-fore, the probability of this image being positive for class kis defined as the image-level score 3,

p(yk|xi) = 1−∏

j∈M(1− pkij). (2)

At test time, we calculate p(yk|xi) by Eq. 2 as the predictionprobability for class k.

Combined loss function. Note that p(yk|xi,bboxki ) andp(yk|xi) are the image-level probabilities. The loss functionfor class k can be expressed as minimizing the negative loglikelihood of all observations as follows,

Lk =− λbbox

∑i ηip(y

∗k|xi, bboxk

i ) log(p(yk|xi, bboxki ))

− λbbox

∑i ηi(1− p(y

∗k|xi, bboxk

i )) log(1− p(yk|xi, bboxki ))

−∑

i(1− ηi)p(y∗k|xi) log(p(yk|xi))

−∑

i(1− ηi)(1− p(y∗k|xi)) log(1− p(yk|xi)), (3)

where i is the index of a data sample, ηi is 1 when theith sample is annotated with bounding boxes, otherwise0. λbbox is the factor balancing the contributions from an-notated and unannotated samples. p(y∗k|xi) ∈ {0, 1} and

3Later on, we notice an similar definition [20] for this multi-instanceproblem. We argue that our formulation is in a different context of solv-ing classification and localization in a unified way for images with limitedbounding box annotation. Yet, this related work can be viewed as a suc-cessful validation of our multi-instance learning based formulation.

p(y∗k|xi,bboxki ) ∈ {0, 1} are the observed probabilities for

class k. Obviously, p(y∗k|xi,bboxki ) ≡ 1, thus equation 3

can be re-written as follows,

Lk =− λbbox

∑i ηi log(p(yk|xi, bbox

ki ))

−∑

i(1− ηi)p(y∗k|xi) log(p(yk|xi))

−∑

i(1− ηi)(1− p(y∗k|xi)) log(1− p(yk|xi)). (4)

In this way, the training is strongly supervised (per patch)by the given bounding box; it is also supervised by theimage-level labels if the bounding boxes are not available.

To enable end-to-end training across all classes, we sumup the class-wise loss to define the total loss as,

L =∑

k Lk.

3.3. Localization generationThe full model predicts a probability score for each patch

in the input image. We define a score threshold Ts to distin-guish the activated patches against the non-activated ones.If the probability score pkij is larger than Ts, we considerthe jth patch in the ith image belongs to the localizationfor class k. We set Ts = 0.5 in this work. Please notethat we do not predict strict bounding boxes for the regionsof disease—the combined patches representing the localiza-tion information can be a non-rectangular shape.

3.4. TrainingWe use ResNet-v2-50 as the image model and select the

patch slicing size from {12, 16, 20}. The model is pre-trained on the ImageNet 1000-class dataset [6] with Incep-tion [29] preprocessing method where the image is normal-ized to [−1, 1] and resized to 299 × 299. We initialize theCNN with the weights from the pre-trained model, whichhelps the model converge faster than training from scratch.During training, we also fine-tune the image model, as webelieve the feature distribution of medical images differsfrom that of natural images. We set the batch size as 5to load the entire model to the GPU, train the model with500k iterations of minibatch, and decay the learning rate by0.1 from 0.001 every 10 epochs of training data. We addL2 regularization to the loss function to prevent overfitting.We optimize the model by Adam [19] method with asyn-chronous training on 5 Nvidia P100 GPUs. The model isimplemented in TensorFlow [1].

Smoothing the image-level scores. In Eq. 1 and 2, thenotation

∏denotes the product of a sequence of probabil-

ity terms ([0, 1]), which often leads to the a product valueof 0 due to the computational underflow if m = |M| islarge. The log loss in Eq. 3 mitigates this for Eq. 1, but doesnot help Eq. 2, since the log function can not directly affectits product term. To mitigate this effect, we normalize thepatch scores pkij and 1− pkij from [0, 1] to [0.98, 1] to makesure the image-level scores p(yk|xi,bboxki ) and p(yk|xi)smoothly varies within the range of [0, 1]. Since we are

4

Disease Atelectasis Cardiomegaly Consolidation Edema Effusion Emphysema Fibrosisbaseline 0.70 0.81 0.70 0.81 0.76 0.83 0.79

ours 0.80± 0.00 0.87± 0.01 0.80± 0.01 0.88± 0.01 0.87± 0.00 0.91± 0.01 0.78± 0.02Disease Hernia Infiltration Mass Nodule Pleural Thickening Pneumonia Pneumothoraxbaseline 0.87 0.66 0.69 0.67 0.68 0.66 0.80

ours 0.77± 0.03 0.70± 0.01 0.83± 0.01 0.75± 0.01 0.79± 0.01 0.67± 0.01 0.87± 0.01

Table 1. AUC scores comparison with the reference baseline model. Results are rounded to two decimal digits for table readability. Boldvalues denote better results. The results for the reference baseline are obtained from the latest update of [31].

thresholding the image-level scores in the experiments, wefound this normalization works quite well. See supplemen-tary material for a more detailed discussion on this.

More weights on images with bounding boxes. InEq. 4, the parameter λbbox weighs the contribution from theimages with annotated bounding boxes. Since the amountof such images is limited, and if we treat them equally withthe images without bounding boxes, it often leads to worseperformance. We thus increase the weight for images withbounding boxes to λbbox = 5 by cross validation.

4. ExperimentsDataset and preprocessing. NIH Chest X-ray dataset

[31] consists of 112, 120 frontal-view X-ray images with14 disease labels (each image can have multi-labels). Theselabels are obtained by analyzing the associated radiologyreports. The disease labels are expected to have accuracyof above 90% [31]. We take the provided labels as ground-truth for training and evaluation in this work. Meanwhile,the dataset also contains 984 labelled bounding boxes for880 images by board-certified radiologists. Note that theprovided bounding boxes correspond to only 8 types ofdisease instances. We separate the images with providedbounding boxes from the entire dataset. Hence we have twosets of images called “annotated” (880 images) and “unan-notated” (111, 240 images).

We resize the original 3-channel images from 1024 ×1024 to 512 × 512 pixels for fast processing. The pixelvalues in each channel are normalized to [−1, 1]. We do notapply any data augmentation techniques.

4.1. Disease identification

We conduct a 5-fold cross-validation. For each fold,we have done two experiments. In the first one, we trainthe model using 70% of bounding-box annotated and 70%unannotated images to compare the results with the refer-ence model [31] (Table 1). To our knowledge, the refer-ence model has the published state-of-the-art performanceof disease identification on this dataset. In the second ex-periment, we explore two data ratio factors of annotatedand unannotated images to demonstrate the effectiveness ofthe supervision provided by the bounding boxes (Figure 4).We decrease the amount of images without bounding boxesfrom 80% to 0% by a step of 20%. And then for each ofthose settings, we train our model by adding 80% or noneof bounding-box annotated images. For both experiments,the model is always evaluated on the fixed 20% annotatedand unannotated images for this fold.

Evaluation metrics. We use AUC scores (the area un-der the ROC4 curve) to measure the performance of ourmodel [7]. A higher AUC score implies a better classifier.

Comparison with the reference model. Table 1 givesthe AUC scores for all the classes. We compare our resultswith the reference baseline fairly: we, as the reference, useImageNet pre-trained ResNet-50 5 , after which a convolu-tion layer follows; both works use 70% images for trainingand 20% for testing, and we also conduct a 5-fold cross-validation to show the robustness of our model.

Compared to the reference model, our proposed modelachieves better AUC scores for most diseases. The over-all improvement is remarkable and the standard errors aresmall. The large objects, such as “Cardiomegaly”, “Em-physema”, and “Pneumothorax”, are as well recognized asthe reference model. Nevertheless, for small objects like“Mass” and “Nodule”, the performance is significantly im-proved. Because our model slices the image into smallpatches and uses bounding boxes to supervise the trainingprocess, the patch containing small object stands out of allthe patches to represent the complete image. For “Hernia”,there are only 227 (0.2%) samples in the dataset. Thesesamples are not annotated with bounding boxes. Thus, thestandard error is relatively larger than other diseases.

Bounding box supervision improves classificationperformances. We consider using 0% annotated imagesas our own baseline (right groups in Figure 4). We use80% annotated images (left groups in Figure 4) to comparewith the our own baseline. We plot the mean performancefor the cross-validation in Figure 4, the standard errors arenot plotted but similar to the numbers reported in Table 1.The number of 80% annotated images is just 704, which isquite small compared to the number of 20% unannotatedimages (22, 248). We observe in Figure 4 that for almostall the disease types, using 80% annotated images to trainthe model improves the prediction performance (by com-paring the bars with the same color in two groups for thesame disease). For some disease types, the absolute im-provement is significant (> 5%). We believe that this isbecause all the disease classifiers share the same underly-ing image model; a better-trained image model using eightdisease annotations can improve all 14 classifiers’ perfor-mance. Specifically, some diseases, annotated and unanno-tated, share similar visual features. For example, “Consoli-dation” and “Effusion” both appear as fluid accumulation

4Here ROC is the Receiver Operating Characteristic, which measuresthe true positive rate (TPR) against the false positive rate (FPR) at variousthreshold settings (200 thresholds in this work).

5Using ResNet-v2 [14] shows marginal performance difference for ournetwork compared to ResNet-v1 [13] used in the reference baseline.

5

AU

C score

AU

C score

Unannotated 80% 60% 40% 20% 0%

AnnotatedAtelectasis Cardiomegaly Consolidation Edema Effusion Emphysema Fibrosis

Hernia Infiltration Mass Nodule Pleural Thickening Pneumonia PneumothoraxAnnotated

0.7958

0.7825

0.8741

0.8494

0.7954

0.7853

0.8821

0.8505

0.8668

0.8634

0.9050

0.8930 0.7837

0.7597

0.7839

0.7737

0.8543

0.8106

0.7960

0.7664

0.8707

0.8415

0.8637

0.8596

0.8945

0.8851 0.7657

0.7048

0.7686

0.7423

0.8461 0.7601

0.7830

0.7236

0.8576 0.7463

0.8520

0.8442

0.8910

0.8269 0.7334

0.6030

0.7337

0.7151

0.8086 0.7275

0.7199 0.6392

0.7677 0.6464

0.8261

0.8236

0.8262

0.7829 0.6631 0.5762

0.6761

0.8685

0.6674

0.7387

0.7611

0.7250

0.52500.5

0.6

0.7

0.8

0.9

1.0

80% 0% 80% 0% 80% 0% 80% 0% 80% 0% 80% 0% 80% 0%

0.7663 0.6782

0.7017

0.6596

0.8277

0.8070

0.7673

0.7231

0.7649

0.7539 0.6732

0.6621

0.8829

0.85270.7351

0.6070

0.6987

0.6538

0.8331

0.7819

0.7498

0.7059

0.7464

0.7177

0.6559

0.6534

0.8664

0.8308

0.6561 0.5462

0.6731

0.6415

0.8083

0.7493

0.7120

0.6691

0.7590

0.6966 0.6209

0.5902

0.8199

0.8070

0.6059 0.5155

0.6466

0.6342

0.7539

0.7117

0.6654

0.6335

0.7082

0.6591 0.5478

0.5225

0.7850

0.7680

0.5495

0.6180

0.6421

0.5731

0.5975

0.6066

0.7072

0.5

0.6

0.7

0.8

0.9

1.0

80% 0% 80% 0% 80% 0% 80% 0% 80% 0% 80% 0% 80% 0%

Figure 4. AUC scores for models trained using different data combinations. Training set: annotated samples,{left: 80% (704 images), right:0% (baseline, 0 images)} for each disease type; unannotated samples, {80% (88, 892), 60% (66, 744), 40% (44, 496), 20% (22, 248),0%(0)} from left to right for each disease type. The evaluation set is 20% annotated and unannotated samples which are not included inthe training set. No result for 0% annotated and 0% unannotated images. Using 80% annotated images and certain amount of unannotatedimages improves the AUC score compared to using the same amount of unannotated images (same colored bars in two groups for the samedisease), as the joint model benefits from the strong supervision of the tiny set of bounding box annotations.

in the lungs, but only “Effusion” is annotated. The fea-ture sharing enables supervision for “Effusion” to improve“Consolidation” performance as well.

Bounding box supervision reduces the demand of thetraining images. Importantly, it requires less unannotatedimages to achieve the similar AUC scores by using a smallset of annotated images for training. As denoted with redcircles in Figure 4, taking “Edema” as an example, using40% (44, 496) unannotated images with 80% (704) anno-tated images (45, 200 in total) outperforms the performanceof using only 80% (88, 892) unannotated images.

Discussion. Generally, decreasing the amount of unan-notated images (from left to right in each bar group) willdegrade AUC scores accordingly in both groups of 0% and80% annotated images. Yet as we decrease the amount ofunannotated images, using annotated images for traininggives smaller AUC degradation or even improvement. Forexample, we compare the “Cardiomegaly” AUC degrada-tion for two pairs of experiments: {annotated:80%, unanno-tated:80% and 20%} and {annotated:0%, unannotated:80%and 20%}. The AUC degradation for the first group isjust 0.07 while that for the second group is 0.12 (accuracydegradation from blue to yellow bar).

When the amount of unannotated images is reduced to0%, the performance is significantly degraded. Because un-der this circumstance, the training set only contains posi-tive samples for eight disease types and lacks the positivesamples of the other six. Interestingly, “Cardiomegaly”achieves the second best score (AUC = 0.8685, the sec-ond green bar in Figure 4) when only annotated imagesare trained. The possible reason is that the location of car-diomegaly is always fixed to the heart covering a large areaof the image and the feature distributions for enlarged heartsare similar to normal ones. Without unannotated samples,the model easily distinguishes the enlarged hearts from nor-

mal ones given supervision from bounding boxes. Whenthe model sees hearts without annotation, the enlarged onesare disguised and fail to be recognized. As more unan-notated samples are trained, the enlarged hearts are recog-nized again by image-level supervision (AUC from 0.8086to 0.8741).

4.2. Disease localizationSimilarly, we conduct a 5-fold cross-validation. For each

fold, we have done three experiments. In the first experi-ment, we investigate the importance of bounding box super-vision by using all the unannotated images and increasingthe amount of annotated images from 0% to 80% by the stepof 20% (Figure 5). In the second one, we fix the amount ofannotated images to 80% and increase the amount of unan-notated images from 0% to 100% by the step of 20% toobserve whether unannotated images are able to help an-notated images to improve the performance (Figure 6). Atlast, we train the model with 80% annotated images andhalf (50%) unannotated images to compare localization ac-curacy with the reference baseline [31] (Table 2). For eachexperiment, the model is always evaluated on the fixed 20%annotated images for this fold.

Evaluation metrics. We evaluate the detected regions(which can be non-rectangular and discrete) against the an-notated ground truth (GT) bounding boxes, using two typesof measurement: intersection over union ratio (IoU) and in-tersection over the detected region (IoR) 6. The localiza-tion results are only calculated for those eight disease typeswith ground truth provided. We define a correct localizationwhen either IoU > T(IoU) or IoR > T(IoR), where T(*) isthe threshold.

6Note that we treat discrete detected regions as one prediction region,thus IoR is analogous to intersection over the detected bounding box arearatio (IoBB).

6

Annotated

Unannotated

0.6299

0.8880

0.7831

0.9068 0.6963

0.2917

0.3057

0.4355

0.7175

0.9348

0.8582

0.9290 0.7183 0.4330

0.4656

0.5302

0.7432

0.9219

0.8993

0.9268

0.7693 0.4821

0.5255

0.6290

0.7648

0.9739

0.8965

0.9387

0.7780 0.4956

0.5869

0.6644

0.8134

0.9861

0.9167

0.9784 0.7849 0.4878

0.6373

0.6687

0.0

0.2

0.4

0.6

0.8

1.0

Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

0% : 100%

20% : 100%

40% : 100%

60% : 100%

80% : 100%

Figure 5. Disease localization accuracy using IoR where T(IoR)=0.1. Training set: annotated samples, {0% (0), 20% (176), 40% (352),60% (528), 80% (704)} from left to right for each disease type; unannotated samples, 100% (111, 240 images). The evaluation set is 20%annotated samples which are not included in the training set. For each disease, the accuracy is increased from left to right, as we increasethe amount of annotated samples, because more annotated samples bring more bounding box supervision to the joint model.

Annotated

Unannotated0.5279

0.9997 0.7531

0.8755

0.4524

0.1114

0.7858

0.4734

0.7238

0.9913

0.8744

0.9208 0.6736

0.2713

0.6436

0.6241

0.7228

0.9948

0.8916

0.9482 0.7315

0.3638

0.6505

0.6452

0.7672

0.9924

0.9003

0.9541 0.7611 0.4640

0.6172

0.6111

0.7568

0.9871

0.8960

0.9498 0.7003

0.5446

0.5581

0.6320

0.8134

0.9861

0.9167

0.9784 0.7849 0.4878

0.6373

0.6687

0.0

0.2

0.4

0.6

0.8

1.0

Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

80% : 0%

80% : 20%

80% : 40%

80% : 60%

80% : 80%

80% : 100%

Figure 6. Disease localization accuracy using IoR where T(IoR)=0.1. Training set: annotated samples, 80% (704 images); unannotatedsamples, {0% (0), 20% (22, 248), 40% (44, 496), 60% (66, 744), 80% (88, 892), 100% (111, 240)} from left to right for each diseasetype. The evaluation set is 20% annotated samples which are not included in the training set. Using annotated samples only can producea model which localizes some diseases. As the amount of unannotated samples increases in the training set, the localization accuracy isimproved and all diseases can be localized. The joint formulation for both types of samples enables unannotated samples to improve theperformance with weak supervision.

Bounding box supervision is necessary for localiza-tion. We present the experiments shown in Figure 5. Thethreshold is set as tolerable as T(IoR)=0.1 to show the train-ing data combination effect on the accuracy. Please referto the supplementary material for localization performancewith T(IoU)=0.1, which is similar to Figure 5. Even thoughthe amount of the complete set of unannotated images isdominant compared with the evaluation set (111, 240 v.s.176), without annotated images (the most left bar in eachgroup), the model fails to generate accurate localization formost disease types. Because in this situation, the model isonly supervised by image-level labels and optimized usingprobabilistic approximation from patch-level predictions.As we increase the amount of annotated images graduallyfrom 0% to 80% by the step of 20% (from left to right ineach group), the localization accuracy for each type is in-creased accordingly. We can see the necessity of boundingbox supervision by observing the localization accuracy in-crease. Therefore, the bounding box is necessary to provideaccurate localization results and the accuracy is positivelyproportional to the amount of annotated images. We havesimilar observations when T(*) varies.

More unannotated data does not always mean bet-ter results for localization. In Figure 6, when we fix theamount of annotated images and increase the amount ofunannotated ones for training (from left to right in eachgroup), the localization accuracy does not increase accord-ingly. Some disease types achieve very high accuracy (evenhighest) without any unannotated images (the most left barin each group), such as “Pneumonia” and “Cardiomegaly”.Similarly as described in the discussion of Section 4.1,unannotated images and too many negative samples degradethe localization performance for these diseases. All dis-

ease types experience an accuracy increase, a peak score,and then an accuracy fall (from orange to green bar in eachgroup). Therefore, with bounding box supervision, unan-notated images will help to achieve better results in somecases and it is not necessary to use all of them.

Comparison with the reference model. In each fold,we use 80% annotated images and 50% unannotated im-ages to train the model and evaluate on the other 20% an-notated images in each fold. Since we use 5-fold cross-validation, the complete set of annotated images has beenevaluated to make a relatively fair comparison with the ref-erence model. In Table 2, we compare our localization ac-curacy under varying T(IoU) with respect to the referencemodel in [31]. Please refer to the supplementary materialfor the comparison between our localization performanceand the reference model with varying T(IoR). Our modelpredicts accurate disease regions, not only for the easy taskslike “Cardiomegaly” but also for the hard ones like “Mass”and “Nodule” which have very small regions. When thethreshold increases, our model maintains a large accuracylead over the reference model. For example, when evalu-ated by T(IoU)=0.6, our “Cardiomegaly” accuracy is still73.42% while the reference model achieves only 16.03%;our “Mass” accuracy is 14.92% while the reference modelfails to detect any “Mass” (0% accuracy). In clinical prac-tice, a specialist expects as accurate localization as possibleso that a higher threshold is preferred. Hence, our modeloutperforms the reference model with a significant improve-ment with less training data. Please note that as we con-sider discrete regions as one predicted region, the detectedarea and its union with GT bboxs are usually larger than thereference work which generates multiple bounding boxes.Thus for some disease types like “Pneumonia”, when the

7

T(IoU) Model Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

0.1 ref. 0.69 0.94 0.66 0.71 0.40 0.14 0.63 0.38ours 0.71± 0.05 0.98± 0.02 0.87± 0.03 0.92± 0.05 0.71± 0.10 0.40± 0.10 0.60± 0.11 0.63± 0.09

0.2 ref. 0.47 0.68 0.45 0.48 0.26 0.05 0.35 0.23ours 0.53± 0.05 0.97± 0.02 0.76± 0.04 0.83± 0.06 0.59± 0.10 0.29± 0.10 0.50± 0.12 0.51± 0.08

0.3 ref. 0.24 0.46 0.30 0.28 0.15 0.04 0.17 0.13ours 0.36± 0.08 0.94± 0.01 0.56± 0.04 0.66± 0.07 0.45± 0.08 0.17± 0.10 0.39± 0.09 0.44± 0.10

0.4 ref. 0.09 0.28 0.20 0.12 0.07 0.01 0.08 0.07ours 0.25± 0.07 0.88± 0.06 0.37± 0.06 0.50± 0.05 0.33± 0.08 0.11± 0.02 0.26± 0.07 0.29± 0.06

0.5 ref. 0.05 0.18 0.11 0.07 0.01 0.01 0.03 0.03ours 0.14± 0.05 0.84± 0.06 0.22± 0.06 0.30± 0.03 0.22± 0.05 0.07± 0.01 0.17± 0.03 0.19± 0.05

0.6 ref. 0.02 0.08 0.05 0.02 0.00 0.01 0.02 0.03ours 0.07± 0.03 0.73± 0.06 0.15± 0.06 0.18± 0.03 0.16± 0.06 0.03± 0.03 0.10± 0.03 0.12± 0.02

0.7 ref. 0.01 0.03 0.02 0.00 0.00 0.00 0.01 0.02ours 0.04± 0.01 0.52± 0.05 0.07± 0.03 0.09± 0.02 0.11± 0.06 0.01± 0.00 0.05± 0.03 0.05± 0.03

Table 2. Disease localization accuracy comparison using IoU where T(IoU)={0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}.The bold values denote thebest results. Note that we round the results to two decimal digits for table readability. Using different thresholds, our model outperformsthe reference baseline in most cases and remains capability of localizing diseases when the threshold is big.

Mass

Infiltration

FibrosisCardiomegaly

Effusion Edema

Atelectasis Pneumothorax

Consolidation

Figure 7. Example localization visualization on the test images. The visualization is generated by rendering the final output tensor asheatmaps and overlapping on the original images. We list some thoracic diseases as examples. The left image in each pair is the originalchest X-ray image and the right one is the localization result. All examples are positive for corresponding labels. We also plot theground-truth bounding boxes in yellow on the results when they are provided in the dataset.

threshold is small, our result is not as good as the reference.

4.3. Qualitative results

Figure 7 shows exemplary localization results of the uni-fied diagnosis model. The localization enables the explain-ability of chest X-ray images. It is intuitive to see that ourmodel produces accurate localization for the diseases com-pared with the given ground-truth bounding boxes. Pleasenote for “Infiltration” (3th and 4th images in the 3rd row ofFigure 7), both sides of lungs for this patient are infiltrated.Since the dataset only has one bounding box for one diseaseper image, it misses annotating other bounding boxes for thesame disease. Our model gives the remedy. Even though theextra region decreases the IoR/IoU score in the evaluation,but in clinical practice, it provides the specialist with suspi-cious candidate regions for further examination. When thelocalization results have no ground-truth bounding boxes tocompare with, there is also a strong consistency betweenour results and radiological signs. For example, our modellocalizes the enlarged heart region (1st and 2nd images in

the 2nd row) which implies “Cardiomegaly”, and the lungperipheries is highlighted (5th and 6th images in the 2ndrow) implying “Fibrosis” which is in accordance with theradiological sign of the net-like shadowing of lung periph-eries. The “Edema” (7th and 8th images in the 1st row) and“Consolidation” (7th and 8th images in the 2nd row) are ac-curately marked by our model. “Edema” always appears inan area that is full of small liquid effusions as the exampleshows. “Consolidation” is usually a region of compressiblelung tissue that has filled with the liquid which appears as abig white area. The model successfully distinguishes bothdiseases which are caused by similar reason.

5. ConclusionWe propose a unified model that jointly models disease

identification and localization with limited localization an-notation data. This is achieved through the same underlyingprediction model for both tasks. Quantitative and qualita-tive results demonstrate that our method significantly out-performs the state-of-the-art algorithm.

8

References[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,

C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghe-mawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane,R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. War-den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor-Flow: Large-scale machine learning on heterogeneous sys-tems, 2015. Software available from tensorflow.org.

[2] A. Akselrod-Ballin, L. Karlinsky, S. Alpert, S. Hasoul,R. Ben-Ari, and E. Barkan. A region based convolutionalnetwork for tumor detection and classification in breastmammography. In International Workshop on Large-ScaleAnnotation of Biomedical Data and Expert Label Synthesis,pages 197–205. Springer, 2016.

[3] B. Babenko. Multiple instance learning: algorithms and ap-plications.

[4] X. Chen, Y. Xu, D. W. K. Wong, T. Y. Wong, and J. Liu.Glaucoma detection based on deep convolutional neural net-work. In Engineering in Medicine and Biology Society(EMBC), 2015 37th Annual International Conference of theIEEE, pages 715–718. IEEE, 2015.

[5] I. S. Committee et al. 754-2008 ieee standard for floating-point arithmetic. IEEE Computer Society Std, 2008, 2008.

[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.In Computer Vision and Pattern Recognition, 2009. CVPR2009. IEEE Conference on, pages 248–255. IEEE, 2009.

[7] T. Fawcett. An introduction to roc analysis. Pattern recogni-tion letters, 27(8):861–874, 2006.

[8] R. Girshick. Fast r-cnn. In Proceedings of the IEEE inter-national conference on computer vision, pages 1440–1448,2015.

[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-ture hierarchies for accurate object detection and semanticsegmentation. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 580–587,2014.

[10] X. Glorot, A. Bordes, and Y. Bengio. Deep sparse recti-fier neural networks. In Proceedings of the Fourteenth Inter-national Conference on Artificial Intelligence and Statistics,pages 315–323, 2011.

[11] B. A. Gylys and M. E. Wedding. Medical terminology sys-tems: a body systems approach. FA Davis, 2017.

[12] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid poolingin deep convolutional networks for visual recognition. InEuropean Conference on Computer Vision, pages 346–361.Springer, 2014.

[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn-ing for image recognition. In Proceedings of the IEEE con-ference on computer vision and pattern recognition, pages770–778, 2016.

[14] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings indeep residual networks. In European Conference on Com-puter Vision, pages 630–645. Springer, 2016.

[15] L. Hou, D. Samaras, T. M. Kurc, Y. Gao, J. E. Davis, andJ. H. Saltz. Patch-based convolutional neural network for

whole slide tissue image classification. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recogni-tion, pages 2424–2433, 2016.

[16] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten.Densely connected convolutional networks. In Proceed-ings of the IEEE conference on computer vision and patternrecognition, volume 1, page 3, 2017.

[17] S. Hwang and H.-E. Kim. Self-transfer learning forfully weakly supervised object localization. arXiv preprintarXiv:1602.01625, 2016.

[18] S. Ioffe and C. Szegedy. Batch normalization: Acceleratingdeep network training by reducing internal covariate shift. InInternational Conference on Machine Learning, pages 448–456, 2015.

[19] D. Kingma and J. Ba. Adam: A method for stochastic opti-mization. arXiv preprint arXiv:1412.6980, 2014.

[20] F. Liao, M. Liang, Z. Li, X. Hu, and S. Song. Evaluate themalignancy of pulmonary nodules using the 3d deep leakynoisy-or network. arXiv preprint arXiv:1711.08324, 2017.

[21] C. Liu, J. Mao, F. Sha, and A. L. Yuille. Attention correct-ness in neural image captioning. In AAAI, pages 4176–4182,2017.

[22] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector.In European conference on computer vision, pages 21–37.Springer, 2016.

[23] J. Long, E. Shelhamer, and T. Darrell. Fully convolutionalnetworks for semantic segmentation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recogni-tion, pages 3431–3440, 2015.

[24] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. Youonly look once: Unified, real-time object detection. In Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 779–788, 2016.

[25] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towardsreal-time object detection with region proposal networks. InAdvances in neural information processing systems, pages91–99, 2015.

[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,et al. Imagenet large scale visual recognition challenge.International Journal of Computer Vision, 115(3):211–252,2015.

[27] J. Shi, X. Zheng, Y. Li, Q. Zhang, and S. Ying. Multimodalneuroimaging feature learning with multimodal stacked deeppolynomial networks for diagnosis of alzheimer’s disease.IEEE journal of biomedical and health informatics, 2017.

[28] H.-C. Shin, K. Roberts, L. Lu, D. Demner-Fushman, J. Yao,and R. M. Summers. Learning to read chest x-rays: recur-rent neural cascade model for automated image annotation.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 2497–2506, 2016.

[29] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.Going deeper with convolutions. In Proceedings of theIEEE conference on computer vision and pattern recogni-tion, pages 1–9, 2015.

[30] J. Wang, H. Ding, F. Azamian, B. Zhou, C. Iribarren, S. Mol-loi, and P. Baldi. Detecting cardiovascular disease from

9

mammograms with deep learning. IEEE transactions onmedical imaging, 2017.

[31] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Sum-mers. Chestx-ray8: Hospital-scale chest x-ray database andbenchmarks on weakly-supervised classification and local-ization of common thorax diseases. In 2017 IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),pages 3462–3471. IEEE, 2017.

[32] J. Wu, Y. Yu, C. Huang, and K. Yu. Deep multiple instancelearning for image classification and auto-annotation. In Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 3460–3469, 2015.

[33] Z. Yan, Y. Zhan, Z. Peng, S. Liao, Y. Shinagawa, S. Zhang,D. N. Metaxas, and X. S. Zhou. Multi-instance deeplearning: Discover discriminative local anatomies for body-part recognition. IEEE transactions on medical imaging,35(5):1332–1343, 2016.

[34] M. D. Zeiler and R. Fergus. Visualizing and understandingconvolutional networks. In European conference on com-puter vision, pages 818–833. Springer, 2014.

[35] Z. Zhang, P. Chen, M. Sapkota, and L. Yang. Tandemnet:Distilling knowledge from medical images using diagnos-tic reports as optional semantic references. In InternationalConference on Medical Image Computing and Computer-Assisted Intervention, pages 320–328. Springer, 2017.

[36] Z. Zhang, Y. Xie, F. Xing, M. McGough, and L. Yang. Md-net: a semantically and visually interpretable medical imagediagnosis network. arXiv preprint arXiv:1707.02485, 2017.

[37] L. Zhao and K. Jia. Multiscale cnns for brain tumor seg-mentation and diagnosis. Computational and mathematicalmethods in medicine, 2016, 2016.

[38] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Tor-ralba. Learning deep features for discriminative localization.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 2921–2929, 2016.

[39] W. Zhu, Q. Lou, Y. S. Vang, and X. Xie. Deep multi-instancenetworks with sparse label assignment for whole mammo-gram classification. In International Conference on Medi-cal Image Computing and Computer-Assisted Intervention,pages 603–611. Springer, 2017.

[40] J. Zilly, J. M. Buhmann, and D. Mahapatra. Glaucoma de-tection using entropy sampling and ensemble learning forautomatic optic cup and disc segmentation. ComputerizedMedical Imaging and Graphics, 55:28–41, 2017.

10

Supplementary materialScale patch-level probability to avoid numerical un-derflow

Notation pkij and 1−pkij represent a patch’s (the jth patchof image i) positive and negative probabilities for class k.Their values are always in [0, 1]. We consider the problemof numerical underflow as follows. The product terms (

∏)

in Eq. 1 and Eq. 2 can quickly go to 0 when many of theterms in the product is small due to the limited precision offloat numbers. The log loss in Eq. 3 mitigates this for Eq. 1,but does not help Eq. 2, since the log function can not di-rectly affect its product term. This effectively renders Eq. 2as a constant value of 1, making it irrelevant on updatingthe network parameters. (The contribution of the gradientfrom Eq. 2 will be close to 0.) Similar things happen at testtime. To do binary classification for an image, we deter-mine its label by thresholding the image-level score (Eq. 2).It is impossible to find a threshold in [0, 1] to distinguish theimage-level scores when the score (Eq. 2) is a constant of 1;all the images will be labeled the same.

Fortunately, if we can make sure that the image-levelscores p(yk|xi,bboxki )’s and p(yk|xi) spread out in [0, 1]instead of congregating at 1, we then can find an appropri-ate threshold for the binary classification. To this end, wenormalize pkij and 1−pkij from [0, 1] to [0.98, 1]. The reasonof such choice is as follows. In the actual system, we oftenuse single-precision floating-point number to represent realnumbers. It can represent a real number as accurate as 7decimal digits [5]. If the number of patches in an image,m = 16×16, a real number p ∈ [0, 1] should be larger thanaround 0.94 (by obtaining p from p256 ≥ 10−7) to makesure that the pm varies smoothly in [0, 1] w.r.t. p changes in[0.94, 1]. To be a bit more conservative, we set 0.98 as ourlower limit in our experiment. This method enables validand efficient training and testing of our method. And in theevaluation, the number of thresholds can be finite to calcu-late the AUC scores, as the image-level probability scoreis well represented using the values in [0, 1]. A downsideof our approach is that a normalized patch-level probabilityscore does not necessarily reflect the meaning of probabilityanymore.

Disease Localization Results

Similarly, we investigate the importance of boundingbox supervision by using all the unannotated images and in-creasing the amount of annotated images from 0% to 80%by the step of 20% (Figure 8). without annotated images(the most left bar in each group), the model is only super-vised by image-level labels and optimized using probabilis-tic approximation from patch-level predictions. The resultsby unannotated images only are not able to generate accu-rate localization of disease. As we increase the amount ofannotated images gradually from 0% to 80% by the step of20% (from left to right in each group), the localization ac-

curacy for each type is increased accordingly.Next, we fix the amount of annotated images to 80%

and increase the amount of unannotated images from 0% to100% by the step of 20% to observe whether unannotatedimages are able to help annotated images to improve theperformance (Figure 9). For some diseases, it achieves thebest accuracy without any unannotated images. For mostdiseases, the accuracy experience an accuracy increase, apeak score, and then an accuracy fall (from orange to greenbar in each group) as we increase the amount of unanno-tated images. A possible explanation is that too many unan-notated images overwhelm the strong supervision from thesmall set of annotated images. A possible remedy is tolower the weight of unannotated images during training.

Lastly, We use 80% annotated images and 50% unanno-tated images to train the model and evaluate on the other20% annotated images in each fold. Comparing with thereference model [31], our model achieves higher localiza-tion accuracy for various T(IoR) as shown in Table 3.

Disease Identification Results using NIH data splitsWe additionally evaluate our method using the updated

data split from Wang et al. [31].7 Note that in this updatedsplit, all images with annotated bounding boxes are in thetest list. Thus, we cannot use any annotation information toimprove the training. Also, in this released split, they splitthe images at the patient level. In Table 4, we explore dif-ferent image models including variants of ResNet [13, 14]and DenseNet [16]. There are some AUC performance dropcompared with Figure 4. Still, without bounding box in-formation, we can see from Table 4 that our MIL formula-tion itself helps to improve the AUC performance comparedwith the baseline.

7This split was posted after CVPR 2018 paper submission deadline.

11

Annotated

Unannotated0.5901

0.8126

0.7255

0.8489 0.6857

0.2861

0.2262

0.3776

0.6768

0.8944

0.8227

0.8702 0.7083 0.4330

0.3741

0.5124

0.7081

0.8957

0.8921

0.8948

0.7426

0.4443

0.4260

0.5802

0.7336

0.9348

0.8697

0.9064

0.7563 0.4802

0.5088

0.6316

0.7811

0.9732

0.8944

0.9568 0.7398 0.4787

0.5573

0.63730.0

0.2

0.4

0.6

0.8

1.0

Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

0% : 100%

20% : 100%

40% : 100%

60% : 100%

80% : 100%

Figure 8. Disease localization accuracy using IoU where T(IoU)=0.1. Training set: annotated samples, {0% (0), 20% (176), 40% (352),60% (528), 80% (704)} from left to right for each disease type; unannotated samples, 100% (111, 240 images). The evaluation set is 20%annotated samples which are not included in the training set. For each disease, the accuracy is increased from left to right, as we increasethe amount of annotated samples, because more annotated samples bring more bounding box supervision to the joint model.

Annotated

Unannotated

0.4883

0.9894 0.6932

0.8421

0.3423 0.0810

0.7150 0.4370

0.6866

0.9784

0.8307

0.8996 0.6340

0.2411

0.5679

0.5762

0.7013

0.9841

0.8697

0.9360 0.6916

0.3515

0.5871

0.6166

0.7233

0.9805

0.8663

0.9245 0.7252

0.4375

0.5666

0.5785

0.7319

0.9750

0.8648

0.9042 0.6570

0.5365

0.4511

0.5942

0.7811

0.9732

0.8944

0.9568 0.7398 0.4787

0.5573

0.6373

0.0

0.2

0.4

0.6

0.8

1.0

Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

80% : 0%

80% : 20%

80% : 40%

80% : 60%

80% : 80%

80% : 100%

Figure 9. Disease localization accuracy using IoU where T(IoU)=0.1. Training set: annotated samples, 80% (704 images); unannotatedsamples, {0% (0), 20% (22, 248), 40% (44, 496), 60% (66, 744), 80% (88, 892), 100% (111, 240)} from left to right for each diseasetype. The evaluation set is 20% annotated samples which are not included in the training set. Using annotated samples only can producea model which localizes some diseases. As the amount of unannotated samples increases in the training set, the localization accuracy isimproved and all diseases can be localized. The joint formulation for both types of samples enables unannotated samples to improve theperformance with weak supervision.

T(IoR) Model Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia Pneumothorax

0.1 ref. 0.62 1.00 0.80 0.91 0.59 0.15 0.86 0.52ours 0.77± 0.06 0.99± 0.01 0.91± 0.04 0.95± 0.05 0.75± 0.08 0.40± 0.11 0.69± 0.09 0.68± 0.10

0.25 ref. 0.39 0.99 0.63 0.80 0.46 0.05 0.71 0.34ours 0.57± 0.09 0.99± 0.01 0.79± 0.02 0.88± 0.06 0.57± 0.07 0.25± 0.10 0.62± 0.05 0.61± 0.07

0.5 ref. 0.19 0.95 0.42 0.65 0.31 0.00 0.48 0.27ours 0.35± 0.04 0.98± 0.02 0.52± 0.03 0.62± 0.08 0.40± 0.06 0.11± 0.04 0.49± 0.08 0.43± 0.10

0.75 ref. 0.09 0.82 0.23 0.44 0.16 0.00 0.29 0.17ours 0.20± 0.04 0.87± 0.05 0.34± 0.06 0.46± 0.07 0.29± 0.06 0.07± 0.04 0.43± 0.06 0.30± 0.07

0.9 ref. 0.07 0.65 0.14 0.36 0.09 0.00 0.23 0.12ours 0.15± 0.03 0.59± 0.04 0.23± 0.05 0.32± 0.07 0.22± 0.05 0.06± 0.03 0.34± 0.04 0.22± 0.05

Table 3. Disease localization accuracy comparison using IoR where T(IoR)={0.1, 0.25, 0.5, 0.75, 0.9}. The bold values denote the bestresults. Note that we round the results to two decimal digits for table readability. Using different thresholds, our model outperforms thereference baseline in most cases and remains capability of localizing diseases when the threshold is big. The results for the referencebaseline are obtained from the latest update of [31].

Disease Atelectasis Cardiomegaly Consolidation Edema Effusion Emphysema FibrosisWang et al. [31] 0.7003 0.8100 0.7032 0.8052 0.7585 0.8330 0.7859ResNet v1 50 0.7261 0.8415 0.7127 0.8109 0.7921 0.8824 0.7695ResNet v2 50 0.7274 0.8357 0.7198 0.8057 0.7886 0.8876 0.7706

ResNet v1 101 0.7227 0.8359 0.7166 0.7958 0.7908 0.8663 0.7588ResNet v2 101 0.7182 0.8495 0.7165 0.8109 0.7884 0.8562 0.7616DenseNet 121 0.7280 0.8477 0.7271 0.8226 0.7824 0.7567 0.7625DenseNet 161 0.7152 0.8322 0.7189 0.8154 0.7725 0.7435 0.7592DenseNet 169 0.7291 0.8459 0.7201 0.8035 0.7813 0.7513 0.7605

Disease Hernia Infiltration Mass Nodule Pleural Thickening Pneumonia PneumothoraxWang et al. [31] 0.8717 0.6614 0.6933 0.6687 0.6835 0.6580 0.7993ResNet v1 50 0.6933 0.6654 0.7826 0.7014 0.7297 0.6519 0.8101ResNet v2 50 0.6933 0.6722 0.7758 0.6961 0.7373 0.6494 0.8075

ResNet v1 101 0.6852 0.6612 0.7759 0.7063 0.7380 0.6239 0.8074ResNet v2 101 0.6910 0.6715 0.7830 0.7092 0.7361 0.6532 0.8069DenseNet 121 0.6532 0.6452 0.7468 0.7017 0.7354 0.6322 0.8015DenseNet 161 0.6453 0.6549 0.7582 0.7005 0.7309 0.6258 0.7941DenseNet 169 0.6679 0.6725 0.7426 0.7104 0.7298 0.6325 0.7925

Table 4. AUC scores comparison among different image models and the baseline. Results are rounded to four decimal digits for accuratecomparison. The data split for training and test set is obtained from the latest update of [31].

12