Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2223–2236November 16 - 20, 2020. c©2020 Association for Computational Linguistics

2223

Reinforcement Learning with Imbalanced Datasetfor Data-to-Text Medical Report Generation

Toru Nishino1

Ryuji Kano1Ryota Ozaki1

Norihisa Nakano1

Tomoko Ohkuma1

1Fuji Xerox Co., Ltd.

Yohei Momoki2

Yuki Tagawa1

Keigo Nakamura2

2Fujifilm Corporation

Tomoki Taniguchi1

Motoki Taniguchi1

nishino.toru@fujixerox.co.jp, toru.nishino@fujifilm.com

Abstract

Automated generation of medical reports thatdescribe the findings in the medical imageshelps radiologists by alleviating their work-load. Medical report generation system shouldgenerate correct and concise reports. However,data imbalance makes it difficult to train mod-els accurately. Medical datasets are commonlyimbalanced in their finding labels because in-cidence rates differ among diseases; moreover,the ratios of abnormalities to normalities aresignificantly imbalanced. We propose a novelreinforcement learning method with a recon-structor to improve the clinical correctness ofgenerated reports to train the data-to-text mod-ule with a highly imbalanced dataset. More-over, we introduce a novel data augmentationstrategy for reinforcement learning to addition-ally train the model on infrequent findings.From the perspective of a practical use, weemploy a Two-Stage Medical Report Genera-tor (TS-MRGen) for controllable report gener-ation from input images. TS-MRGen consistsof two separated stages: an image diagnosismodule and a data-to-text module. Radiolo-gists can modify the image diagnosis moduleresults to control the reports that the data-to-text module generates. We conduct an experi-ment with two medical datasets to assess thedata-to-text module and the entire two-stagemodel. Results demonstrate that the reportsgenerated by our model describe the findingsin the input image more correctly.

1 Introduction

Writing medical reports manually from medical im-ages is a time-consuming task for radiologists. Towrite reports, radiologists first recognize what find-ings are included in medical images, such as com-puted tomography (CT) and X-ray images. Thenradiologists compose reports that describe the rec-ognized findings correctly without omission. Doc-tors prefer radiology reports written in natural lan-guage. Other types of radiology reports, such as

Figure 1: Overview of our Two-Stage Medical ReportGenerator (TS-MRGen) system.

Figure 2: Imbalanced distribution of the part of the find-ing labels in the MIMIC-CXR dataset.

tabular reports, are difficult to understand becauseof their complexity.

The purpose of our work is to build an auto-mated medical report generation system to reducethe workload of radiologists. As shown in Fig-ure 1, the medical report generation system shouldgenerate correct and concise reports for the inputimages. However, data imbalance may reduce thequality of automatically generated reports. Med-ical datasets are commonly imbalanced in theirfinding labels because incidence rates differ among

2224

diseases; moreover, the ratios of abnormalities tonormalities are also significantly imbalanced. Fig-ure 2 shows an imbalanced distribution of find-ing labels in the MIMIC-CXR dataset (Johnsonet al., 2019). For example, the finding label “En-larged Cardiomediastinum.Negative” appears ap-proximately 70 times more frequently than the find-ing label “Atelectasis.Negative”. As a result of thatimbalance, the generation model tends to train onlythe frequent finding labels, and tends to omit de-scriptions of the infrequent labels. This tendencyincreases incorrectness of generated reports.

To improve the correctness of generated reports,we propose a novel reinforcement learning (RL)strategy for a data-to-text generation module with areconstructor. We introduce a new reward, ClinicalReconstruction Score (CRS), to quantify how muchinformation the generated reports retain about theinput findings. The reconstructor calculates CRSand uses it as a reward for RL to train the modelto generate a greater number of correct reports.Additionally, we introduce a new ReinforcementLearning with Data Augmentation method (RL-DA) to alleviate data imbalance problems that arisefrom infrequent findings.

To replace the entire workflow of radiologists,end-to-end image captioning approach is primarilyconsidered (Monshi et al., 2020). They generatereports solely from input medical images. How-ever, such approaches are difficult to apply to thereal medical field for the following two reasons.First, the quality of generated reports is adverselyaffected by the insufficient accuracy of image diag-nosis systems. To generate correct reports, radiolo-gists must be able to correct wrong image diagnosisresults. Second, end-to-end models cannot reflectthe intentions of radiologists to reports. In contrastto abnormalities, normalities are less important butfrequently appear in the images. Radiologists some-times deliberately omit the descriptions of somenormalities to write concise reports, especially atreturn visits. To generate concise reports, radiol-ogists should be able to select which findings thesystem should include in the reports.

We employed the Two-Stage Medical ReportGenerator (TS-MRGen), a novel framework forcontrollable report generation. Figure 1 presentsan overview of TS-MRGen. TS-MRGen consistsof two separate stages: an image diagnosis moduleand a data-to-text generation module. The imagediagnosis module recognizes the findings in the

image. Subsequently, reports are generated by thedata-to-text module. Radiologists can modify thewrong or unintended results of the image diagnosismodule. Next, the modified findings are used asthe input to the data-to-text module. This approachgreatly improves the correctness and concisenessof generated reports.

Overall, the main contributions of this study areas follows:

• We introduce a reinforcement learning strat-egy with Clinical Reconstruction Score (CRS)to generate more clinically correct reports.

• We propose a novel Reinforcement Learningwith Data Augmentation (RL-DA) to addressdata imbalance difficulties.

• We design and conduct experiments to vali-date the effectiveness of Two-Stage MedicalReport Generator (TS-MRGen) with a modifi-cation process.

We evaluate the proposed approach on twodatasets: the Japanese Computed Tomography(JCT) dataset and the MIMIC-CXR dataset. Auto-matic and manual evaluations on the JCT datasetshow that our CRS and RL-DA improve the cor-rectness of generated reports. An experiment con-ducted on the MIMIC-CXR dataset shows the gen-erality of CRS and RL-DA; moreover, the experi-ment on the MIMIC-CXR dataset demonstratesthat TS-MRGen with the modification processgenerates more correct reports than the two-stagemodel without a modification process.

2 Related Work

Medical Report Generation. Many end-to-endmedical report generation models have been pro-posed (Monshi et al., 2020) to generate reportsfrom images. Jing et al. (2018) introduced a co-attention mechanism to align semantic tags andsub-regions of images. However, this model tendsto generate sentences describing normalities to anexcessively degree. This tendency results from animbalanced frequency of findings among the med-ical images. Jing et al. (2019) and Harzig et al.(2019) use different decoders to generate normal-ities or abnormalities to address these data imbal-ance difficulties.

Biswal et al. (2020) accepts doctors’ anchorwords for controllable medical report generation.

2225

This model generates reports that are more faith-ful to doctors’ preferences by retrieving templatesentences from the word entered by the doctor.Data-to-Text Generation. Data-to-text genera-tion is a task to generate a fluent text that is faithfulto the input data. Wiseman et al. (2017) proposed adata-to-text model with reconstruction-based tech-niques. The method trains the model so that theinput data can be reconstructed from the decoderhidden state. This reconstruction makes it morelikely that the hidden state of the decoder can cap-ture the input data properly.

Ma et al. (2019); Moryossef et al. (2019) pro-posed a two-step data-to-text model, comprising atext planning module and a text realization module.This model not only generates a text that is morefaithful to the input data than end-to-end models,but it also allows for user control over the gener-ated text by supplying a modified plan to the textrealization module.Data Augmentation for Text Generation. Typi-cal machine learning approaches that address dataimbalance, such as undersampling and oversam-pling, are difficult to apply to this task becausethe input images or finding labels are sets of mul-tiple finding class label and the target reports arediscrete sequences. Kedzie and McKeown (2019)applied data augmentation method to data-to-textgeneration. To obtain additional training data, theygenerated data-text pairs by the model itself usingthe noise injection sampling method.Text Generation with Reinforcement Learning(RL). Text generation with Reinforcement Learn-ing (RL) enables the model to train with indiffer-entiable rewards, such as BLEU (Papineni et al.,2002) and ROUGE (Lin, 2004) metrics. Zhang et al.(2020b) improved the radiology report summariza-tion model with RL using a factual correctnessreward. Liu et al. (2019a) applied RL for medicalreport generation with Clinically Coherent Reward(CCR) to directly optimize the model for clinicalefficacy. Both methods leverage CheXpert Labeler(Irvin et al., 2019), a medical observation annotator,to calculate rewards.

Our work addresses the data imbalance diffi-culties beyond the imbalance between normalitiesand abnormalities, as Jing et al. (2019) addressed.Moreover, with our approach, the doctors can re-flect their intentions to reports more directly to agreater degree than Biswal et al. (2020). We extendthe factual-based RL method (Liu et al., 2019a) to

cases for which rule-based annotators are not avail-able. Furthermore, we propose data augmentation(Kedzie and McKeown, 2019) for RL to train themodel using only the input labels.

3 Method

Medical report generation is a task to gener-ate reports consisting of a sequence of wordsY = {y1, y2, ...yN} from a set of images X ={xk}Mk=1. Most cases Y include more than onesentence. We annotated a set of finding labelsF = {f1, f2, ...fT } for each set of images. Thefinding labels include abnormalities (indicated as.Positive), normalities (indicated as .Negative) anduncertain findings (indicated as .Uncertain) . Eachfinding label can be disassembled into a sequenceof words as ft = {wt1, wt2, ...wtK}. For example,an abnormality “Airspace Opacity.Positive” labelis divided into a sequence of {airspace, opacity,positive}.

3.1 Two-Stage Medical Report Generator

We employ Two-Stage Medical Report Generator(TS-MRGen), a framework that consists of twoseparate stages: an image diagnosis module and adata-to-text generation module. The image diag-nosis module can be regarded as an image classi-fication task that recognizes input images X andclassifies them into a set of findings F . Radiolo-gists can modify the image diagnosis module resultF if errors are found in F . Alternatively, they canintentionally omit or append findings labels. Thedata-to-text generation module generates a reportY from F . We consider the text generation moduleas a data-to-text task.

3.2 Image Diagnosis Module

We train an image classification model that takesas input a single-view chest X-ray and output a setof probabilities of four types of labels (positive,negative, uncertain, and no mention) for each pos-sible finding label. We use EfficientNet-B4 (Tanand Le, 2019) as a network architecture that wasinitialized with the pretrained model on ImageNet(Deng et al., 2009).

In some cases, the reports are described based ontwo images: front view and lateral view. FollowingIrvin et al. (2019), this module outputs the meanprobability of the model between two images.

2226

!!

!"#$%&"

'(($)*+,

!+)*-*.#

!,#$/+,*&

0#1&-*.#

!"

0+

2,#$/+,*&

*)

)##,

&,3445

"# !"#$%&'()#*%)$+",-'+-'."/01$/(

2'$*3&#-045$.0)-6$7+$3042&(-)-6$

!"#$ %&

!"#$%&' (&#$%&' )&#$"*+',#+$'

-./"/#0.1)&#$"*+',#+/$"12#$'&13-)24

"$# 641#,#%&-#34)#,-#,7#489

)&/2"*,14

!!"-'8*)"

,-'+-'."/01$/(

!"#$%&'(

!"#$%&'(

!"#$%&'(

!"#$%&'(

!"#$%&'(

!"#

!""

#$$%

Figure 3: Overview of our reinforcement learning (RL) with a reconstructor. We leverage the clinical Reconstruc-tion Score (CRS), which estimates the factual correctness of generated reports, as a reward for RL.

3.3 Text Generation Module

We adopt a table-to-text encoder-decoder model(Liu et al., 2018) for the text generation module touse words in the findings class labels. The encoderof the text generation module has two layers: aword-level encoder and a label-level layer.

hwtk = Encword(wtk, hwtk−1) (1)

hlt = MLPlabel([hwt0, h

wtK ]) (2)

Therein, [hwt0, hwtK ] denotes the concatenation of

vectors hwt0 and hwtK . MLPlabel represents a multi-layer perceptron. We use a one-layer bi-directionalgated recurrent unit (GRU) for the word level en-coder.

For the decoder, we use one-layer GRU with anattention mechanism (Bahdanau et al., 2015):

yn = Dec(yn−1, hl, hdn−1, cn) (3)

where hl represents the max-pooled vector from{hl0, ..., hlT }. The context vector cn is calculatedover the label-level hidden vectors hlt and the de-coder hidden state hdn.

3.4 RL with Reconstructor

We use RL to train the text generation model toimprove the clinical correctness of the generatedreports. A benefit of RL is that the model can betrained to produce sentences that maximize the re-ward, even if the word sequence does not matchthe correct answer. Many studies of text generationwith RL (Keneshloo et al., 2019) use rewards, suchas the BLEU and ROUGE metrics, to improve thegenerated text. To improve the clinical correctnessof the generated reports, Liu et al. (2019a) and Irvinet al. (2019) adopted clinically coherent rewardsfor RL with CheXpert Labeler (Irvin et al., 2019),

a rule-based finding mention annotator. However,in the medical domain, no such annotator is avail-able in most cases other than English chest X-rayreports.

We propose a new reward, Clinical Reconstruc-tion Score (CRS), to quantify the factual correct-ness of reports with a reconstructor module. Figure3 shows an overview of our method, RL with CRS.Contrary to the data-to-text generator, the recon-structor reversely predicts the appropriate findinglabels from the generated reports. This reconstruc-tor quantifies the clinical correctness of the reports.Therefore, we can estimate the correctness of re-ports without rule-based annotators.

We utilize BERT (Devlin et al., 2019) as a recon-structor and reconstructed the finding labels F as amulti-label text classification task:

F = FC(BERT(Y )) (4)

where FC and BERT represent the fully connectedlayer and the BERT layer, respectively. Y denotesa generated report. In addition, CRS is definedas an F-score of the predicted finding labels Fagainst the input finding labels for the data-to-textmodule F . This BERT reconstructor is trained witha Class-Balanced Loss (Cui et al., 2019) to addressimbalanced datasets.

We design the overall reward as a combinationof ROUGE-L score and CRS:

R(Y ) = λrougeROUGE(Yt, Y ) +

(1− λrouge)CRS(Yt) (5)

where Yt represents a gold report regarding thepredicted report Y and λrouge is a hyperparameter.

The goal of RL is to find parameters to minimizethe negative expected reward R(Y ) for Y :

Lrlθ = −EY∼PθR(Y ) (6)

2227

where Pθ denotes a policy network for the textgeneration model.

We adopt SCST (Rennie et al., 2017) to approxi-mate the gradient of this loss:

∇θLrlθ ≈ −∇θ logPθ(Y s)(R(Y s)−R(Y g)) (7)

where Y s is a sampled sequence with a MonteCarlo sampling. We use the softmax function withtemperature τ for sampling sequences. R(Y g) is abaseline reward calculated from a greedily decodedsequence Y g.

To train the language model, RL with only CRSand ROUGE as a reward is insufficient. Therefore,we use the cross-entropy loss to generate fluentsentences. We design an overall loss function fortraining as a combination of the RL loss and cross-entropy loss Lxent:

Lall = λrlLrl + (1− λrl)Lxent (8)

where Lxent is the cross-entropy loss calculatedbetween the gold reports and generated reports,and λrl is a hyperparameter.

3.5 Reinforcement Learning with DataAugmentation (RL-DA)

We propose a novel method, RL with Data Aug-mentation method (RL-DA), to encourage themodel to focus on infrequent findings. We focuson the asymmetricity between the augmentationcost of the input data and that of the target reportsentences. The input data, which comprise a set offinding labels, can be augmented easily by addingor removing a finding label automatically. How-ever, the augmentation cost is higher for the targetreports than the input data because the target re-ports are written in natural language. Therefore, weintroduce a semi-supervised reinforcement learn-ing method to train the model solely by augmentingthe input data.

We conduct a data augmentation process of RL-DA as the following steps.Step 1: List and Filter all Candidate Find-ing Labels. Given a set of finding labels F ={f1, f2, ...fT }, the objective of the data augmenta-tion is to obtain a new set of finding labels F , forwhich an additional finding label fT+1 is added toF . We list all finding labels that can be appendedto F . We filter the finding labels inappropriatefor appending F according to the clinical relationbetween the labels. Some pairs of finding labelshave clinically contradictory relations. We filterthe labels based on the following two rules.

a. Contradictory Relation. We exclude a pairof contradictory finding labels. For example,the abnormality “Pleural Effusion.Positive”and the normality “Pleural Effusion.Negative”must not be included in the same set F .

b. Supplementary Relation. We exclude a pairof contradicting finding labels that supple-ment other finding labels in F . For example,“Pleural Effusion.Mild” is excluded if “Pleu-ral Effusion.Positive” not in F .

Step 2: Assign Sample Finding Labels. We sam-ple an additional finding label fT+1 to append to F .The label is extracted from a set of candidates byrandom sampling. The data imbalance is mitigatedbecause the data augmentation process appends anew finding label irrespective of the frequency ofthis finding labels in the training data.

We use this augmented set of finding labels Ffor RL. The overall loss function is as follows:

Lall = λrl(Lrl + λaugL

aug) + (1− λrl)Lxent (9)

where λrl and λaug are hyperparameters. Laug de-notes the RL loss calculated using the augmentedset F . Laug is calculated in the same way asLrl with a reward R(Y ) under the condition ofλrouge = 0. This is because no reference report isavailable for the augmented set F . Hence, RL-DAmethod enables training of the model with moredata at a low cost.

4 Experiment

First, to evaluate the effects of our proposed CRSand RL-DA on the data-to-text module, we con-duct an experiment with the Japanese ComputedTomography (JCT) dataset. Moreover, to evaluatethe generality of CRS and RL-DA and the effectsof the modification process on TS-MRGen, we con-duct an experiment with the MIMIC-CXR dataset.

4.1 Evaluation on the JCT Dataset

Dataset and Experimental Settings. We evaluatethe data-to-text module with the JCT dataset. TheJCT dataset has pairs of input sets for finding labelsand target medical reports written in Japanese. TheJCT dataset is used only to evaluate the data-to-textmodule. Therefore, we did not prepare medicalimages for the JCT dataset.

We defined a system of finding labels after con-sultation with radiologists. Annotators with fully

2228

sufficient knowledge in the radiology report manu-ally annotate the finding labels to the reports. De-scriptions that were unrelated to any finding labelsin the reports were omitted from preprocessing forprivacy reasons.

We chose all hyperparameters based on the CRSscores of the validation data. Details of our models,metrics, training, and dataset are included in theSupplementary section for reproducibility.

We compare the following six text generationmodels :(1) Table-to-Text (Baseline): Table-to-text modelwithout RL proposed in Section 3.3(2) 1-NN: calculates the relevance of input findinglabels using TF-IDF and selects the most relevantreports from the training data.(3) Rule-Based: generates reports based on manu-ally prepared templates. We prepared one templatesentence per one finding label, and the method con-catenates template sentences to construct the entirereports.(4) Seq2Seq: normal encoder-decoder model withGRU.(5) CNN-enc: encoder-decoder model with a CNNencoder.(6) Hier-Dec: encoder-decoder model with a hier-archical decoder (Jing et al., 2019).

Additionally, we compare the following four RLstrategies to train the table-to-text text generationmodel:(7) RLR: trains the model by RL using onlyROUGE as a reward.(8) RLCRS: trains the model by RL using only CRSas a reward.(9) RLCRS+R: trains the model by RL with CRSand ROUGE as a reward.(10) RL-DACRS+R: trains the model by RL withCRS and ROUGE and applies RL-DA proposed inSection 3.5.Results. The upper part of Table 1 presents au-tomatic evaluation results regarding the text gen-eration models. The rule-based method obtainedthe lowest ROUGE-L because it generated con-siderably redundant reports. Table-to-Text modelachieved the best CRS, so we selected the table-to-text model as a text generation module.

The lower part of Table 1 presents auto-matic evaluation results regarding training strate-gies. This result demonstrates that applicationof ROUGE as a reward improves ROUGE-Lscores, whereas application of CRS as a reward

ROUGE BLEU CRSComparison of Text Generation Model

(1) Table-to-Text (Baseline) 66.4 39.7 78.2(2) 1-NN 62.4 33.6 73.1(3) Rule-Based 60.9 36.2 80.3(4) Seq2Seq 64.6 38.7 76.3(5) CNN-enc 62.0 37.2 74.1(6) Hier-Dec 62.1 37.0 77.5

Comparison of Training Strategy(7) RLR 69.6 42.0 79.3(8) RLCRS 64.8 38.4 79.4(9) RLCRS+R 67.2 40.5 80.7(10) RL-DACRS+R

(Proposed) 68.2 41.1 81.3

Table 1: Automatic evaluation results of the data-to-text module with the JCT dataset. The pro-posed RL-DACRS+R achieved the best score on CRS,whereas RLR achieved the best score on BLEU andROUGE. The CRS of the proposed model is statis-tically significant compared with the baseline model(p < 0.01).

Correctness Gramma-P R F ticality

Baseline 89.7 70.2 77.8 94.0RLR 92.9 75.1 82.2 95.5RL-DACRS+R 95.1 75.2 83.1 95.0

Table 2: Comparison of manual evaluation results ofthe data-to-text module on the JCT dataset. Correctnessand fluency scores represent an average of the scores oftwo workers. P, R, and F denote the precision, recall,and F-score of correctness, respectively. The correct-ness scores of the proposed model are statistically sig-nificant compared with the baseline model (p < 0.01).

improves CRS scores. This indicates that RL im-proves the metric used as a reward. Our pro-posed RL-DACRS+R achieved higher both CRSand ROUGE scores than RLCRS+R.

From the automatic evaluation results, we can-not conclude that our proposed CRS and RL-DAimproved correctness because we have no meansof deciding which metric is more appropriate forevaluating the generated reports. To estimate theeffects of our proposed method on the data-to-textmodel, we also conducted a manual evaluation. Asin previous research (Zhang et al., 2020a), two spe-cialists who are knowledgeable in radiology reportsmeasured 100 randomly selected samples for eachexperimental condition.

We defined the following metrics for a manualevaluation:

• Grammaticality: The percentages of reportsthat contain no grammatical errors.

• Correctness: Measure how well the reports

2229

describe the clinically correct information.We define the correctness of Y as an F-scorewith the following precision and recall:

Precision(Y ) =NTP

NTP +NFP(10)

Recall(Y ) =NTP

NTP +NFN(11)

where NTP indicates the number of findings cor-rectly noted in Y , NFN indicates the number ofmissing findings in Y , and NFP indicates the num-ber of findings mistakenly noted in Y .

Table 2 presents manual evaluation results. Com-pared with RLR, our proposed RL-DACRS+R alsoimproves the correctness of manual evaluation.This indicates that proposed RL-DACRS+R doesnot merely improve the CRS score; it improves theclinical correctness of the generated reports.

4.2 Evaluation on the MIMIC-CXR Dataset

Datasets. We evaluated the data-to-text moduleand the entire system on the MIMIC-CXR dataset,which includes chest X-ray images and their corre-sponding medical reports written in English. No-tably, these reports include descriptions other thanfindings, such as indications and impressions. Weomitted these descriptions other than findings be-cause these descriptions cannot be generated fromthe input images. We used the CheXpert dataset(Irvin et al., 2019) to train the image diagnosismodule.

We annotated finding labels to the MIMIC-CXRdataset with CheXpert Labeler (Irvin et al., 2019)and the image diagnosis module. CheXpert Labelerannotated findings labels for 14 categories of threetypes: positive, negative, and uncertain labels. Forthe training data of the data-to-text module, welabeled the reports using CheXpert Labeler only.

In addition to BLEU metrics, we adopted CheX-pert accuracy, precision, and F-score metrics toquantify the correctness of generated reports. Thisis because the domain-agnostic metrics (such asBLEU) are doubtful in evaluating the quality ofreports, and CheXpert-based metrics are more reli-able metrics, as reported by (Boag et al., 2019). Wechose all hyperparameters based on the F-scores ofthe validation data. Details of our models, metrics,training, and dataset are described in the Supple-mentary section for reproducibility.Experimental Settings. For evaluation, we pre-pare the following four experimental conditions.

(a) Data-to-Text Evaluation. We provide onlythe gold finding labels as inputs to the data-to-textmodule, and then evaluate the generated reports.This evaluation is intended to assess whether ourproposed method is also applicable to the MIMIC-CXR dataset or not. Therefore, in this evaluation,we focus only on the data-to-text module. We com-pare our proposed model RL-DACRS+R which istrained by RL with CRS and ROUGE, and appliedRL-DA with the baseline table-to-text model.(b) End-to-End Evaluation. We compare ourTS-MRGen with the end-to-end models, such asCNN-RNN (Boag et al., 2019) and CCR appliedmodels (Liu et al., 2019a). As shown on the leftside of Figure 1, the end-to-end model directlygenerates target reports from the input images. Thisevaluation setting do not use the finding labels inany way.(c) Two-Stage Evaluation without Modification.We evaluate our TS-MRGen using the same inputsand outputs as the end-to-end models. As shownin Figure 1, TS-MRGen first predicts the findinglabels to describe the findings in the input images.Next, it generates reports from the finding labels.We employ RL-DACRS+R to the data-to-text mod-ule of TS-MRGen.(d) Two-Stage Evaluation with Modification. Inaddition to (c) above, we apply the modificationprocess to the finding labels predicted by the imagediagnosis module. However, it is too expensive toevaluate the model in this condition because thecost of radiologist services is too high. Therefore,we imitate this modification flow using CheXpertLabeler using the following process.(i) Obtain the output probability vector p(ft|X) ofthe finding labels predicted by the image diagnosismodule.(ii) Classify the predicted finding labels as con-fident or untrustworthy according to probabil-ity p(ft|X). If p(ft|X) is within the range of(plowth , phighth ), then we regard the predicted result ftas untrustworthy, and the result is discarded.(iii) Apply the modification process to the predictedfinding labels. We obtain the finding labels usingCheXpert Labeler and replace all untrustworthylabels classified in (ii).

This replacement process imitates the modifica-tion flow of radiologists.Results. The upper part of Table 3 presents a com-parison related to the data-to-text module. This partof the table shows that our proposed RL-DACRS+R

2230

Evaluation BLEU CheXpertCondition 1 2 3 4 Acc Prec F (micro) F (macro)

Comparison of Data-to-Text ModuleBaseline (Table-to-Text) (a) 35.2 23.0 16.1 11.9 92.2 75.9 66.3 50.5RL-DACRS+R (a) 36.4 23.3 16.4 12.1 93.2 77.1 68.9 55.9

Comparison of Entire Report Generation SystemEnd-to-End (CNN-RNN) (Boag et al., 2019) (b) 30.5 20.1 13.7 9.2 83.7 30.4 30.2 18.6End-to-End (CCR+NLG) (Liu et al., 2019a) (b) 35.9 23.7 16.4 11.3 91.8 58.6 33.8 18.7TS-MRGen w/o modification (c) 21.7 11.8 7.3 4.8 87.3 48.2 29.6 21.7TS-MRGen with modification (d) 36.3 23.1 16.3 12.1 91.7 71.0 63.4 49.5

Table 3: Automatic evaluation of the data-to-text module using the MIMIC-CXR dataset. Acc, Prec, F (micro),and F (macro) indicate accuracy, precision, micro F-score, and macro F-score, respectively. CheXpert scoresquantify the correctness of generated reports. For the data-to-text module, our proposed RL-DACRS+R achievedthe best result (bold) for all metrics. For the entire report generation system, TS-MRGen with the modificationprocess improved the correctness of the generated reports. The CheXpert scores of the proposed model and TS-MRGen with modification were statistically significant compared with the baseline model and TS-MRGen withoutmodification (p < 0.05), respectively.

Figure 4: Evaluation of generated reports for each finding label. The horizontal axis shows the frequency of eachfinding label in the training data. The vertical axis shows the CheXpert F-scores for each finding label. The left plotpresents a comparison between the proposed RL-DACRS+R and baseline method. Our proposed method improvesCheXpert F-scores, especially for infrequent finding labels. The right plot presents an effect of the modificationprocess. Our TS-MRGen presents the important benefit of improving correctness through modification processes.

improves the clinical accuracy of the generated re-ports for the MIMIC-CXR dataset.

The lower part of Table 3 presents a compari-son related to the entire report generation system.Compared with the TS-MRGen without the mod-ification process, the TS-MRGen with the modifi-cation process achieved significantly better resultfor BLEU, CheXpert precision, micro and macroF-scores. CheXpert F-score quantifies the clinicalcorrectness more adequately. Therefore, this resultdemonstrates that our TS-MRGen has an importantadvantage because the system enables radiologiststo modify the mistakenly predicted finding labels.

5 Discussion

5.1 Effects on an Imbalanced DatasetFigure 4 presents an evaluation of generated reportsfor each finding label evaluated using CheXpert La-

beler. Both our proposed RL-DACRS+R and thebaseline method exhibit the same tendency: moreinfrequent finding labels in the training data are as-sociated with the lower correctness of the generatedreports. RL-DACRS+R outperforms the baselinemodel, especially for the infrequent finding labels,This result demonstrates that our proposed RL-DAand CRS generate more accurate reports, especiallywith infrequent labels in the training data.

5.2 Qualitative Results

The upper part of Table 4 presents an example ofa generated report for the JCT dataset. The base-line model generated a report with an incorrectdescription: “is accompanied by a pleural inden-tation.” The data imbalance causes such an error.“Pleural Indentation.Positive” is more frequent find-ing label than “Pleural Indentation.Negative” in

2231

Example of generated reports of JCT dataset.Input Finding Labels: Nodule.Positive, Nodule.Solid, Pleural.Indentation.Negative, (truncated) Border.Well DefinedReport Generated by the Baseline ModelThere is a 20 mm dilated nodule in the lung. (truncated)It is well-defined and is accompanied by a pleuralindentation.

Report Generated by the Proposed (RL-DACRS+R) ModelThere is a 20 mm dilated nodule in the lung. (truncated)It is well-defined and there is no pleural indentation.

Examples of generated reports of MIMIC-CXR dataset.Gold Finding Labels:Cardiomegaly.Positive, Enlarged Cardiomediastinum.Negative, Edema.Negative, Consolidation.Negative,Pneumothorax.Negative, Pleural Effusion.NegativeLabels Predicted by Image Diagnosis ModuleCardiomegaly.Positive

Modified LabelsCardiomegaly.Positive, Enlarged Cardiomediastinum.Negative,Edema.Negative, Consolidation.Negative,Pneumothorax.Negative, Pleural Effusion.Negative

Report generated by TS-MRGen without Modificationthe heart is mildly enlarged. moderate cardiomegaly isunchanged.

Report Generated by TS-MRGen with Modificationthe lungs are clear without focal consolidation. no pleuraleffusion or pneumothorax is seen. the cardiac silhouette ismildly enlarged. the mediastinal and hilar contours are withinnormal limits. there is no pulmonary edema.

Table 4: (upper) Example of a report generated from the JCT dataset. The italic part represents the fault inthe baseline model. The underlined part represents the correct description corresponding to the italic part. AJapanese-English translation is applied. (lower) Example of a report generated from the MIMIC-CXR dataset.The modification process compensates for the missing labels predicted by the image diagnosis module. It therebygenerates a report more faithful to the gold finding labels.

the training data. Therefore, the baseline modelmistakenly outputted a more frequently occurringdescription. However, our proposed RL-DA gen-erated a correct description: “there is no pleuralindentation”. This result demonstrates that our pro-posed RL-DA and CRS trained the model moreaccurately on infrequent finding labels.

The lower part of Table 4 presents an exam-ple of a generated report for the MIMIC-CXRdataset. Without modification processes, the gener-ated report includes only the description for “Car-diomegaly.Positive.” The image diagnosis modulehas a tendency to omit normalities because theimage diagnosis module is not able to train the in-tention of radiologists of whether normalities areomitted or not. With modification processes, thegenerated reports include the exact description ofthe gold finding labels with no omissions. Modifi-cation processes correct the missing finding labelsto the predicted labels, thereby generating morefaithful reports.

6 Conclusion

We proposed a novel Clinical Reconstruction Score(CRS) and Reinforcement Learning and Data Aug-mentation (RL-DA) methods to train a data-to-textmodel for an imbalanced dataset. Additionally, weemployed a Two-Stage Medical Report Generator(TS-MRGen) for controllable medical report gen-eration from input medical images.

An evaluation of the data-to-text module re-vealed that our proposed CRS and RL-DA methodsimproved the clinical correctness of generated re-ports, especially for infrequent finding labels. Anevaluation of the entire medical report generationsystem revealed that our TS-MRGen generatedmore correct reports than an end-to-end generationmodel.

In future work, we would like to explore whetherour method is applicable to other domain tasks indata-to-text generation, such as sports summarygeneration and biography generation tasks.

Acknowledgement

Computational resource of AI Bridging Cloud In-frastructure (ABCI) provided by National Instituteof Advanced Industrial Science and Technology(AIST) was used. The authors would like to thankthe anonymous reviewers for their constructive re-views and suggestions.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. The 2015 Interna-tional Conference on Learning Representation.

Siddharth Biswal, Cao Xiao, Lucas M Glass, M Bran-don Westover, and Jimeng Sun. 2020. CLARA:Clinical Report Auto-completion. In Proceedings ofThe Web Conference 2020.

2232

William Boag, Tzu-Ming Harry Hsu, Matthew McDer-mott, Gabriela Berner, Emily Alsentzer, and PeterSzolovits. 2019. Baselines for Chest X-Ray ReportGeneration. In Proceedings of Machine LearningResearch Volume 116: Machine Learning for HealthWorkshop.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, andSerge Belongie. 2019. Class-balanced loss basedon effective number of samples. In Proceedings of2019 IEEE Conference on Computer Vision and Pat-tern Recognition.

Jia Deng, Wei Dong, R Socher, Li-Jia Li, Kai Li, andLi Fei-Fei. 2009. ImageNet: A large-scale hierarchi-cal image database. In 2009 IEEE Conference onComputer Vision and Pattern Recognition.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofDeep Bidirectional Transformers for Language Un-derstanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers).

Philipp Harzig, Yan-Ying Chen, Francine Chen, andRainer Lienhart. 2019. Addressing data bias prob-lems for chest x-ray image report generation. Pro-ceedings of the British Machine Vision Conference2019.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu,Silviana Ciurea-Ilcus, Chris Chute, Henrik Mark-lund, Behzad Haghgoo, Robyn Ball, Katie Shpan-skaya, et al. 2019. CheXpert: A large chest radio-graph dataset with uncertainty labels and expert com-parison. In Thirty-Third AAAI Conference on Artifi-cial Intelligence.

Baoyu Jing, Zeya Wang, and Eric Xing. 2019. Show,Describe and Conclude: On Exploiting the StructureInformation of Chest X-ray Reports. In Proceedingsof the 57th Annual Meeting of the Association forComputational Linguistics.

Baoyu Jing, Pengtao Xie, and Eric Xing. 2018. On theAutomatic Generation of Medical Imaging Reports.In Proceedings of the 56th Annual Meeting of theAssociation for Computational Linguistics (Volume1: Long Papers).

Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz,Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.2019. MIMIC-CXR: A large publicly availabledatabase of labeled chest radiographs. arXivpreprint arXiv:1901.07042.

Chris Kedzie and Kathleen McKeown. 2019. A GoodSample is Hard to Find: Noise Injection Samplingand Self-Training for Neural Language GenerationModels. In Proceedings of the 12th InternationalConference on Natural Language Generation.

Y Keneshloo, T Shi, N Ramakrishnan, and CK Reddy.2019. Deep Reinforcement Learning for Sequence-to-Sequence Models. IEEE transactions on neuralnetworks and learning systems.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim,Donghyeon Kim, Sunkyu Kim, Chan Ho So,and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation modelfor biomedical text mining. Bioinformatics,36(4):1234–1240.

Chin-Yew Lin. 2004. ROUGE: A Package for Auto-matic Evaluation of Summaries. In Proceedings ofWorkshop on Text Summarization Branches Out.

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc-Dermott, Willie Boag, Wei-Hung Weng, PeterSzolovits, and Marzyeh Ghassemi. 2019a. Clini-cally Accurate Chest X-Ray Report Generation. InMachine Learning for Healthcare Conference 2019.

Liyuan Liu, Haoming Jiang, Pengcheng He, WeizhuChen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han.2019b. On the Variance of the Adaptive LearningRate and Beyond. arXiv preprint arXiv:1908.03265.

Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang,and Zhifang Sui. 2018. Table-to-text generation bystructure-aware seq2seq learning. In Thirty-SecondAAAI Conference on Artificial Intelligence.

Shuming Ma, Pengcheng Yang, Tianyu Liu, Peng Li,Jie Zhou, and Xu Sun. 2019. Key Fact as Pivot: ATwo-Stage Model for Low Resource Table-to-TextGeneration. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics.

Maram Mahmoud A Monshi, Josiah Poon, and VeraChung. 2020. Deep learning in generating radiol-ogy reports: A survey. Artificial Intelligence inMedicine, page 101878.

Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019.Step-by-Step: Separating Planning from Realizationin Neural Data-to-Text Generation. In Proceedingsof the 2019 Conference of the North American Chap-ter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Longand Short Papers).

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic eval-uation of machine translation. In Proceedings of the40th Annual Meeting on Association for Computa-tional Linguistics.

Yifan Peng, Xiaosong Wang, Le Lu, Mohammad-hadi Bagheri, Ronald Summers, and Zhiyong Lu.2018. Negbio: a high-performance tool for nega-tion and uncertainty detection in radiology reports.AMIA Summits on Translational Science Proceed-ings, 2018:188.

2233

Steven J Rennie, Etienne Marcheret, Youssef Mroueh,Jerret Ross, and Vaibhava Goel. 2017. Self-criticalsequence training for image captioning. In Proceed-ings of 2017 IEEE Conference on Computer Visionand Pattern Recognition.

Toshinori Sato, Taiichi Hashimoto, and Manabu Oku-mura. 2017. Implementation of a word segmen-tation dictionary called mecab-ipadic-NEologd andstudy on how to use it effectively for information re-trieval. In Proceedings of the 23th Annual Meetingof the Association for Natural Language Processing.

Mingxing Tan and Quoc Le. 2019. EfficientNet: Re-thinking Model Scaling for Convolutional NeuralNetworks. In Proceedings of the 36th InternationalConference on Machine Learning.

Sam Wiseman, Stuart M Shieber, and Alexander MRush. 2017. Challenges in Data-to-Document Gen-eration. In Proceedings of the 2017 Conference onEmpirical Methods in Natural Language Process-ing.

Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu,Alan Yuille, and Daguang Xu. 2020a. When Radiol-ogy Report Generation Meets Knowledge Graph. InThirty-Fourth AAAI Conference on Artificial Intelli-gence.

Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christo-pher D Manning, and Curtis P Langlotz. 2020b. Op-timizing the Factual Correctness of a Summary: AStudy of Summarizing Radiology Reports. In Pro-ceedings of the 58th Annual Meeting of the Associa-tion for Computational Linguistics.

A Supplementary Material

A.1 Dataset and Preprocessingthe JCT Dataset. We built the JCT dataset to trainthe data-to-text module of the medical report gen-eration system. For the JCT dataset, we collected4,454 medical reports regarding pulmonary nod-ules from a hospital. To train an accurate medicalreport generation system, we focused only on thefindings in the reports and excluded the sentencesthat violated patient privacy. During a consultationwith radiologists, we defined 57 types of findinglabels. As preprocessing, all descriptions that werenot related to any findings were truncated by an-notators. We lexicalized phrases referring to theexistence of nodules and phrases referring to thesize of the nodules to improve the stability of train-ing of the data-to-text generation model. We usedMeCab 1 and mecab-ipadic-NEologd (Sato et al.,2017) to tokenize the reports, and keep tokens with2 or more occurrences.

To prevent data leakage in validation/testdatasets, we split the dataset in a way to ensure thatthe same sets of finding labels are not included inthe training, validation, and test data. Additionally,to avoid the negative influence of the imbalancedfrequency of sets of finding labels, we omitted thesamples with duplicated sets of finding labels inthe validation/test dataset. These strategies for datasplitting and duplicate input handling caused dif-ferences in average labels and lengths, as shown inTable 5. If samples contained shorter sentences andfewer input labels, the validation and test datasetstended to contain longer sentences and a greaternumber of input labels.the MIMIC-CXR Dataset. Medical reports in theMIMIC-CXR dataset 2 contain descriptions thatare irrelevant to the findings in the input images.Hence, we extracted the finding sections of thereports using the scripts provided in Boag et al.(2019) 3. In training data, we truncated the sen-tences in the reports that were not related to anyfindings using CheXpert Labeler and NegBio (Penget al., 2018) parser to improve the stability of train-ing the model. We omitted the reports that didnot mention any findings or had no finding sec-tions from the training data. Note that the reportsin the validation and test data may contain a de-scription that does not mention any findings. We

1https://taku910.github.io/mecab/2https://physionet.org/content/mimic-cxr/2.0.0/3https://github.com/wboag/cxr-baselines

2234

Number of Average AverageReports labels length

the JCT datasetTraining data 3,637 4.71 27.5Validation data 418 9.46 52.7Test data 399 9.49 51.4

the MIMIC-CXR datasetTraining data 131,016 4.92 43.6Validation data 1,156 4.90 44.5Test data 2,299 5.01 54.7

Table 5: Statistics of the JCT dataset and the MIMIC-CXR dataset.

dataset JCT MIMIC-CXRData-to-Text Module Hyperparameters

Vocabulary size 339 2222Number of labels 57 40Dropout rate 0.2 0.2Word embedding size 32 64Label embedding size 16 16Hidden size 32 32Beam search width 5 5

Training HyperparametersBatch size 32 32Optimizer Adam AdamLearning rate 5.0× 10−3 2× 10−4

Learning rate decay 0.99 0.98λrouge 0.2 0.2λrl 0.2 0.03λaug 0.1 0.05τ (Softmax temperature) 0.5 0.4Dropout 0.2 0.2Gradient clipping 2.0 2.0

Table 6: List of hyperparameters of the data-to-textmodules.

use this approach to align our experimental condi-tions with previous end-to-end research Boag et al.(2019). We used the Natural Language Toolkit 4

to tokenize the reports, and keep tokens with 10 ormore occurrences. We have split the dataset intotrain, validation, and test data based on the split dis-tributed in the MIMIC-CXR-JPG (Johnson et al.,2019) 5 dataset. Table 5 presents the statistics ofthe MIMIC-CXR dataset.

A.2 Training Details

Image Diagnosis Module All images were fedinto a network with a size of 512 × 512 pixels.We set up the loss as the sum of the multi-classcross-entropy for each observations and used theRAdam (Liu et al., 2019b) optimizer with a learn-ing rate of 1.0 × 10−4. We trained the model for5 epochs with the CheXpert dataset (Irvin et al.,2019).

4https://www.nltk.org/5https://physionet.org/content/mimic-cxr-jpg/2.0.0/

Subsequently, we evaluated the image diagno-sis module with the CheXpert dataset. To evalu-ate the accuracy of image classification correctlyfor the infrequent labels, we performed a 5-foldcross-validation. Table 7 presents F-scores for eachfinding labels evaluated in 5-fold cross-validation.Although the F-scores of the no-mention labels arehigh, the F-scores of the positive, negative, anduncertain finding labels are relatively low. This isbecause the CheXpert dataset is significantly imbal-anced, and almost all finding labels in the trainingdata are in the no-mention category.Data-to-Text Module For the JCT and MIMIC-CXR datasets, we trained the data-to-text modulefor 50 and 20 epochs, respectively. We used a CRSscore of the validation data as the stopping criteria.Finally, we reported evaluation scores that achievedthe highest CRS score on the validation data. Ta-ble 6 presents hyperparameters used to train ourmodels. Before we trained the model with RL, wepretrained the model with only cross-entropy lossfor an epoch. The number of parameters of thedata-to-text module was 127k for the JCT datasetand 463k for the MIMIC-CXR dataset.Reconstructor Module To train the reconstruc-tor for the JCT dataset, we used the pretrainedJapanese BERT model 6. We have split the train-ing data of the data-to-text module into 4:1 andused the former part as training data and the latterpart as validation data for the reconstructor. Forfine-tuning, we used the AdamW optimizer with alearning rate of 2.0× 10−5 for the BERT layer and2.0× 10−3 for the fully connected layer. We usedbinary cross-entropy loss to train the model, and ap-plied Class Balanced Loss (CBL) (Cui et al., 2019)with β = 0.999. The number of parameters ofthe reconstruction module is 110M. We fine-tunedthe model with 10 epochs, and the F-score on thevalidation dataset was 90.3.

To train the reconstructor for the MIMIC-CXRdataset, we use the pretrained bert-base-uncasedmodel. We also verified the BioBERT model (Leeet al., 2020), but the results showed no significantdifferences with the bert-base-uncased model. Forfine-tuning, we used the AdamW optimizer with alearning rate 2.0 × 10−5 for the BERT layer and2.0× 10−3 for the fully connected layer. By anal-ogy with the JCT dataset, we have split the trainingdata into 4:1 and used the former part as the train-ing data and the latter part as the validation data

6https://github.com/cl-tohoku/bert-japanese

2235

Labels Negative Positive Uncertain No MentionNo Finding - 0.468 - 0.907Enlarged Cardiomediastinum 0.436 0.197 0.040 0.858Cardiomegaly 0.209 0.525 0.013 0.873Lung Opacity 0.002 0.696 0.000 0.602Lung Lesion 0.150 0.246 0.092 0.936Edema 0.223 0.615 0.254 0.740Consolidation 0.489 0.215 0.254 0.740Pneumonia 0.008 0.163 0.278 0.883Atelectasis 0.002 0.333 0.325 0.713Pneumothorax 0.458 0.513 0.000 0.770Pleural Effusion 0.524 0.759 0.036 0.639Pleural Other 0.335 0.217 0.165 0.963Fracture 0.234 0.207 0.007 0.890Support Devices 0.046 0.844 0.007 0.771Overall F1-Score 0.240 0.428 0.103 0.807

Table 7: Evaluation of the image diagnosis module for each finding label. All scores are measured by F-score in5-fold cross validation.

Dataset JCT MIMIC-CXROptimizer AdamW AdamWLearning rateof BERT layer 2.0× 10−5 2.0× 10−5

Learning rateof FC layer 2.0× 10−3 1.0× 10−4

CBL β (Cui et al., 2019) 0.999 0.999Warm up steps 200 200

Table 8: List of hyperparameters of the reconstructormodules.

for the reconstructor. We used binary cross-entropyloss to train the model, and applied Class BalancedLoss (CBL) (Cui et al., 2019) with β = 0.999. Thenumber of parameters of the reconstruction mod-ule was 109M. We fine-tuned the model with 10epochs, and the F-score on the validation datasetwas 97.9.

We used an Intel Core i7-6850K CPU andNVIDIA GTX 1080Ti GPU for training on theJCT dataset, and the training time was approxi-mately 3 h. We used an Intel Xeon Gold 6148CPU and NVIDIA Tesla V100 GPU for training onthe MIMIC-CXR dataset, which required approxi-mately 12 hours.

A.3 Evaluation Settings.

We use an approximate randomization test 7 toevaluate the statistical significance.Evaluation Metrics on the JCT Dataset. For au-tomatic evaluation on the JCT dataset, we usedBLEU (Papineni et al., 2002), F-scores of ROUGE-L (Lin, 2004), and CRS as metrics. We usedNatural Language Toolkit 8 to calculate BLEU

7https://github.com/smartschat/art8https://www.nltk.org/

scores, and the ROUGE Python library 9 to cal-culate ROUGE-L scores.Evaluation Metrics on the MIMIC-CXRDataset. For comparison with the previous imagecaptioning approaches, we used BLEU-1, BLEU-2,BLEU-3, and BLEU-4 metrics calculated by thenlg-eval 10 library. However, word-overlap basedmetrics, such as BLEU, fail to assume the factualcorrectness of generated reports. We compared thelabels assigned in CheXpert Labeler between thegenerated reports and gold reports to calculate theCheXpert accuracy, precision, micro F-score, andmacro F-score. The micro F-score was obtainedby the overall numbers of true positives, falsepositives, and false negatives. The macro F-scorewas obtained by the average of F-scores perclass label. Although the micro F-score neglectsinfrequent labels, the score is significantly biasedby the imbalanced distribution of the test dataset.

Note that precision and F-score are preferred toevaluate the clinical correctness of the reports inCheXpert. In contrast, CheXpert accuracy does notquantify the clinical correctness of the generatedreports adequately. The imbalanced dataset resultsin an excessive number of true negatives rather thantrue positives. Hence, CheXpert accuracy overesti-mates the clinical correctness of generated reportsif the reports comprise many descriptions that arenot related to the findings.Modification Flow We apply the modification pro-cess to the image diagnosis module result with theparameters of (plowth , phighth ) = (0.1, 0.9) for thepositive finding labels. However, we regard all neg-

9https://github.com/pltrdy/rouge10https://github.com/Maluuba/nlg-eval

2236

ative and uncertain labels predicted by the imagediagnosis module as unreliable. This is becausenegative or uncertain findings are highly dependenton the radiologist’s judgment.