Your name here

Introduction ResultsBreast cancer is the second most prevalent cancerworldwide, with nearly 2 million new cases each year.Histopathology evaluation, which involves pathologists’visual assessment of microscopic tissue slides, isindispensable for the diagnosis of breast cancer. However,studies have shown that the billions of pixels from the slidescans contain a wealth of untapped biological signal that hasyet to be interpreted. In this project, we combineconvolutional neural network (CNN) based image analysis ofthese scans with transcriptomic analyses to uncover novelmolecular and morphological profiles associated withhormone receptor status and genomic subtypes.

Methods• Data from 1099 BRCA patients was collected from the

TCGA cohort.

• Whole slide images were tiled at their densest location.

• 200 256x256 tiles per slide were used.

• Caffe5 was used with the Inception V3 CNN to predictclinical/genomic parameters.

• Tumor vs. Normal

• Hormone Receptor Status (ER, PR, HER2, +/-)

• Histological Status

• PAM50 Status

• Rotations were used for balanced tile-level training sets.

• Models were initialized without pretrained model weights,while hyperparameters were determined through transferlearning from lung cancer slides2.

• Held-out cross validation was conducted.

Hormone Receptor Status:

• ER/PR status is normally determined with the aid ofantibodies or immunohistochemistry stains, as opposed todirectly from an H&E stained slide.

• Ridge regression models to distinguish receptor statuswere trained using RNA-seq gene expression profiles.

• Gene ontology enrichment analysis over Ridge-identifiedgenes was conducted.

• Transcriptomics-based immune scores were calculated foreach patient1,4.

• Lymphocyte detector: CellProfiler6 modules trained over aset of slides labeled and segmented by a pathologist3.

• Patients were scored by computing the fraction oflymphocytes to cells for all slides.

• Nuclei/lymphocyte masks created for the labeled slideswere co-localized with thresholded convolutional filters.

• Classifier accuracy over patients with discordant hormonereceptor status was specifically examined.

PAM50 Status:

2 models were constructed to classify PAM50 status: (i) asingle 4 class classifier and (ii) three binary classifiersarranged in a tree.

Luminal?

Luminal A vs B

Basal vs HER2

• The penultimate feature vector ofthe Luminal A/B binary classifier wasextracted, and its correlation withmircroarray expression wasexamined.

• 4 Class Model:

• Distributions of reported confidence associated withcorrectly/incorrectly classified patients were compared.

• t-SNE was conducted over microarray data for PAM50genes, predictions were compared with the resultingclusters.

• The CNN-based classifiers showed strong performanceover every examined task.

• Our CNNs replicated the strong tumor vs. normalperformance described by Liu, et al8 in the TCGA cohort.

• Our analysis extends to histological status, hormonereceptor status, and PAM50 status for the first time.

• To correlate strong CNN predictive performance withgenomic and morphological features, we investigated (i)ER and PR status and (ii) PAM50 status in greater detail.

Hormone Receptor Status:

• Hormone receptor classifiers had 92.6% accuracy overpatients with discordant receptor status (ER+/PR- or ER-/PR+).

• Implies that ER/PR-specific knowledge was learned byclassifiers.

• Ridge regression over RNAseq expression had AUCs of0.91/0.838 for determining ER/PR status (+/-) respectively.

• Genes associated with ER/PR status were enriched inimmune-related terms, suggesting that the morphologicalsignal might come from immune infiltration.

• 56% and 64% of significant GO terms were immune-related for ER and PR status respectively.

• Terms included innate immune response, defenseresponse to bacterium, and regulation of STAT protein.

• The Random Forest trained lypmhocyte detector had 85%true positive/negative accuracy rate over pathologistlabeled slides.

• A thresholded convolutional node in the ER classifier co-localized preferentially with lymphocytes (R = 0.55263, p <0.01) compared to cell nuclei (R = 0.035364, p < 0.01).

• In 97/99 analyzed tiles, co-localization between theconvolutional filter and the lymphocytes was strongerthan the co-localization with the nuclei.

• This implies that learning of morphology resemblinglymphocytes took place.

• Do visible lymphocytes have enough signal to distinguishhormone receptor status in H&E stained slides?

• The lymphocyte detector score distribution wascompared between positive and negative receptorstatus patients.

• Scoring was validated against two separate geneexpression based methods for computing immuneinfiltration (Yoshihara, et al., Li, et al., respectively).

• Pathologic stage was used as a negative control.

ClassificationTask TestSetAUC

TumorvsNormal 0.998

HistologicalStatus 0.974

ERStatus 0.976-0.987

PRStatus 0.930

HER2Status 0.853

PAM50:Luminal? 0.817-0.924

PAM50:LuminalAvsB 0.860-0.939

PAM50:BasalvsHER2Enriched 0.760-0.850

PAM50:4Class 0.663(Accuracy)

Overall(1099)

PR+(819)

PR-(280)

ER+(855)

ER-(244)

Average Age 60.04 60.42 59.05 60.65 57.89StdevAge 13.21 13.18 13.07 13.33 12.30AverageAgeatDiagnosis 59.49 59.89 58.47 60.11 57.33Stdev AgeatDiagnosis 13.22 13.19 13.08 13.33 12.32%PostMenopause 76.05 75.55 77.48 76.32 76.27% ER+ 77.79 97.92 37.40 100 0%PR+ 74.52 100 0 84.27 6.29%HER2+ 22.89 21.83 25.55 22.45 24.22%Lobular (vsDuctal) 21.58 27.68 9.16 26.61 4.00

H&E Stained Slide (lymphocytes w.

green dot)

Lymphocyte Mask

Convolutional Filter

(thresholded)

Density

LymphocyteDetectorScore LymphocyteDetectorScore

ProgesteroneReceptor(+/-) EstrogenReceptor(+/-)

NegativePositive

NegativePositive

KSTestp-val(ER)

KSTestp-val(PR)

BootstrapΔmeanp-val(ER)

BootstrapΔmeanp-val(PR)

Lymphocyte Detector <2.2e-16 3.997e-15 <1e-05 <1e-05

ESTIMATE ImmuneInfiltration(Yoshihara,etal.)

5.371e-08 2.141e-03 <1e-05 3.40e-03

TIMERImmune Infiltration(Li,etal.)

3.754e-07 1.396e-02 <1e-05 1.43e-02

PathologicStage**discontinuous distribution

N/A N/A 0.2509 0.9812

• All immune related distributions had statistically signficantdifferences between receptor status positive/negative.

• These results suggest that immune morphology visible inH&E stained slides contributes to the differentiation ofhormone receptor status by CNN-based classifiers.

PAM50 Status:

• The 4 Class PAM50classifier had muchhigher confidence oncorrect classificationscompared to incorrect.

• Leads to a hypothesis:do incorrectly classifiedpatients have moreambiguous status?

• A majority of missclassifications involved classifyingLuminal B patients as HER-2 Enriched/Luminal B.

• Consistent with existing understanding of PAM50:hierarchical clustering over microarray data groupedLuminal B and HER-2 enriched7, while RNA-seq t-SNEclustering was unable to resolve Luminal A from B.

• The Luminal A/B binary classifier was able to separate thetwo classes with high AUC (0.860-0.939).

• t-SNE dimensionality reduction of 6000+ features from theLuminal A/B classifier revealed a composite feature withsignificant correlations with a set of PAM50 geneexpression values.

IncorrectlyClassified

CorrectlyClassified

Density

PAM50ClassificationConfidence

4ClassPAM50Classifier

t-SNE

Dimensio

n2

t-SNE

Dimensio

n2

t-SNEDimension1 t-SNEDimension1

IncorrectlyClassified

CorrectlyClassified

Basal

HER-2Enriched

LuminalA

LuminalB

PAM50TrueClass PAM50ClassifierPerformance

Conclusions

• Morphology associatedwith these genes maybe prominent enoughto identify visually fromH&E stained images.

• CNN based classifiersmay be sufficientlypowered to examineexpression or mutationstatus of certain genes.

Gene Correlation(R) Correctedp-val

EXO1 0.840634288

5.34e-22

UBE2C 0.727297529

9.70e-15

NAT1 0.694928572

2.65e-13

MKI67 0.687046795

5.55e-13

CENPF 0.658904822

6.53e-12

Our work harnesses the recent advancements in CNNs andthe availability of clinical, transcriptomic, and histology datato provide a quantitative approach for combining -omics andhistopathology analyses. We have shown the ability of imageanalysis techniques to classify histopathological slides andlink the classification to a biological motif. This pipeline canbe immediately applied to examine underlying biology ofother breast cancer markers. Since the study was conductedretrospectively on a study cohort, further validation on theclinical utility of our models is needed. Our methods areextensible to other cancers, potentially influencingdiagnostics and the study of microscopic morphologicalaberrations in human cancer.References: 1) YoshiharaK,ShahmoradgoliM,MartínezE,VegesnaR,KimH,Torres-GarciaW,etal.

NatCommun.2013;4:2612.2)YuK,WangF,BerryGJ,RéC,AltmanRB,SnyderM,KohaneIS.(submitted)3)JanowczykA,MadabhushiJPatholInform.2016;7:29.4) LiB,SeversonE,PignonJC,ZhaoH,LiT,NovakJ,etal.GenomeBiol.2016;17(1).5)JiaY,ShelhamerE,DonahueJ,KarayevS,LongJ,GirshickR,etal..ProcACMIntConfMultimed- MM’14[Internet].2014;675–8.6)CarpenterAE,JonesTR,LamprechtMR,ClarkeC,KangIH,FrimanO,etal.GenomeBiol.2006;7(10).7)BernardPS,ParkerJS,MullinsM,CheungMCU,LeungS,VoducD,etal.JClinOncol.2009;27(8):1160–7.8)Liu,J.Gadepalli,K.,Norouzi,M.,Dahl,GE.,Kohlberger,T.,Boyko,A.,Venugopalan,S.,Timofeev,A.,Nelson,PQ.,Corrado,GS.,Hipp,JD.,Peng,L:2017arXiv:1703.02442.

AcknowledgementsK.-H.Y.isaHarvardDataScienceFellow,andWYissupportedbyNIHRuthL.KirschsteinNationalResearchServiceAward#4TL1TR001101-04and5TL1TR001101-05.WethanktheAWSCloudCreditsforResearch,MicrosoftAzureforResearchAward,andtheNVIDIAGPUGrantProgramfortheircomputationalsupport.

Transcriptome-Histopathology Analysis for Breast Cancer ClassificationWilliam Yuan, MChem1, Isaac S. Kohane, MD, PhD1,2,3, Kun-Hsing Yu, MD, PhD1

1Department of Biomedical Informatics, Harvard Medical School, Boston, MA; 2Boston Children’s Hospital, Boston, MA; 3Brigham and Women's Hospital, Boston, MA