Multi-Scale, Multi-Modal Fusion of Histological and MRIVolumes for Characterization of Lung Inflammation

Mirabela Rusua,*, Haibo Wanga,*, Thea Goldenb, Andrew Gowb, and Anant Madabhushia

aBiomedical Engineering, Case Western Reserve University, Cleveland, OH;bDepartment of Pharmacology and Toxicology, Rutgers University, Piscataway, New Jersey;

ABSTRACT

Mouse lung models facilitate the investigation of conditions such as chronic inflammation which are associatewith common lung diseases. The multi-scale manifestation of lung inflammation prompted us to use multi-scaleimaging - both in vivo , ex vivo MRI along with ex vivo histology, for its study in a new quantitative way.Some imaging modalities, such as MRI, are non-invasive and capture macroscopic features of the pathology,while others, e.g. ex vivo histology, depict detailed structures. Registering such multi-modal data to the samespatial coordinates will allow the construction of a comprehensive 3D model to enable the multi-scale study ofdiseases. Moreover, it may facilitate the identification and definition of quantitative of in vivo imaging signaturesfor diseases and pathologic processes. We introduce a quantitative, image analytic framework to integrate invivo MR images of the entire mouse with ex vivo histology of the lung alone, using lung ex vivo MRI as conduitto facilitate their co-registration. In our framework, we first align the MR images by registering the in vivo andex vivo MRI of the lung using an interactive rigid registration approach. Then we reconstruct the 3D volume ofthe ex vivo histological specimen by efficient groupwise registration of the 2D slices. The resulting 3D histologicvolume is subsequently registered to the MRI volumes by interactive rigid registration, directly to the ex vivoMRI, and implicitly to in vivo MRI. Qualitative evaluation of the registration framework was performed bycomparing airway tree structures in ex vivo MRI and ex vivo histology where airways are visible and may beannotated. We present a use case for evaluation of our co-registration framework in the context of studyingchronic inflammation in a diseased mouse.

Keywords: Mouse lung model, in vivo MRI, ex vivo MRI, branchi, histopathology, inflammation, chronic

1. INTRODUCTION

Often multi-modal images including MRI and CT allow for characterizing the lung pathology across differentscales. However, multi-modal in vivo imaging needs to be closely and tightly integrated with histopathologyin order to be able to define and characterize imaging signatures of inflammation and other lung diseases.Additionally it is desireable to be able to merge the in vivo imaging and ex vivo pathology into an integratedcanonical framework which would provide a unified, integrated multi-scale model for disease characterization.Such an integrated model would be valuable to characterize in vivo imaging signatures of diseases.1–3 Recentlythere has been great interest in developing fused multi-modal multi-scale models by combining ex vivo and invivo imaging and pathology data to study diseases, such as prostate cancer1,4–6 and the brain.7,8

Mouse models facilitate the study of various pulmonary diseases, e.g. chronic inflammation.9 Lung inflam-mation is a condition that can be visualized at different imaging scales from the thickening of the airways wallsthrough the accumulations of foamy appearing alveolar macrophages.10 Methods to investigate in vivo lunginflammation currently used include micro-CT11 as well as transillumination and fluorescence molecular to-mography12 (see13 for a review). However, none of these studies involved a precise co-registration and fusionwith ex vivo histopathology. With pre-clinical models (e.g. mouse) there is the opportunity to quantitativelyand rigorously evaluate imaging features of pathology by correlating the in vivo imaging attributes with exvivo histopathology on a voxel-by-voxel basis. For example, in vivo MRI shows macroscopic characteristicsin three dimensions (3D), while histology captures higher level of details in two dimensions (2D) allowing thevisualization of bronchi and their divisions within the lung.

MR and HW contributed equally to this work. Corresponding author: AM (anant.madabhushi@case.edu)

Multi-modal registration is however encumbered by a few significant challenges. Also, ex vivo histologyimages often suffer from artifacts caused by fixation and sectioning. Additionally in vivo imaging might sufferfrom deformation induced by the presence of other organs such as the heart. Consequently the multi-modalimage registration algorithms need to be able to deal with the various sources of deformation induced within thedifferent modalities.

While registration of in vivo imaging and ex vivo pathology images have been previously presented in theliterature, the different approaches employed are typically a function of the type of pathology data available. Forinstance if only a few pathology sections are available, the only option to co-register with the in vivo imagingmight be to first determine slice correspondences between the two modalities. This can then be followed by 2D-2Dco-registration methods.4,5 Alternately if multiple concurrent histologic sections are available there might be anopportunity to reconstruct histologic volumes14–17 which could then allow for a true volumetric co-registrationwith the in vivo imaging.6 Such 3D-3D registration has the advantage of introducing additional spatial constrainsto facilitate the accurate alignment. Also 3D-3D registration approaches for MRI and histology typically do notrequire establishment of slice correspondences from an expert.

In this work we introduce a co-registration framework that allows the integration of in vivo MRI and exvivo histology data for the investigation of chronic inflammation in the lung. Our framework involves the co-registration of in vivo MRI, which shows the lung in situ, with ex vivo digitized histopathology of the entirelung. The 2D histological slices are collected from the excised lung at very fine intervals and are stained withhematoxylin and eosin (H&E) to capture inflammation at high resolution. Such high level of detail is needed,particularly in the case of inflammation, to visualize and characterize the pathology to be studied at a very fine,granular length scale. The inflammation may be manually annotated by a pathologist on each individual sliceand serves as ground truth that is then mapped via our co-registration and image analytic framework on thecorresponding in vivo MRI. Preceding the co-registration is a three dimensional (3D) reconstruction performedon the H&E slices to create the corresponding volume of the histology specimen.17 In addition to the in vivo MRIand histology slices, ex vivo MRI was also collected to visualize smaller structure, particularly airways, that maynot be discernible in vivo; hence, the ex vivo MRI is used by the framework as a conduit to facilitate the accurateco-registration of in vivo MRI and corresponding ex vivo histology specimens. Our fusion framework allows usto construct a 3D imaging multiscale model of the lung, allowing us to quantitatively characterize inflammationon the in vivo imaging.

Our work introduces several novel contributions. (1) We present a fusion framework to reconstruct multi-scale3D model of the mouse lung by integrating ex vivo histology with in vivo MRI. The solution is robust and allowsto potentially include additional imaging modalities, such as micro CT or PET. (2) The multi-scale, multi-lengthscale model allows for quantitative identification of in vivo imaging signatures of chronic inflammation in apre-clinical setting for the first time.

The remainder of the paper is organized as follows. The experimental design section describes first thedata collected in the current study followed by the detailed description of the framework including the 3Dreconstruction. Next, the results of the fusion are presented for each step in the framework. Finally, we discussour results and present future directions.

2. EXPERIMENTAL DESIGN

2.1 Data Acquisition

In this work, two mice were considered: a mouse that lacks the surfactant protein D (SP-D) showing lunginflammation, and the wild type mouse without inflammation. Utilizing the Aspect MRI, the in vivo MRI datawas collected on the peak inspiratory volume, which allows for the reconstruction of the in situ lung volume.Upon completion of the MRI, the mouse lung was extracted and inflation fixed with 4% paraformaldehyde. Thefixed lung, while still in suspension within fluid, was examined by MRI using a 6-hour data collection window,which allows for a much higher resolution image to be gathered. The fixed lung was then embedded in paraffinand 5 µm sections were cut with a spacing of 55 µm (Figure 3). These sections were stained with Hematoxylinand Eosin (H&E) and examined using the Olympus VS120-SL scanning microscopy system.

Our mouse model is constructed using various images.

(g) In vivo - Ex vivo

MRI fusion

Lung

Segmentation

(c) Histological Slices

3D

Reconstruction

(f) 3D Reconstruction

(h) Ex vivo MRI -

3D histology fusion

(i) 3D Multi-Scale

Lung Model

Lung

Segmentation

Rigid

Registration

Rigid

Registration(b) Ex vivo MRI (e) Ex vivo Lung

(a) In vivo MRI (d) In vivo Lung

Figure 1. Fusion framework: (a) in vivo MRI, (b)ex vivo MRI and (c) histological slices are sequentially acquired. (d-e)Thelung is manually segmented in MRI, (f) a 3D volume is reconstructed from the histology slices to create a volume of thehistopathology. (g) The ex vivo MR volume is aligned to the in vivo MR volume. (h) The histology volume is registerto the ex vivo MRI using rigid transformations. (i) Finally, the lung model is obtained that captures multi-scale detailsfrom the different modalities.

1. Whole body in vivo MRI has the lowest resolution and shows deformation of the lung due to thepresence of other organs, e.g. heart or liver.

2. Ex vivo MRI shows the ex vivo lung at medium resolution allowing the visualization of the trachea andlarge bronchi, yet might show artifacts due to the sample preparation.

3. Ex vivo histopathology slices of the entire lung are imaged at high resolution; they may suffer fromimaging artifacts due to sample preparation, e.g. folding and shrinking.

2.2 Annotation of pathology for ground truth extent determination

The ground truth extent for inflammation is determined via manual annotation on the ex vivo digitized histologicimage sections. Inflammation is visible not only in the airways that are expected to thicken, but also at thelevel of the alveoli.10 Accurate delineation of inflammation is needed, such that a 3D map of inflammation canbe identified. Accurate registration of the histology slices is thus needed to ensure a meaningful 3D map of theinflammation onto in vivo MRI.

2.3 Annotation of airways for qualitative validation

Along with the extent of inflammation, large airways are also delineated on each slice using a semi-automaticthresholding based approach. The segmentation of the large airways provides us the opportunity to (a) quali-tatively evaluate the reconstruction results as well as to (b) guide the interactive registration of the histologicvolume with the ex vivo MRI. Specifically, we used the airways to assess the tissue shrinkage caused by the slidepreparation and staining.

1 2 31111 4 5 n...

(a)

1 2 31111 4 5 n

...

...

(b)

Figure 2. One-to-one vs One-to-many registration. Each triangle symbolizes an H&E slice, while the number representsits order in the stack. Black arrow represents the registration direction, i.e. arrow heading on the template image.The difference between the one-to-one and one-to-many registration approaches is in the number of templates employed.Only one template is required in (a) allowing for fast piecewise image registration but may cause registration errors topropagate. On the other hand, groupwise registration in (b) uses multiple templates, requiring additional computationsyet providing better global accuracy.

2.4 Registration of ex vivo MRI to in vivo MRI

First, the lung was manually segmented in both ex vivo MRI and the in vivo MRI. Then, the lung segmented inex vivo MRI was rigidly aligned to the lung segmented in in vivo MRI using 3D Slicer.18 In our framework, thein vivo MRI is the reference fix image to which all other imaging protocols are aligned.

2.5 Reconstruction of the 3D histological volume

Several steps are employed in order to reconstruct the histological volume from the 2D histological slices. First,we apply one-to-one piecewise19–21 registration to generate an initial alignment of the slices which will be furtheroptimized by the one-to-many groupwise approach.17 Given a set of 2D histological slices, adjacent slices areiteratively registered until all slices are aligned (Figure 2(a)). The alignment of two slices is achieved bymaximizing the mutual information using rigid transformations. This piecewise step is able to quickly align thelarge number of slices corresponding to the lung. However with piecewise approaches, registration errors maypropagate between slices resulting in slice drifting.

The groupwise step eliminates the drifting by considering more than one section as a template. Here we applyone-to-many groupwise registration.17 Given the set of slices aligned via the piecewise approach, we choose oneslice that is most similar to the mean shape as an initial template and regard all others as moving slices to beregistered (Figure 2(b)). A moving slice is first selected and aligned to the template such that the mean distancebetween the template and moving image is reduced (Algorithm 1). Next, the transformed moving slice is addedto the template stack. Then, the next moving slice is selected and aligned to all the template slices. The processiterates until all moving slices have been aligned. The metric optimized here is the sum of squared differencesbetween slices. We adopt the efficient Gauss-Newton22 optimization approach to minimize the metric. Since wealign one moving slice to all previously aligned slices, our approach keeps the global consistency between slicesduring the registration.

2.6 Registration of the Histology Volume with ex vivo MRI

The 3D histology lung was aligned with the ex vivo MRI via interactive rigid registration using the outlinedairways of the two modalities as landmarks to guide the alignment. Through the registration of the 3D histologylung and ex vivo MRI, it became apparent that the histogical specimen has suffered considerable shrinkage duringsample preparation. We assessed the amount of shrinkage by iteratively updating the scaling factor that wasapplied to the histological volume. For each scaling factor, the alignment of the airway tree structure in theex vivo histology volume and ex vivo MRI was uses to qualitatively estimate of the registration. In this work,we found a scaling factor of 2 to allow the best alignment between the histology volume and ex vivo MRI. In

Data: a set of N unaligned slices;Result: aligned images;Init: Pick the first slice as template;for each unaligned slices do

Pick one as the moving image while not converged docompute a small transformation displacement via gradient decent;add the displacement to the computed transformation;warp the moving image with the computed transformation;

endforeach unaligned slices do

warp the slice with the computed transformation;endadd the aligned moving image as a new template;

end

Algorithm 1: Summary of our groupwise registration approach

the future, we consider acquiring additional data such as block face image of the histological sections before thesample preparation to accurately assess the scaling factor.6

Following these steps, the ex vivo histology specimen is aligned to the in vivo MRI. Such alignment enablesthe mapping of lung inflammation (annotated on histology slices) on to the in vivo MRI.

3. RESULTS AND DISCUSSION

Qualitative results are shown for each step of the framework (Figures 3-6).

3.1 Histology Volume Reconstruction

The groupwise registration allowed the building of the 3D volume representing the lung from the histopathologysections (Figure 3(b)). A careful inspection of the reconstruction reveals that the approach was able to align theslices without causing significant drifting between slices. Such drifting is visible when misalignment errors arepropagated between consecutive slices and the model progressively rotates. Moreover, a fairly smooth surfaceof the volume suggests that the groupwise registration was able to properly identify the rigid transformationsthat were required to align the slices. Occasional minimal misalignment may be observed at the edges of thereconstruction, yet they are caused by the folding of the underlying tissue during the histology preparation. Wealso evaluate the alignment of slices by visual inspection of the airways (colored red in Figure 3(c)). When theproper transformation was identified, the airways were closely aligned by reconstructing the tree like structure.

3.2 Registration

3.2.1 Histology - ex vivo MRI registration

Following 3D reconstruction, the histological volume was registered to the ex vivo MRI, using iterative scalingfollowed by interactive rigid registration. Figure 4 shows the results of the histology volume fusion with exvivo MRI. Although the airways align fairly well between the two modalities (Figure 4(b)), it may be observedthat the lung surfaces appear misaligned between the ex vivo MRI and the corresponding 3D histologic recon-struction (green and purple surfaces in Figure 4(a)). Such misalignment is in part due to the imaging artifactsinduced during the histology preparation, e.g. tissues shrinkage, and in the ex vivo MRI acquisition, e.g. largedisplacement of the lobes due to the fixation protocol.

...

(a) (b) (c)

Figure 3. Illustration of the individual of 2D histology slices (a) and corresponding 3D reconstruction(b-c); the entirelung is sectioned into individual slices and stained with H&E. Three non-adjacent slices resulting from the sectioning aredisplayed in (a). Following the groupwise registration17 of the individual histologic sections a 3D reconstruction of thelung is obtained (purple outline in (b)-(c)). The airways segmented on the histology slices are aligned in 3D following theregistration protocol indicating a smooth contiguous alignment of the underlying histology slices (red in (c)).

(a)

90O

(b)

Figure 4. Fusion of ex vivo MRI with the reconstructed 3D histological volumen (a) Overlay of the ex vivo MRI (green)with the 3D reconstructed histologic volume (purple) following rigid registration. The airways of the two modalitieswere used to identify the scaling of the histology to correct for the shrinkage caused by sample preparation. (b) Airwayalignment between the ex vivo MRI (blue) and histology (red) reveals a relatively accurate overlay.

3.2.2 In vivo and ex vivo MRI registration

The segmented lungs on the in vivo and ex vivo MRI were aligned using interactive rigid registration in 3DSlicer.18 Anatomic landmarks such as the trachea were utilized to guide the alignment. Overall, the shape ofthe lung was preserved between the two acquisitions (yellow versus green surfaces in Figure 5), allowing for agood alignment between the different lung surfaces.

Following the registration of ex vivo to in vivo MRI and subsequently co-registering the ex vivo MRI and theex vivo histology, all 3 modalities were co-registered to a common reference frame allowing for the constructionof a 3D imaging model of the lung in mice (Figure 6).

(a) (b)

Figure 5. Fusion of in vivo MRI with ex vivo MRI seen from different viewing angles, (a) front view and (b) tilted. Outlineof the lung and mouse from in vivo MRI are shown in yellow, while the ex vivo MRI lung is shown in green. The mainairways segmented from ex vivo MRI are depicted in blue.

(a) (b) (c)

Figure 6. 3D Multi scale model of the mouse lung; The 3D reconstruction of the histology slices are shown in purplewhile the airways segmented from the later showed in red. The ex vivo MRI airways are represented in blue, while the invivo MRI is shown in yellow. (a-c) different structures are made visible or invisible to show the fusion results.

4. CONCLUDING REMARKS

In this work, we present a unified imaging framework for building a 3D imaging model of the lung with the goalof identifying imaging signatures of lung inflammation in vivo. The new framework was utilized to co-registerthe in vivo MRI with histology using the ex vivo MRI as conduit to facilitate the data fusion. The frameworkwas applied here to create a 3D imaging model of a SP-D knock out mouse which shows lung inflammation,thus allowing for the mapping of regions of inflammation onto the corresponding in vivo MRI. Registrationmethods such as described in this work are required as lung inflammation shows a discontinuous distribution23

within the lung. In future work we intend to further develop our framework to allow incorporation of non-lineartransformations and allowing for independent transformations for both sides of the lung. Moreover, we hope toincrease the size of our cohort and utilize fully automatic registration approaches to integrate the multi-modaldata. The co-registration framework will allow for direct measurement of changes that occur with inflammation,thus allowing for identification and evaluation of in vivo imaging signature of inflammation.

5. ACKNOWLEDGEMENTS

This work was made possible by grants from the National Institute of Health (R01CA136535, R01CA140772,R43EB015199, R21CA167811), National Science Foundation (IIP-1248316),the QED award from the UniversityCity Science Center and Rutgers University, and HL086621.

REFERENCES

[1] Viswanath, S. E., Bloch, N. B., Chappelow, J. C., Toth, R., Rofsky, N. M., Genega, E. M., Lenkinski,R. E., and Madabhushi, A., “Central gland and peripheral zone prostate tumors have significantly differentquantitative imaging signatures on 3 tesla endorectal, in vivo T2-weighted MR imagery,” J. Magn. Reson.Imaging 36, 213–224 (2012).

[2] Segal, E., Sirlin, C. B., Ooi, C., Adler, A. S., Gollub, J., Chen, X., Chan, B. K., Matcuk, G. R., Barry, C. T.,Chang, H. Y., and Kuo, M. D., “Decoding global gene expression programs in liver cancer by noninvasiveimaging,” Nature biotechnology 25(6), 675–680 (2007).

[3] Gevaert, O., Xu, J., Hoang, C. D., Leung, A. N., Xu, Y., Quon, A., Rubin, D. L., Napel, S., and Plevritis,S. K., “Non–small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public geneexpression microarray datamethods and preliminary results,” Radiology 264(2), 387–396 (2012).

[4] Chappelow, J., Madabhushi, A., Rosen, M., Tomaszeweski, J., and Feldman, M., “Multimodal image reg-istration of ex vivo 4 tesla prostate mri with whole mount histology for cancer detection,” SPIE MedicalImaging 6512, S1–S12 (2007).

[5] Chappelow, J., Bloch, B. N., Rofsky, N., Genega, E., Lenkinski, R., DeWolf, W., and Madabhushi, A., “Elas-tic registration of multimodal prostate MRI and histology via multiattribute combined mutual information,”Med. Phys. 38(4), 2005–18 (2011).

[6] Park, H., Piert, M. R., Khan, A., Shah, R., Hussain, H., Siddiqui, J., and Meyer, C. R., “Registrationmethodology for histological sections and in-vivo imaging of human prostate,” AR 15(8), 1027 (2008).

[7] Dauguet, J., Delzescaux, T., Conde, F., Mangin, J.-F., Ayache, N., Hantraye, P., and Frouim, V., “Three-dimensional reconstruction of stained histological slices and 3D non-linear registration with in-vivo MRI forwhole baboon brain,” in [Journal of Neuroscience Methods ], 164, 191–204 (2007).

[8] Palm, C., Penney, G. P., Crum, W. R., Schnabel, J. A., Pietrzyk, U., and Hawkes, D. J., “Fusion of ratbrain histology and MRI using weighted multi-image mutual information,” 69140M–69140M–9 (2008).

[9] Sheth, V., van Heeckeren, R., Wilson, A., van Heeckeren, A., and Pagel, M., “Monitoring infection andinflammation in murine models of cystic fibrosis with magnetic resonance imaging,” J. Magn. Reson. Imag-ing 28(2), 527–532 (2008).

[10] Atochina, E. N., Beers, M. F., Hawgood, S., Poulain, F., Davis, C., Fusaro, T., and Gow, A. J., “Surfactantprotein-D, a mediator of innate lung immunity, alters the products of nitric oxide metabolism.,” AmericanJournal of Respiratory Cell and Molecular Biology 30(3), 271 (2004).

[11] Jobse, B. N., Johnson, J. R., Farncombe, T. H., Labiris, R., Walker, T. D., Goncharova, S., and Jordana,M., “Evaluation of allergic lung inflammation by computed tomography in a rat model in vivo,” EuropeanRespiratory Journal 33(6), 1437–1447 (2009).

[12] Ntziachristos, V., “Optical imaging of molecular signatures in pulmonary inflammation,” Proceedings of theAmerican Thoracic Society 6(5), 416–418 (2009).

[13] Jannasch, K., Missbach-Guentner, J., and Alves, F., “Using in vivo imaging for asthma,” Drug DiscoveryToday: Disease Models 6(4), 129 – 135 (2009).

[14] Yushkevich, P., Avants, B., Ng, L., Hawrylycz, M., Burstein, P., Zhang, H., and Gee, J., “3D mouse brainreconstruction from histology using a coarse-to-fine approach,” Biomedical Image Registration , 230–237(2006).

[15] Cifor, A., Pridmore, T., and Pitiot, A., “Smooth 3-D reconstruction for 2-D histological images,” in [Infor-mation Processing in Medical Imaging ], 350–361, Springer (2009).

[16] Feuerstein, M., Heibel, H., Gardiazabal, J., Navab, N., and Groher, M., “Reconstruction of 3-D histologyimages by simultaneous deformable registration,” MICCAI , 582–589 (2011).

[17] Wang, H., Rusu, M., Golden, T., Gow, A., and Madabhushi, A., “Mouse lung volume reconstruction fromefficient groupwise registration of individual histological slices with natural gradient,” in [SPIE MedicalImaging ], Accepted (2013).

[18] Pieper, S., Halle, M., and Kikinis, R., “3D Slicer,” IEEE International Symposium on Biomedical Imaging, 632–635 (04 2004).

[19] Bagci, U. and Bai, L., “Automatic best reference slice selection for smooth volume reconstruction of a mousebrain from histological images,” IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1688–1696 (2010).

[20] Pitiot, A., Bardinet, E., Thompson, P. M., and Malandain, G., “Piecewise affine registration of biologicalimages for volume reconstruction,” MIA 10(3), 465–483 (2005).

[21] Schmitt, O., Modersitzki, J., Heldmann, S., Wirtz, S., and Fischer, B., “Image registration of sectionedbrains,” International Journal of Computer Vision 73(1), 5–39 (2007).

[22] Baker, S. and Matthews, I., “Lucas-Kanade 20 Years On: A Unifying Framework: Part 1,” Tech. Rep.CMU-RI-TR-02-16, Robotics Institute (July 2002).

[23] Korfhagen, T. R., Sheftelyevich, V., Burhans, M. S., Bruno, M. D., Ross, G. F., Wert, S. E., Stahlman, M. T.,Jobe, A. H., Ikegami, M., Whitsett, J. A., and Fisher, J. H., “Surfactant protein-D regulates surfactantphospholipid homeostasis in vivo,” Journal of Biological Chemistry 273(43), 28438–28443 (1998).