Massively Parallel Large-scale Multi-model Simulation of TumorDevelopment

MARCO BERGHOFF, Karlsruhe Institute of Technology, GermanyJAKOB ROSENBAUER, Forschungszentrum Jülich, GermanyALEXANDER SCHUG, Forschungszentrum Jülich, Germany

The temporal and spatial resolution in the microscopy of tissues has in-creased significantly within the last years, yielding new insights into thedynamics of tissue development and the role of the single-cell within it. Athorough theoretical description of the connection of single-cell processesto macroscopic tissue reorganizations is still lacking. Especially in tumordevelopment, single cells play a crucial role in advance of tumor properties.We developed a simulation framework that can model tissue development upto the centimeter scale with micrometer resolution of single cells. Througha full parallelization, it enables the efficient use of HPC systems, thereforeenabling detailed simulations on a large scale. We developed a generalizedtumor model that respects adhesion driven cell migration, cell-to-cell sig-naling, and mutation-driven tumor heterogeneity. We scan the response ofthe tumor development depending on division inhibiting substances such ascytostatic agents.

CCSConcepts: •Computingmethodologies→Massively parallel algo-rithms; Massively parallel and high-performance simulations; Mod-eling and simulation; Distributed simulation; Self-organization; • Appliedcomputing→ Computational biology; Health care information systems;Bioinformatics; • Networks → Peer-to-peer protocols.

Additional Key Words and Phrases: distributed memory, scalable parallelalgorithms, massive-parallel performance

ACM Reference Format:Marco Berghoff, Jakob Rosenbauer, and Alexander Schug. 2019. MassivelyParallel Large-scale Multi-model Simulation of Tumor Development. InProceedings of Supercomputing’19. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 MODELWithin the massively parallel NAStJA framework [Berghoff andKondov 2018; Berghoff et al. 2018], we have implemented a multi-model solver with a cell-geometric resolution for the simulationof tissue growth in general and cancer in particular. We have onthe microscale a cellular Potts model which simulates adhesiondriven cell movements. Simultaneously, this scale limits our spatialresolution. On the most coarse scale, an agent-based model is used,where each cell corresponds to one agent. On this macroscale, thesignals are processed, and cell death and cell division, includingmutations, are simulated. Another model layer lies between thesetwo scales. It represents the mesoscale and ensures signal diffusionthrough the surfaces of the cells. The Fig. 1 shows an overview

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).Supercomputing’19, November 17–22, Denver, CO, USA© 2019 Copyright held by the owner/author(s).ACM ISBN 978-x-xxxx-xxxx-x/YY/MM.https://doi.org/10.1145/nnnnnnn.nnnnnnn

Fig. 1. A two-dimensional schematic diagram shows the three differentmodel levels. The lowest layer, the microscale, represents the cellular Pottsmodel. On the highest layer, the macroscale, an agent-based model, is acting.In between, on the mesoscale, the transport of signals is processed.

of the different model levels. Following the three model layer aredescribed in detail. Each model layer acts on its specific scale.

Microscale. Themodel is based on a regular rectangular grid. Eachvoxel has an integer number represents the cell ID. All voxels withthe same cell ID are assigned to a biological cell. The cellular Pottsmodel is a Monte Carlo method [Graner and Glazier 1992]: Thesystem evaluates under random changes of voxels to the cell ID oftheir neighboring voxel. These changes are accepted according to aMetropolis acceptance criterion. A Hamiltonian defines the energyof the system. It is the sum of several energy functions, reads,

HCPM =∑i ∈Ω

∑j ∈N (i)

Jτ (ςi ),τ (ς ′j )(1 − δ (ςi , ς′j ))

+ λv∑i ∈Ω

(v(ςi ) −V (τ (ςi )))2

+ λs∑i ∈Ω

(s(ςi ) − S(τ (ςi )))2.

(1)

Here, the first sum corresponds to the cell adhesion, the second tothe volume, and the last corresponds to the surface of individualcells.

Mesoscala. On this scale, the diffusion of signals is simulated.Signals can be among others oxygen, nutrient, or drugs. Based on

1

Supercomputing’19, November 17–22, Denver, CO, USA Berghoff, M, Rosenbauer, J., and Schug, A.

0 50 100 150 200 250 300x in µm

0

50

100

150

200

250

300

yinµm

Fig. 2. Diffusion of the nutrient Signal, the four black squares are blood ves-sels, that supply the nutrient. Yellow cells have high nutrient concentrationsand blue ones low.

the surface with neighboring cells, the signals diffuse from one cellto the other, see Fig. 2.

Macroscale. At the most coarse level, each cell is assigned toan agent. The individual agents are then responsible for signalprocessing or cell death and cell division, including cell mutation.

2 SCALINGThe code is implemented within the NAStJA framework so that itcan benefit directly from the excellent scalability. The domain isdivided into subdomains and distributed to the MPI ranks. Each MPIrank then holds a block containing the field on the microscale withthe cell IDs and additional cell data, which are held for the higherscales, per cell. After each calculation sweep, a halo exchange tothe first 26 neighbor blocks is used for the field of cell IDs. Initially,the additional data of the cells must be exchanged globally, sinceit is not known where the cells are located. That would result in anon-scaling collective communication. With the domain knowledgethat the size of blocks can be chosen and that blocks are typicallylarge compared to the small cells, it can be ensured that a cell isnever in more than eight blocks, e.g., two per dimension, at thesame time, compare Fig. 3. With this knowledge, it can be ensured

a) b)

Fig. 3. A cell (blue) is distributed to several blocks. (a) shows an inadequatedistribution and (b) shows an allowed distribution.

that the bookkeeping of the additional data can be handled by localcommunications to the next 26 neighbors. The efficiency of thisexchange is excellent, as shown in Fig. 4(a). Fig. 4(b) right shows anefficiency of > 80 percent for blocks with an edge length of 100 andeven an efficiency of > 90 percent for an edge length of 200. Thisresult was measured on Jureca with up to 256 nodes correspondingto 6144 processes.

1 4 16 64 256

0

0.2

0.4

0.6

0.8

Nodes/20 cores

Ecien

cyη

400 B8 kB

800 kB

(a) 1

Fig. 4. (a) The efficiency of the exchange of the additional data on ForHLRII and (b) the efficiency for the whole simulation on Jureca, using up to 256nodes with 24 cores each. It is shown for a subdomain distribution with ablock edge length of 100 and 200.

0

1

2

3

4

5

6

VolumeinVoxels

1e6

0 25 50 75 100 125 150 175 200 225 250 275 300 325Time in days

Treatment

ChemotherapyRadiation

0

1

2

3

4

5

6

VolumeinVoxels

1e6

0 25 50 75 100 125 150 175 200 225 250 275 300 325Time in days

Treatment

ChemotherapyRadiation

0

1

2

3

4

5

VolumeinVoxels

1e6

0 25 50 75 100 125 150 175 200 225 250 275 300 325Time in days

Treatment

ChemotherapyRadiation

Fig. 5. Different treatment plans of chemotherapy and radiation, single orin combination. The colors represent different cell types.

3 APPLICATIONBesides, we show the application of heterogeneous cancer growth.We have filled a domain measuring 1000 × 1000 × 1000 voxels withone million cells, each cell has a volume of about 1000 voxels. Inaddition, blood vessels that transport nutrient and a small nucleusof cancer cells was placed in the middle of the simulation. Cancerproliferates and grows through cell division until it is too far awayfrom the nutrient supply of the blood vessels. Thereby, the cells canmutate into one of the 30 predefined cell types.

To research personalized medicine and better treatments, we haveadded various additional treatments.

Chemotherapy. inserts a drug over the blood vessels that changesthe cell division rate.

Radiation therapy. reduces the division rate globally and immedi-ately leads to dying some cells.With these simulations, treatments can then be played through

and the best treatment plan can be found for the given tumor hetero-geneity. This is illustrated in Fig. 5, where chemotherapy or radiationtherapy individually do not lead to success, but a combination does.

REFERENCESMarco Berghoff and Ivan Kondov. 2018. Scalable Global Network Based on Local Non-

Collective Communications. SC Workshop: Latest Advances in Scalable Algorithmsfor Large-Scale Systems (ScalA’18) (2018).

Marco Berghoff, Ivan Kondov, and Johannes Hötzer. 2018. Massively parallel StencilCode Solver with Autonomous Adaptive Block Distribution. IEEE Transactions onParallel and Distributed Systems 29, 10 (Oct 2018), 2282–2296.

François Graner and James A. Glazier. 1992. Simulation of biological cell sorting using atwo-dimensional extended Potts model. Physical Review Letters 69, 13 (1992), 2013.

Received July 2019

2