Tingjun YangLArSoft Coordination MeetingMar 5, 2020

Memory usage in ProtoDUNE simulation

• ProtoDUNE-SP production 3 started in early April. • There are two changes to simulation in this production

– Refactorized larg4 framework– Wirecell detector simulation

• Ken Herner reported a larger memory consumption in the larg4 stage compared with production 2.

Introduction

5/4/20 Presenter | Presentation Title or Meeting Title2

https://indico.fnal.gov/event/24127/contributions/75346/attachments/46951/56389/PDSPProd3_MondayMeeting_20200413.pdf

Peak at 3.3 GB Peak at 4.2 GB

• One big change from Production 2 to Production 3 is that now we save EM shower daughters.

• Understand the memory consumption– TPC simulation– Photon detector simulation– RootIO

• Discuss possible approaches to reduce memory consumption.

• Most materials are prepared by David Rivera and Hans Wenzel. Thank you!

Investigation

5/4/20 Presenter | Presentation Title or Meeting Title3

RootIO

Bulk

Valgrind -- Massif

4

David Rivera

Bulk MemoryTotal : 2.4 GB• 846MB – phot::PhotonLibrary::LoadLibrary() [stl_vector.h:847]• 318MB – larg4::ParticleListActionService::endOfEventAction() [new_allocator.h:111]• 236MB – various places each taking less than 1%• 185MB – larg4::SimEnergyDepositSD::ProcessHits() [new_allocator.h:111]• 185MB – larg4::IonAndScint::produce() [new_allocator.h:111]• 136MB – larg4::ParticleListActionService::endOfEventAction() [vector.tcc:109]• 125MB – larg4::LArG4DetectorService::doFillEventWithArtHits() [new_allocator.h:111]• 55MB – larg4::postUserTrackingAction() (AddPointToCurrentParticle) [vector.tcc:109]• 47MB – larg4::larg4Main::produce() [DataViewImpl.h:725]• 44MB – simOpDetBackTracker::AddScintillationPhotons() [stl_vector.h::1136]• 37MB – phot::PDFastSimPVS::produce() [stl_map.h:499]• 35MB – larg4::ParticleListActionService::preUserTrackingAction() [new_allocator.h:111]• 31MB – llvm::SmallVectorBase::grow_pod() [Cling]• 29MB – larg4::ParticleListActionService::preUserTrackingAction() [stl_map.h:499]

5

David Rivera

Photon library is the biggest memory consumer!

RootIO

In this stage the memory usage grows and peaks at 4.12 GB• The Bulk usage is still present, but additionally we have:• 1.617GB – Tstorage::ReAllocChar() [libCore.so]• 47MB – std::vector<std::pair<art::Refcore, unsigned long>, … ? And an increase to:• 372MB (+136MB) – various places each falling below 1% memory

usage• 55MB (+11MB) –

sim::OpDetBackTrackerRecord::AddScintillationPhotons• 48MB (+11MB) – phot::PDFastSimPVS::produce

6

David Rivera

7

David Rivera Running product_sizes_dumper on a g4 event

Uncompressed

Compressed

• Big containers:simb::MCParticles andsim::SimEnergyDeposit

• Large compression factors for those two.

MCParticles• One, +1GeV event in PDSP• Storing full trajectories only for the (single) primary beam particle and all its

descendants• Storing start and end points for all other particles simulated, including:

• Beam : 237 primaryBackground + descendants (78,185 total)• Radiologicals :

• kr85 (272) + descendants (291 total)• ar42 (1) + descendants (1 total)• rn222 (8) + descendants (8 total)• ar39 (2362) + descendants (2491 total)

• Corsika cosmic generator (753) + descendants (655,973 total)

• 736,949 simb::MCParticles with at least 2 Trajectory points each • > 1,473,898 TLorentzVector

8

David Rivera

9

LArSoft has recommended not using TLorentzVector for a long time:https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/From_ROOT_vectors_(TVector3)_to_ROOT_GenVector

David Rivera

The current SimEnergyDeposit

The current SimEnergy deposit:l we are not really filling numPhotons and numElectrons in Geant4 anymore but in a separate

module (ISCalculationSeparate). So this doesn’t need to be stored here. l trackID and pdgCode are repeated for every step on a trajectoryl two points for every stepl do we need double precison for time? l do we need geo::Point_t for the persistent object (it’s just x,y,z)?

Can we save space and time? Make back tracking more efficient? Be independent of external libraries?

Hans Wenzel

Electron recombination from ISCalculationSeparate (Icarus)

Note: PhotonYield just proportional to dE/dx, not affected by recombination

Hans Wenzel

SimTrajectory as vector of SimSteps

OutPut of sensitive detector (active liquid Ar Volume) Sorted by time need at least 2 SimSteps (start/stop for trajectory)Len can include multiple scattering straggling

Hans Wenzel

Benchmarking (very prelim.)

SimEnergyDeposit SimTrajectory

CPU (user) 16.183 12.742

Space 77Mb 60Mb

Hans Wenzel

• Currently photo library uses 800M+ memory. • Other approaches have been proposed for DUNE:

– Parameterized PD response• https://indico.fnal.gov/event/15812/contributions/33738/attachment

s/21036/26222/photlib_hybrid.pdf– CNN based PD simulation

• https://indico.fnal.gov/event/20403/contributions/57455/attachments/36001/43862/photon_gan.pptx

• We should seriously consider alternative approach for photon detector simulation in ProtoDUNE.

Photo library

5/4/20 Presenter | Presentation Title or Meeting Title14

• Large memory usage increase in ProtoDUNE Production 3 most likely caused by including EM shower daughters.

• There are a few possible ways to reduce memory usage in the simulation– Revamp simb::MCParticle and sim::SimEnergyDeposit.– Brainstorm on photon detector simulation.

Conclusions

5/4/20 Presenter | Presentation Title or Meeting Title15

We're trying to fit a whale in a bathtub. We can put it on a diet but it's still a whale and it's still a bathtub. - Heidi Schellman