opal introduction lecture 1 - psiamas.web.psi.ch/people/aadelmann/pub/ffag14/opal-lect... ·...

OPAL Introduction Lecture 1

Andreas Adelmann

PSI OPAL lectures FFAG Workshop 2014

OPAL Introduction Lecture 1 PSI OPAL lectures FFAG Workshop 2014 / 46

General Introduction

Models in OPALArchitecture3D Space ChargeExamples

Data Post Processing

Output formats of OPALH5rootgnuplotPython scriptsBuilding OPAL

Use Cases

Cyclotron and FFAG simulations

URL of the lectures https://amas.psi.ch/OPAL/wiki/OPAL


https://amas.psi.ch/OPAL/wiki/OPAL

Outline

1 History

2 Models in OPAL for Precise Accelerator Modelling

3 Open-Source Software @ Work: New FFAG modelling capabilities


History

OPAL originated from MAD9p (2001)

MAD9 (Cern)Pooma (LANL)split operator M(s) =Mext(s/2)⊗Msc(s)⊗Mext(s/2) +O(s3)

MAD9 & Pooma died: I refactored, improved and started OPAL

Changed from s to t: OPAL-t

With J.Y. Yang we wrote OPAL-cycl (2009)

With R. Bakker & Y. Ineichen we integrated Bet (HOMDYN)OPAL-envelope (2008)

Developer summit in Ticino 2012: lot of refactoring, C++11

D. Winklehner (MIT/PSI) generalised OPAL-cycl in 2014

Ch. Rogers added FFAG capabilities to OPAL-t in 2013-2014

Version 1.3.0 to be released soon


Outline

1 History

2 Models in OPAL for Precise Accelerator ModellingOPAL in a NutshellParticle Matter InteractionArchitectureA fast Direct FFT-Based PIC Poisson SolverIterative Poisson Solver SAAMG-PCG3D Geometry Handling CapabilityInterfacing to the OutsideSelected Results

3 Open-Source Software @ Work: New FFAG modelling capabilities


Outline

1 History


3 Open-Source Software @ Work: New FFAG modelling capabilitiesResults from ERIT-FFAG (energy/emittance recovery internaltarget) Tracking StudiesResults from 330 MeV to 1 GeV ns-FFAG Design Studies


OPAL in a Nutshell I

OPAL is an open-source tool for charged-particle optics in largeaccelerator structures and beam lines including 3D space charge, particlematter interaction and multi-objective optimisation.

OPAL is built from the ground up as a parallel applicationexemplifying the fact that HPC (High Performance Computing) isthe third leg of science, complementing theory and the experiment

OPAL runs on your laptop as well as on the largest HPC clusters

OPAL uses the mad language with extensions

OPAL (and all other used frameworks) are written in C++ usingOO-techniques, hence OPAL is easy to extend.

Documentation is taken very seriously at both levels: source codeand user manual

Regression tests running evert day on the head of the repository


OPAL in a Nutshell II

At the moment we have an international team of 14 developers

A. Gsell (PSI), T. Kaman (PSI/ETH), Ch. Kraus (PSI), Y.Ineichen(IBM), S. Russell X. Pang (LANL), Y. Bi, Ch. Wang, J. Yang(CIAE), H. Zha (Thinghua University) C. Mayes (Cornell), D.Winklehner (MIT/PSI) Ch. Rogers, S. Sheehy (Rutherford) & AA(PSI)

webpage: https://amas.psi.ch/OPAL

manual:http://amas.web.psi.ch/docs/opal/opal_user_guide.pdf

problems, suggestion etc. mailing: opal AT lists.psi.ch

the OPAL Discussion Forum:https://lists.web.psi.ch/mailman/listinfo/opal


https://amas.psi.ch/OPAL

http://amas.web.psi.ch/docs/opal/opal_user_guide.pdf

https://lists.web.psi.ch/mailman/listinfo/opal

OPAL and its Flavours I

4 OPAL flavours exist:

OPAL-t

OPAL-t tracks particles which 3D space charge using time as theindependent variable, and can be used to model beamlines, guns,injectors and complete FEL’s but without the undulator.Field emission (dark current studies)many more linac features like auto-phasing, wake fields (short range)

OPAL-envelope

OPAL-envelope is based on the 3D-envelope equation (a laHOMDYN) and can be used to design FEL’s.OPAL-envelope could also be used for an on-line model (incl.space charge)same lattice than OPAL-t


OPAL and its Flavours II

OPAL-map (not yet released)

OPAL-map tracks particles with 3D space charge using splitoperator techniques.M(s) =Mext(s/2)⊗Msc(s)⊗Mext(s/2) +O(s3)

OPAL-cycl

3D space chargeneighbouring turnstime is the independent variable.from p to Uranium (q/m is a parameter)Solve Poisson equation with spectral methodsUse 4th-order RK, Leap Frog or adaptive schemes [Toggweiler]Single particle tracking mode & tune calculationParticle Matter InteractionMultipacting capabilities


Sketch of an inputfile

ffag: Cyclotron, TYPE="FFAG", CYHARMON=1, PHIINIT=phi0,

PRINIT=pr0, RINIT=r0 , SYMMETRY=1.0,

RFFREQ=200.0, BSCALE=10.0, FMAPFN="NSFFAGfieldPSI.dat";

rf0: RFCavity, VOLT=cavvol1, FMAPFN="Cav1.dat",

TYPE="SINGLEGAP", FREQ=200.0,

RMIN = 2100.0, RMAX = 4000.0,

ANGLE=45.0, GAPWIDTH = 300.0, PHI0=ph;

Di1:DISTRIBUTION, DISTRIBUTION = GAUSS, INPUTMOUNITS = EV,

SIGMAX = 0.00001, SIGMAPX = 0.00001,

SIGMAY = 0.00001, SIGMAPY = 0.00001,

SIGMAT = 1e-8 , OFFSETPZ = Edes*1e9,

CORRX = -0.0, CORRY = 0.5;

l1: Line = (ffag,rf0);


Sketch of an inputfile

Fs1:FIELDSOLVER, FSTYPE=FFT, MX=64, MY=64, MT=64,

PARFFTX=TRUE, PARFFTY=TRUE, PARFFTT=FALSE,

BCFFTX=OPEN, BCFFTY=OPEN, BCFFTT=OPEN;

beam1: BEAM, PARTICLE=PROTON, pc=P0,

NPART=1E6, BCURRENT=1.0E-9, BFREQ=200.0;

TRACK, LINE=l1, BEAM=beam1, MAXSTEPS=turns, STEPSPERTURN=180;

RUN, METHOD = "CYCLOTRON-T",

BEAM=beam1,

FIELDSOLVER=Fs1,

DISTRIBUTION=Di1;

ENDTRACK;


OPAL Object Oriented Parallel Accelerator Library

ElectroMagneto

Optics

N-BodyDynamics

Field Maps &

Analytic Models

∇ ·Esc = −ρ/ε0 = ∇ · ∇φsc∆φsc = − ρ

ε0& BC’s

H = Hext + Hsc

Energy loss −dE/dx (Bethe-Bloch)

Coulomb scattering is treated as two independent events:

multiple Coulomb scatteringlarge angle Rutherford scattering

Field Emission Model (Fowler-Nordheim)

Secondary Emission Model ([Furman & Pivi] & [Vaughan])


Outline

1 History




Particle Matter Interaction

Energy loss −dE/dx (Bethe-Bloch)Coulomb scattering is treated as two independent events:

multiple Coulomb scatteringlarge angle Rutherford scattering

Field Emission Model (Fowler-Nordheim)Secondary Emission Model ([Furman & Pivi] & [Vaughan])

Surface normal n

Incident electron Backscattered electron

Rediffused electronTrue secondaries

θ

Phenomenological- don’tinvolve secondary physics butfit the data.

Model 1 developed by M.Furmann and M. Pivi

Model 2 (Vaughan) is easierto adapt to SEY curves


Particle Matter Interaction

Energy loss −dE/dx (Bethe-Bloch)Coulomb scattering is treated as two independent events: themultiple Coulomb scattering and the large angle RutherfordscatteringSpace charge & halo generated by the obstacle

A 72 MeV cold Gaussian beam with σx = σy = 5 mm passing a copperslit with the half aperture of 3 mm from 0.01 m to 0.1 m.


Particle Matter Interaction cont.

0 20 40 60 8010-4

10-2

100

102

104

E (MeV)

1/G

eV/p

artic

le

0 5 1010-5

100

105

angle (deg)

1/so

lid a

ngle

/GeV

/par

ticle

FLUKAOPAL

OPALMCNPXFLUKA


Outline

1 History




OPAL Architecture

OPALExt. MAD Lang. Parser Flavors: t,Cycl,Envelope Optimization

Solvers: Direct, Iterative Integrators Distributions

IP 2L

FFT D-Operators NGP, CIC, TSI

Fields Mesh Particles

Load Balancing Domain Decomp. Message Passing

Particle Cash PETE Trilinos Interface

classic

H5hut

Trilinos & GSL

OPAL Object Oriented Parallel Accelerator LibraryIP2L Independent Parallel Particle LayerClass Library for Accelerator Simulation System and ControlH5hut for parallel particle and field I/O (HDF5)Trilinos http://trilinos.sandia.gov/


Outline

1 History




A fast Direct FFT-Based PIC Poisson SolverAssume you know G the Green’s function

. Assign (scatter) all particles charges qi to nearby mesh points to obtain ρ

. Lorentz transform to obtain ρ in beam rest frame Sbeam.

. Use FFT on ρ and G to obtain ρ and G

. Determine φ on the grid using φ = ρ · G

. Use inverse FFT on φ to obtain φ

. Compute E = −∇φ

. Lorentz back transform to obtain Esc and Bsc in laboratory frame Slab

. Interpolate (gather) E, B at particle positions x from Esc and Bsc

Charge assignment and electric field interpolation are related to theinterpolation scheme. If ei is the charge of a particle, we can write thedensity at mesh point km as

ρ(km)D =

N∑i=1

ei ·W (qi,km), m = 1 . . .M (1)

where W is a suitably chosen weighting function.OPAL Introduction Lecture 1 PSI OPAL lectures FFAG Workshop 2014 / 46

Parallel PerformanceOPAL Scaling on Cray XT3 ”Horizon” at CSCS

Production Run Setup

Tracking 107 particles

3D FFT on a 2563 grid

Gaussian distribution

no parallel I/O

ObservationsSolver scales still well

Load balancing not optimal anymore

Moving mesh has a problem

10

100

1000

64 128 256 512 1024

Exe

cutio

n tim

e (r

eal t

ime)

[s]

Number of cores

tot cpuinteg

solver bb-adapt

repart 0

10 20 30 40 50 60 70 80 90

100

64 128 256 512 1024

Para

llel E

ffice

ncy

[%]

Number of cores

tot cpuinteg

solver bb-adapt

repart


Outline

1 History




Iterative Poisson Solver SAAMG-PCG

Boundary Problem

∆φ = − ρ

ε0, in Ω ⊂ IR3,

φ = 0, on Γ1

∂φ

∂n+

1

dφ = 0, on Γ2

Ω ⊂ IR3: simply connectedcomputational domain

ε0: the dielectric constant

Γ = Γ1 ∪ Γ2: boundary of Ω

d: distance of bunchcentroid to the boundary

x3 = z

x1

x2

Γ2

Γ1

Γ2

Γ1 is the surface of an

1 elliptic beam-pipe

2 arbitrary beam-pipe element


Iterative Poisson Solver SAAMG-PCG cont.

We apply a second order finite difference scheme which leads to a set oflinear equations

Ax = b,

where b denotes the charge densities on the mesh.

Ω ∈ R3∂Ω : φ = 0hNhW

hEhS

solve anisotropic electrostatic PoissonPDE with an iterative solver

reuse information available from previoustime steps

achieving good parallel efficiency

irregular domain with “exact” boundaryconditions

easy to specify boundary surface


SAAMG-PCG Parallel Efficiency

number of cores

effi

cien

cy [

%]

75

80

85

90

95

100

512 1024 2048

solution timeconstruction timeapplication timetotal ML time

obtained for a tubeembedded in a1024× 1024× 1024 grid

construction phase isperforming the worst withan efficiency of 76%

influence of problem sizeon the low performanceof the aggregation in ML

[A. Adelmann, P. Arbenz, et al., JCP, 229 12 (2010)]


Outline

1 History




3D Geometry Handling Capability of OPAL

obtain geometry in STEP format

mesh geometry with GMSH and export to native VTK

convert to h5 with vtk2h5grid


Triangle-line segment intersection

Boundary bounding box to speedup the collision tests

We can handle arbitrary structure as long as it is closed

w

ti

si

t0t1

t2n

v

u

I

x0

x1

ri

n

n

[C. Wang, A. Adelmann, et al.]OPAL Introduction Lecture 1 PSI OPAL lectures FFAG Workshop 2014 / 46

Outline

1 History




ArchitectureConnection with other Frameworks

STEP file gmsh/netgen vtk

vtk2h5grid

OPAL

gnuplot

H5root& Visit

Disk

.stat

.in, .stat. .h5n-way parallel

.h5

.h5

H5root


Outline

1 History




PSI 590 MeV Ring - last 8 turns @ 2.2 mA

initial conditions from 72 MeV transfer line simulation (OPAL-t )rf parameters from control roomusing measured mid-plane field and analytic trim-coil (tc15)single particle run to verify tun numbers and tunes

1500

2000

2500

3000

3500

4000

2000 2500 3000 3500 4000 4500

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8

FIXPOOPAL-cycl

nur=2nuz


PSI 590 MeV Ring - last 8 turns @ 2.2 mA

[Y. Bi, A. Adelmann, et al., PR-STAB 14(5) (2011)]


Neighboring Bunch Effects- Multi Bunch Model

In the model, the injection-to-extraction simulation is divided into twostages:

1 First stage, big ∆r ⇒ single bunch tracking2 Second stage, small ∆r ⇒ multiple bunches tracking

Full 3DEnergy bins & re-binningLarge grids needed


Neighboring Bunch Effects- Multi Bunch ModelSingle bunch and multiple bunches at turn 80 and 130

PSI 590MeV Ring

longitudinal (mm) -6 -4 -2 0 2 4 6

tran

sver

se (

mm

)

-6

-4

-2

0

2

4

6

0

10

20

30

40

50

1 bunch


tran

sver

se (

mm

)

-6

-4

-2

0

2

4

6

0

20

40

60

80

100

7 bunches


tran

sver

se (

mm

)

-6

-4

-2

0

2

4

6

0

10

20

30

40

50

60

70

80

90

9 bunches

longitudinal (mm)-6 -4 -2 0 2 4 6

tran

sver

se (m

m)

-6

-4

-2

0

2

4

6

0

10

20

30

40

50

60

70

80

90

1 bunch


tran

sver

se (m

m)

-6

-4

-2

0

2

4

6

0

20

40

60

80

100

120

140

160

7 bunches


tran

sver

se (m

m)

-6

-4

-2

0

2

4

6

0

20

40

60

80

100

120

140

160

9 bunches

[J. Yang, A. Adelmann, et al., PR-STAB 13(6) (2010)]OPAL Introduction Lecture 1 PSI OPAL lectures FFAG Workshop 2014 / 46

Outline

1 History

2 Models in OPAL for Precise Accelerator Modelling



New OPAL Element Ringmostly contributed by C. Rogers

OPAL requires modification to adequately track FFAG field maps

OPAL-t allows tracking through a set of beam elements in linac-typegeometry

OPAL-cycl previously hard coded to use 2D mid plane field map +single RF cavity

Aim to introduce the capability to track through a set of arbitrarybeam elements in ring-type geometry

Additionally introduce specific capability to track through a 3D fieldmap in a sector-type geometry

Use ERIT ring as test-bed for this development


Implementation in OPALSoftware Engineering @ Work


Outline

1 History




Getting closed orbit through OPALAll ERIT simulations are curtesy of C. Rogers


Getting closed orbit through OPALAll ERIT simulations are curtesy of C. Rogers

OPAL Introduction Lecture 1 PSI OPAL lectures FFAG Workshop 2014 Page 41 / 46

Dynamic Apperture∆t = 10−10 s

After 100 turns aperture looks okay in OPALIs this dynamic aperture or field map aperture?Field map extent is ± 250 mm in xParticles outside aperture are lost after < 1 turn


Outline

1 History




Lattice design & tracking with OPALS. L. Sheehy, C. Johnstone, A 1 GeV CW FFAG High Intensity Proton Driver, IPAC2012

300 MeV to 1 GeV ring

4-cell quasi-isochronous non-scaling FFAG design

short (1cm long) 1 π mm mrad proton bunch

acceleration is achieved with 3 cavities with an energy gain of 22MV per turn, which is sufficient to open the serpentine channel

the beam exhibits expected phase space distortion and filamentationeven in the 0 mA


Lattice design & tracking with OPAL cont.S. L. Sheehy, C. Johnstone, A 1 GeV CW FFAG High Intensity Proton Driver, IPAC2012


References

A. Adelmann, P. Arbenz, et al., J. Comp. Phys, 229 (12): 4554 (2010)

M. A. Furman and M. Pivi, Phys. Rev. STAB 5, 124404 (2002)

Y. Bi, A. Adelmann et al., Phys. Rev. STAB 14(5) 054402 (2011)

J. Yang, Adelmann et al., Phys. Rev. STAB 13(6) 064201 (2010)

J. Yang, Adelmann et al., NIM-A 704(11) 84-91 (2013)

J. R. M. Vaughan, IEEE Transactions on Electron Devices 40, 830 (1993)

M. Toggweiler, A. Adelmann, P. Arbenz, J.J. Yang, arXiv:1211.3664 (2012)

C. Wang, A. Adelmann, et al., arXiv:1208.6577


opal introduction lecture 1 - psiamas.web.psi.ch/people/aadelmann/pub/ffag14/opal-lect... ·...

Documents