Parallel Finite-Difference Time-Parallel Finite-Difference Time-Domain Computations Aided by Domain Computations Aided by

Modal DecompositionModal Decomposition

Dmitry A. GorodetskyDmitry A. Gorodetsky

Philip A. WilseyPhilip A. Wilsey

OutlineOutline

IntroductionIntroduction• FDTDFDTD• Distributed ComputationDistributed Computation

Model Order ReductionModel Order Reduction ConclusionConclusion ReferencesReferences

IntroductionIntroduction FDTD: evolutionary algorithm solves Maxwell’s FDTD: evolutionary algorithm solves Maxwell’s

eqs. by marching. eqs. by marching. Some typical problems:Some typical problems:

• Aircraft Radar Cross SectionAircraft Radar Cross Section• Microwave ICs, High Speed Electronic CircuitsMicrowave ICs, High Speed Electronic Circuits• Optical Pulse PropagationOptical Pulse Propagation• AntennasAntennas• Bioelectromagnetic Systems (Retina, EM hypothermia Bioelectromagnetic Systems (Retina, EM hypothermia

cancer therapy)cancer therapy)• Bodies of RevolutionBodies of Revolution

Computed surface electric currents induced on a prototype military jet fighter plane by a radar beam at 100 MHz. The incident plane wave propagates from left to right head-on to the airplane. The surface currents re-emit electromagnetic energy which can be used to create RCS plots [1].

Simulation ComplexitySimulation Complexity

Example of 2Example of 2ndnd order FDTD evolution: order FDTD evolution:HHzz(n+1/2)=k(n+1/2)=k11[[ΔΔEExx(n)]+k(n)]+k22[[ΔΔEEyy(n)]+k(n)]+k33HHzz(n-1/2)(n-1/2)

Grid size as well as number of time steps Grid size as well as number of time steps can make the simulation prohibitive.can make the simulation prohibitive.

Computational burden grows as ~Computational burden grows as ~NN4/3 4/3 [1][1]

A single FDTD cell

Reducing Simulation TimeReducing Simulation Time

Methods to Improve Simulation Time:Methods to Improve Simulation Time:• Distributed Computation [1-3]Distributed Computation [1-3]

Domain Decomposition Domain Decomposition SynchronizationSynchronization Load BalancingLoad Balancing

• Model Order ReductionModel Order Reduction State Transition Matrix – Modal Approach [4-7] - ExactState Transition Matrix – Modal Approach [4-7] - Exact

• Entire DomainEntire Domain• Sub DomainSub Domain

Linear Estimation Methods [1, 8] - ApproximateLinear Estimation Methods [1, 8] - Approximate• Prony’s Method (complex exponentials)Prony’s Method (complex exponentials)• System Identification TechniqueSystem Identification Technique

Distributed ComputationDistributed Computation FDTD requires knowledge of FDTD requires knowledge of

state of adjacent points to state of adjacent points to compute the current point.compute the current point.

Hence it exhibits fine-grain Hence it exhibits fine-grain parallelism and its speedup is parallelism and its speedup is limited by surface/volume ratio.limited by surface/volume ratio.

Surface to volume ratio of FDTD Surface to volume ratio of FDTD partitions is in effect partitions is in effect communication/computation communication/computation ratio.ratio.

0

0.2

0.4

0.6

0.8

1

1.2

0 5 10 15 20 25

Processors

Sp

eed

up

Eff

icie

ncy

Blocking Send& ReceiveNon BlockingReceiveIdeal

ProposedMethod

Figure 1. Speedup Efficiency of parallel FDTD [9]

OutlineOutline

IntroductionIntroduction Model Order ReductionModel Order Reduction

• State Transition Matrix (exact)State Transition Matrix (exact) Entire DomainEntire Domain

• Expensive SetupExpensive Setup• Cheap IterationCheap Iteration• Setup ParallelizationSetup Parallelization

Sub Domain (Macromodel) Sub Domain (Macromodel) ConclusionConclusion ReferencesReferences

After Chen [10], we can express the After Chen [10], we can express the FDTD update equations as:FDTD update equations as:

EE(n) (n) == D D11HH(n-(n-1/21/2) + ) + GG11EE(n-(n-11))

HH(n+(n+1/21/2) ) = = DD22EE(n) + (n) + GG22HH(n-(n-1/21/2)) With these equations, the state With these equations, the state

transition matrix becomes:transition matrix becomes:

21212

11

GDDGD

DGA

Entire DomainEntire Domain

Entire Domain (2)Entire Domain (2) Then can express FDTD as:Then can express FDTD as:

QQ(n)(n)==AQAQ(n-1),(n-1), (1)(1)where Q represents the present state.where Q represents the present state.

Every step takes Every step takes NN22 multiplications. multiplications. If we assume that the system starts out from If we assume that the system starts out from

Q(0)=aQ(0)=a11vv11+a+a22vv22+…+a+…+aNNvvNN, then (1) can be written then (1) can be written as:as:

(2)(2)

where where vvi i are the eigenvectors and are the eigenvectors and λλii are are eigenvalues of eigenvalues of AA..

,)(1

N

ii

niian vQ

Entire Domain (3)Entire Domain (3)Cheap IterationCheap Iteration

The advantage of the modal method The advantage of the modal method for FDTD is that time-stepping is de-for FDTD is that time-stepping is de-coupled (see eq.2)coupled (see eq.2)

Solution can be obtained at any time Solution can be obtained at any time step without knowledge of previous step without knowledge of previous time step.time step.

Time-stepping can be parallelized Time-stepping can be parallelized and does not require communication.and does not require communication.

Entire Domain (4)Entire Domain (4)Expensive SetupExpensive Setup

The matrix The matrix AA is sparse, diagonally is sparse, diagonally dominant, and banded.dominant, and banded.

With standard techniques (LAPACK), With standard techniques (LAPACK), getting the eigendecomposition of getting the eigendecomposition of AA is an O(is an O(NN33) process.) process.

LAPACK uses QR iteration to obtain LAPACK uses QR iteration to obtain the the SchurSchur form and hence is not easy form and hence is not easy to parallelize.to parallelize.

Entire Domain (5)Entire Domain (5)Setup ParallelizationSetup Parallelization

We can take advantage of the modal We can take advantage of the modal make-up of the make-up of the AA matrix because in matrix because in practice we do not need all the practice we do not need all the modes [10,11].modes [10,11].

One alternative method is spectral One alternative method is spectral divide and conquer (SDC) [12].divide and conquer (SDC) [12].

SDC: SDC: signsign ( (AA--bbII), where ), where bb represents represents the the xx-coordinate of a vertical line in -coordinate of a vertical line in the complex plane.the complex plane.

Entire Domain (6)Entire Domain (6)Setup ParallelizationSetup Parallelization

SDCSDC Advantages:Advantages:

• Compute only needed eigenvalues.Compute only needed eigenvalues.• Easy to parallelize.Easy to parallelize.• Computation time is Computation time is kNkN33 but but kk depends on the depends on the

number of eigenvalues.number of eigenvalues. Disadvantages:Disadvantages:

• Requires several iterations before sign function Requires several iterations before sign function converges.converges.

• Requires knowledge of where eigenvalues do Requires knowledge of where eigenvalues do not lie otherwise sign function may not not lie otherwise sign function may not converge quickly.converge quickly.

Entire Domain (7)Entire Domain (7)Setup ParallelizationSetup Parallelization

Alternatives: Iterative Techniques Alternatives: Iterative Techniques Simultaneous IterationSimultaneous Iteration

Arnoldi and Lancsoz Arnoldi and Lancsoz [12,13][12,13] Advantages:Advantages:

• Exploit sparsity.Exploit sparsity.• Can be parallelized.Can be parallelized.

Disadvantages:Disadvantages:• Require computation of Require computation of allall eigenvalues. eigenvalues.

ConclusionConclusion

The setup time of this method is expensive The setup time of this method is expensive for a reason.for a reason.

Very good accuracy results even after Very good accuracy results even after eigenmodes are discarded.eigenmodes are discarded.

Setup and time-stepping can be Setup and time-stepping can be parallelized and need not be limited by parallelized and need not be limited by communication as conventional FDTD.communication as conventional FDTD.

Imprvmnt = function (#steps x #CPUs)Imprvmnt = function (#steps x #CPUs)

ReferencesReferences1.1. A. Taflove, A. Taflove, Computational Electrodynamics: the finite-difference time-domain Computational Electrodynamics: the finite-difference time-domain

method, method, Norwood, MA: Artech House, 1995.Norwood, MA: Artech House, 1995.2.2. N. P. Chrisochoides, E. Houstis, and J. Rice, “Mapping algorithms and software N. P. Chrisochoides, E. Houstis, and J. Rice, “Mapping algorithms and software

environment for data parallel PDE iterative solvers,” environment for data parallel PDE iterative solvers,” Special issue of the Journal of Special issue of the Journal of Parallel and Distributed Computing on Data-Parallel Algorithms and ProgrammingParallel and Distributed Computing on Data-Parallel Algorithms and Programming, , Vol 21, No 1, pp 75--95, April, 1997.Vol 21, No 1, pp 75--95, April, 1997.

3.3. N. P. Chrisochoides and J. R. Rice, “Partitioning heuristics for PDE computations N. P. Chrisochoides and J. R. Rice, “Partitioning heuristics for PDE computations based on parallel hardware and geometry characteristics.” based on parallel hardware and geometry characteristics.” In Advances in In Advances in Computer Methods for Partial Differential Equations VII,Computer Methods for Partial Differential Equations VII, (R. Vichnevetsky. D. (R. Vichnevetsky. D. Knight and G. Richter, eds) IMACS, New Brunswick, NJ, pp. 127-133, 1992.Knight and G. Richter, eds) IMACS, New Brunswick, NJ, pp. 127-133, 1992.

4.4. Z. Chen, “Analytic Johns matrix and its application in TLM diakoptics,” Z. Chen, “Analytic Johns matrix and its application in TLM diakoptics,” IEEEIEEE MTT-S MTT-S Digest, Digest, vol. 2, pp. 777-780, 1995.vol. 2, pp. 777-780, 1995.

5.5. W. J. Hoefer, “The discrete time domain green’s function or john’s matrix – a new W. J. Hoefer, “The discrete time domain green’s function or john’s matrix – a new powerful concept in transmission line modeling (TLM),” powerful concept in transmission line modeling (TLM),” Int. J. Num. ModelingInt. J. Num. Modeling, vol. , vol. 2, pp. 215-225, 1989.2, pp. 215-225, 1989.

6.6. P. B. Johns and K. Akhtarzad, “Time domain approximations in the solution of P. B. Johns and K. Akhtarzad, “Time domain approximations in the solution of fields by time domain diakoptics,” fields by time domain diakoptics,” Int. J. Num. Methods Eng., Int. J. Num. Methods Eng., vol. 18, pp. 1361-vol. 18, pp. 1361-1373, 1982.1373, 1982.

7.7. P. B. Johns and K. Akhtarzad, “The use of time domain diakoptics in time discrete P. B. Johns and K. Akhtarzad, “The use of time domain diakoptics in time discrete models of fields”, models of fields”, Int. J. Num. Methods Eng.,Int. J. Num. Methods Eng.,vol. 17, pp. 1-14, 1981.vol. 17, pp. 1-14, 1981.

References (2)References (2)8.8. W. Kumpel and I. Wolff, “Digital signal processing of time domain field W. Kumpel and I. Wolff, “Digital signal processing of time domain field

simulation results using the system identification method,” simulation results using the system identification method,” IEEE Trans. IEEE Trans. Microwave Theory Techniq., Microwave Theory Techniq., vol. 42, no. 4, pp. 667-671, 1994.vol. 42, no. 4, pp. 667-671, 1994.

9.9. D. A. Gorodetsky and P. A. Wilsey, “Innovative approaches to D. A. Gorodetsky and P. A. Wilsey, “Innovative approaches to parallelizing finite-difference time-domain computations,” parallelizing finite-difference time-domain computations,” IEEE IEEE Workshop on Direct and Inverse Problems in ElectrodynamicsWorkshop on Direct and Inverse Problems in Electrodynamics, 2005., 2005.

10.10. Z. Chen and P. P. Silvester, “Analytic solutions for the finite-difference Z. Chen and P. P. Silvester, “Analytic solutions for the finite-difference time-domain and transmission-line-matrix methods,” time-domain and transmission-line-matrix methods,” Microwave and Microwave and Optical Technology LettersOptical Technology Letters, vol. 7, no.1, pp. 5-8, 1994., vol. 7, no.1, pp. 5-8, 1994.

11.11. D. A. Gorodetsky and P. A. Wilsey, “Reduction of FDTD simulation time D. A. Gorodetsky and P. A. Wilsey, “Reduction of FDTD simulation time with modal methods,” with modal methods,” Progress in Electromagnetics Research Progress in Electromagnetics Research SymposiumSymposium, 2006, in press., 2006, in press.

12.12. J. W. Demmel, M. T. Heath, and H. A. van der Vortst, J. W. Demmel, M. T. Heath, and H. A. van der Vortst, Parallel numerical Parallel numerical linear algebralinear algebra, in Acta Numerica 1993, Cambridge, 1993, Cambridge , in Acta Numerica 1993, Cambridge, 1993, Cambridge University Press, pp. 111–197University Press, pp. 111–197

13.13. Z. Bai, “Progress in the numerical solution of the nonsymmetric Z. Bai, “Progress in the numerical solution of the nonsymmetric eigenvalue problem,” eigenvalue problem,” Journal of Numerical Linear Algebra with Journal of Numerical Linear Algebra with ApplicationsApplications, vol. 2, pp. 219--234, 1995., vol. 2, pp. 219--234, 1995.