HackLatt 2008 1

MILC Code Basics

Carleton DeTarFirst presented at Edinburgh EPCC HackLatt 2008

Updated 2013

HackLatt 2008 2

MILC Code Capabilities• Molecular dynamics evolution

– Staggered fermion actions (Asqtad, Fat7, HISQ, etc)– Clover fermion action– Pure gauge– Schroedinger functional

• Hadron spectroscopy– Staggered mesons and baryons– Clover mesons and baryons– Mixed staggered/clover mesons– Static/light spectroscopy– Quarkonium spectroscopy (S and P-wave)

• Current matrix elements– Leptonic decay (fpi, fB, fD)– Semileptonic decay (heavy-light)

• Miscellaneous– Topological charge– Dirac matrix eigenvectors and eigenvalues– Nonperturbative renormalization of currents

HackLatt 2008 3

Two views of the code. Powerful and capable.

HackLatt 2008 4

Mathias Gruenewald: Temptation of St Anthony (1515)

Or a path to confusion and misery.

HackLatt 2008 5

Supported File Formats

• Gauge configuration file formats– MILC, SciDAC (ILDG), NERSC, Fermilab

• Dirac propagator file formats– USQCD, Fermilab

• Staggered propagator file formats– USQCD, MILC, Fermilab

HackLatt 2008 6

Supported SciDAC C-Coded Packages

• QIO (I/O)• QMP (Message passing)• QLA (linear algebra – single processor)• QDP/C (linear algebra – data parallel)• QOPQDP (“Level 3” optimized)• (More in the next session)

http://usqcd.jlab.org/usqcd-software/

HackLatt 2008 7

Precision

• Global single or double precision• Mixed precision in some applications

Portability

• Any scalar machine• Any MPP machine with MPI• GPU, Intel Xeon Phi (somewhat)

HackLatt 2008 8

MILC Code Organization

• Application directories– With compilation targets

• Library directory– Linear algebra routines

• Shared procedures (“generic”) directories– Shared across applications

HackLatt 2008 9

MILC Code Organization

• Application directories: examples– cd ks_imp_dyn (application)

• make su3_rmd (target) (Asqtad R algorithm)• make su3_spectrum (another target) (staggered

spectroscopy)– cd ks_imp_rhmc (application)

• make su3_rhmc (Asqtad RHMC algorithm)• make su3_rhmc_hisq (target) (HISQ algorithm)

– cd clover_invert2 (application)• make su3_clov (clover spectroscopy, etc.)

HackLatt 2008 10

MILC Code Organization

• Shared procedures directories: examples– generic (common to all applications)– generic_ks (common to staggered fermion applications)– generic_wilson (common to clover and Wilson fermion apps)

HackLatt 2008 11

Building the MILC Code

• Download source http://www.physics.utah.edu/~detar/milc_qcd.html

• Unpack• Configure• Build• Check

HackLatt 2008 12

Building the MILC Code

• Unpack– tar –xvzf milc_qcd-7.7.10-a7.tar.gz

HackLatt 2008 13

Building the MILC Code

• Configure (crude old fashioned!)– Copy default Makefile to application directory

•cd ks_imp_rhmc•cp ../Makefile .

– Edit (example in next slide)•Makefile•../libraries/Make_vanilla•../include/config.h

HackLatt 2008 14

#----------------------------------------------------------------------# 2. Architecture

# Compiling for a parallel machine? Uncomment this line.#MPP = true

#----------------------------------------------------------------------# 3. Precision

# 1 = single precision; 2 = doublePRECISION = 1

#----------------------------------------------------------------------# 4. Compiler# Choices include mpicc cc gcc pgcc g++

ifeq ($(strip ${MPP}),true) CC = /usr/local/mvapich/bin/mpiccelse CC = gccendif

Editing the Makefile -- examples

HackLatt 2008 15

Optimization Possibilities

• Build plain MILC version• Build with SciDAC optimized QOP support

– Requires installing the SciDAC packages. (See second tutorial and code home page).

• Build with GPU QUDA support– Requires installing QUDA and QUDA-MILC

packages. (See the code home page)• Build with OpenMP directives on loops

– Very rudimentary and spotty. May need to edit some code by hand to insert OMP macros.

HackLatt 2008 16

#----------------------------------------------------------------------# 5. Compiler optimization level OPT = -O3#----------------------------------------------------------------------# 6. Other compiler optimizations (depending on compiler) OCFLAGS = # Compiling with OpenMP? OMP = true#----------------------------------------------------------------------# 10. SciDAC package options WANTQOP = true # turns on all optimized QOPQDP modules#----------------------------------------------------------------------# 14. GPU/QUDA Options WANTQUDA = true # turns on possible QUDA selections WANT_FN_CG_GPU = true # turns QUDA CG WANT_FL_GPU = true # QUDA link fattening WANT_FF_GPU = true # QUDA fermion force term WANT_GF_GPU = true # QUDA gauge force

ifeq ($(strip ${MPP}),true) CC = /usr/local/mvapich/bin/mpiccelse CC = gccendif

Otimization choices controlled in the Makefile -- examples

HackLatt 2008 17

Building and Checking the Code

• Build (for example)– make su3_rmd

• Check single precision su3_rmd– make check “PROJS=su3_rmd” “PRECLIST=1”

• Check all targets in this directory– make check

HackLatt 2008 18

Running the code

• su3_rhmc < inputfile > outputfile• su3_rhmc inputfile > outputfile• su3_rhmc inputfile outputfile

HackLatt 2008 19

Sample parameter input (su3_rmd)prompt 0nflavors1 2nflavors2 1nx 16ny 16nz 16nt 64iseed 5682304

warms 0trajecs 2traj_between_meas 1beta 6.76mass1 0.05mass2 0.5u0 0.8441microcanonical_time_step 0.02steps_per_trajectory 4max_cg_iterations 300max_cg_restarts 5error_per_site .000005error_for_propagator .000002npbp_reps 1prec_pbp 1reload_serial ../../binary_samples/lat.sample.l4444save_serial_scidac lat.test.scidac

molecular dynamics

measurements

lattice dimensions

HackLatt 2008 20

Summary

• The MILC code is versatile and portable• I have given a brief overview of the code

structure• I have touched on the procedures for

building and running the code

HackLatt 2008 21

Tutorial 1 Goals

• Run precompiled code• Modify the input parameters• Build a different target• Modify the Makefile and build• Download and unpack the code (if time)