scidac software jlab ahm may 6, 2011 - usqcd
TRANSCRIPT
SciDAC Software JLab AHM May 6, 2011
New Machines: BG/Q & CPS (Mawhinney)Blue Waters (Gottlieb)GPU & Chroma (Joo)Strong Scaling GPU (QUDA group)
New Software/Algorithms QDPOP Multi-grid (Osborn)QULA (Pochinsky)QUDA slides (Babich at Software Workshop)
Possible Topics for Discussion
Friday, May 6, 2011
Mawhinney
Friday, May 6, 2011
Mobius/DWF Preconditioning On 323 × 64× 32 Lattice
March 16, 2011
Testing was done on BG/P, 512 node machine. Solves to 1e-10.
1 Mobius Ls = 12
DWF Ls = 32, Mobius Ls = 12. Mobius parameters are as follows:
b5 = 1.841556, c5 = 0.841556 (1)
The procedure for Mobius guided solve is as follows
1. solves on DWF to 1e-2
2. solves on Mobius to 2e-5 (repeated 3 times, all on residue)
3. solves on DWF to 1e-10
The red solves in the following table are done on Ls = 12 Mobius lattice. All the rest aredone on the original Ls = 32 DWF lattice.
CG From zero init guess CG via Mobius
16121 4447 106101 4581 106102 4775 106
51711290
equivalent:63512.68e3 seconds 1.14e3 seconds
1
Gottlieb
Friday, May 6, 2011
Blue Waters
• USQCD has a PRAC grant from NSF to support travel to NCSA and interaction with NCSA staff in preparation for Blue Waters, NSF’s sustained petascale computer
• Public schedule: early science in late 2011, full service mid-2012
• Greg Bauer is our primary point of contact and has been a source of great help.
• He has access to prototype hardware and has been running both Chroma and MILC codes.
• Some of us have access to Blue Drop (Power 7 system at NCSA).
• Performance information is still NDA, but we can list some of the activities.
• Code has compiled and been run.
• Variation on performance depending on local volume
• data padding needed?
• NUMA issues are a concern
• VSX/VMX routines (see MILC) should be easy to integrate.
Chroma
CPS
• Peter Boyle has NDA access through Edinburgh.
• He plans to port BAGEL to Power 7.
MILC• SG spent 2009-10 sabbatical at NCSA.
• VSX/VMX routines written by Brad Elkins (IBM) tested by Greg Bauer.
• SMT tested. SMT=2 provides most of the gain. (SMT=4 is maximum.)
• Derived datatypes reduce copying of data to and from MPI buffers.
• EuroMPI paper by Hoefler and Gottlieb
• Now tested on BW prototype
• MILC code (su3_rmd) has been run on up to 256 cores of prototype hardware by Greg Bauer.
• various options tested
• Performance model by Gottlieb and Hoefler
• model is public; BW parameters are not
• Independent model by Hoisie et al.
• It’s late and have not seen a write-up.
• With so many cores/node, a hybrid OpenMP and MPI program may get better performance.
• This is expected for BlueGene/Q
• Doug Toussaint has been trying this approach on Hopper (NERSC Cray XE6)
• Greg Bauer is trying this on BW prototype.
Joo
Friday, May 6, 2011
Status: Chroma+QUDA • Chroma wraps the QUDA Clover solvers
– QUDA Propagator, two flavor & shifted solvers • Chroma HMC trajectories possible with solver work on GPU • Multi-Dimensional parallelization of QUDA has now happened
– Wilson/Clover/AsqTAD (so far) -- (clover only in Chroma) – PCIe still choke point
• Additive Schwarz Domain Decomposed Solver helped Clover inverter (usefully) scale to 256 GPUs
– Some cleanup needed in some of the wrappers • Multi-Dim work changed QUDA interfaces...
– Lots of work out there to do (volunteers?)
Current & Future Efforts at JLab • ‘General Computing’ R&D
– Porting/Optimization for Emerging Systems • BlueWaters, BG/Q
– Direct QDP++ support for GPU/Heterogeneous systems – A lot of work in the invisible plumbing (beneath Chroma)
• e.g. Optimized Dslash-es, Clover Terms, solvers etc. – Architectural Exploration Work
• E.g. CUDA-4.0, PCIe networks, Intel Knights series, future NVIDA GPUs, Intel AVX etc.
– Algorithmic Work • Scalable Solvers (e.g. Domain Decomposed, Mixed
Precision, Multi-level etc)
Current & Future Efforts at JLab • Analysis Methods R & D:
– Improvements on Distillation • Better Smearing Techniques • Hybrid Distillation/Stochastic (Noisy Methods) • Very large Q2 form-factors • Changes to our workflow: Will need infrastructure
support: tape libraries, parallel I/O, etc – Software
• Három -- A 3D code for contractions • Redstar -- Compute 2pt-3pt correlation functions
using the output of Három
GPU Strong Scaling(Babich, Clark, Joo, Shi, Brower &Gottlieb)
Friday, May 6, 2011
4 8 16 32 64 128 256Number of GPUs
1
2
4
8
16
32Tf
lops
BiCGstabGCR-DD
V = 323 × 256
Friday, May 6, 2011
8 16 32 64 128 256Number of GPUs
4
8
16
32
64Ti
me
to so
lutio
n (s
ec)
BiCGstabGCR-DD
V = 323 × 256
Friday, May 6, 2011
4K 8K 12K 16K 20K 24K 28K 32KNumber of cores
0
5
10
15
20Tf
lops
Sus
tain
ed in
Sol
ver
Intrepid BG/P BiCGStab DPJaguar XT4, Rel. IBiCGStab, Mixed Prec.Jaguar XT5, Rel. IBiCGStab, Mixed Prec.
Friday, May 6, 2011
Osborn
Friday, May 6, 2011
James C. Osborn - USQCD Software 2011 1
James C. Osborn - USQCD Software 2011 2
James C. Osborn - USQCD Software 2011 3
Q* “C” libraries
!"#$%&'(&)%*+,-&
./0'1&%+23",$&#$$4-5&
6789:&6$*(&;<==&
James C. Osborn - USQCD Software 2011 5
Outline
! !"#$
! !%&$
! !'($
! !)#$
! !&#!)#$
! *!+",$
! !'-($
James C. Osborn - USQCD Software 2011 6
QMP
! !"#$%#&'$(%)*+&,-.-/0"1&2%*3(4$&)+&56"7&8)#9&:$'$6&7("+4;<&
= >"?&)%&@*(&"6A;"9&73#&%;*36:&7$&%#"76$&
= B*36:&%C66&7$&4;"+8$%&#*&+$D&@$"#3($%&
! E$D&@$"#3($%F&
= G"H*(&($*(8"+)I"C*+9&"66*D%&$"%)$(&A638&)+&*@&%A$4)"6)I$:&4*:$&
= J::$:&4*KK3+)4"#*(&%3AA*(#&
= L+:$M$:&K$K*(N&#NA$&
= 5*7&8$*K$#(N&2B"(6$#*+<&
= B;"+8$&"::($%%&2B;36D**<&
= :$46"($O%$+:O($4'OA")(%P&
James C. Osborn - USQCD Software 2011 7
QIO
! !"#$%#&'$(%)*+&,-.-/&0%)+1$&.232456&0)+&78"9&:)#&"+;&1'%<&%=)#1>&#*&:)#?6&
! !"@1$&'*8AB$&C*(B"#%D&E FGHI!JKG!J&0%$()"8&*(&L"("88$86&
E MN!OGKG!J&
E PQROKG!J&
E PQROKG!JSTGR&0'*8HHHH2UV8$+"B$W6&0F$(:$X&FX()#%X+<&L()'"#$&'$(%)*+6&
! P*%%)98$&)BL(*'$B$+#%D&E F)+:8$V8$&L"("88$8&=)#>&'"()"98$&+AB9$(&*C&G2Y&+*;$%&0"8%*&;*&G2Y&1*+Z:A*A%8X6&
E !"X*A#-[+*;$S+AB9$(<+*;$S)+;$\<:$#S1**(;%]&%>*A8;&#"^$&A%$(&%ALL8)$;&"(:D&
";;&'*);&_8"X*A#-"(:&=>)1>&1"88%&=)#>*A#&"(:&)C&)#&)%&HN!!?&
E MPG`GY?&0L(*98$B&)%&#>"#&8)B$&*L$+%&V8$>"+;8$<&MPG`GY&L(*9"98X&9$8*+:%&)+&aMP6&
James C. Osborn - USQCD Software 2011 8
QLA
! !"##$%&'($#)*+%',-.-/012'3*%'4()5'61%&'&+'7+($'&+'8*&9'
! :$6';$1&"#$)'
< =$>$#'1#?*#@':4')"AA+#&5'")*%8'!BB'(1#*1?C$'C$%8&D'1##1@)'
#$E"*#$)'6+#F*%8'!BB'4+7A*C$#'3)+7$'+CG$#'844H)'D1($'A#+?C$7)9'
< :$6';"%4I+%)'*%4C"G*%8'*%($#)$5'G$&5'$JA5')E#&5'C+8'+;'!+C+#K1&#*J'
< L"CC')"AA+#&';+#'MA$%KN'300$%1?C$0+A$%7A9'
! NC1%)'
< =OPQ'3D1($')+7$'4+G$'A*$4$)';#+7'R=K9'
< =C"$'S1&$#)'
< T)$'=UVWPUVNV!XY'
James C. Osborn - USQCD Software 2011 9
QDP
! !"##$%&'($#)*+%',-.-/012'314)+'*%'5()'6'7*&8'9 :"4;041<5$')"==+#&'3>*&?'")$#')"==4*$@'41A+"&)8'
9 B*&$4++='C15#+'
! D+))*E4$'*C=#+($C$%&)'9 !+C=4$&$'C"4;41<5$')?*F)'3@"=4*51;+%)'1%@'#$@"5;+%)8'
9 G=$%:D')"==+#&'3*%)&$1@'+H'*%'IJK8'
9 L?#$1@'M7#+"=)N'9'51%'?1($'*%@$=$%@$%&'7#+"=)'+H'&?#$1@)'>+#O*%7'+%'@*P$#$%&'&1)O)'
QDP_loop_sites(i, s, { QLA_ColorMatrix *mi = QDP_site_ptr_readwrite_M(m, i); QLA_DiracFermion *di = QDP_site_ptr_readonly_D(d, i); QLA_c_eq_r_times_c(QLA_elem_M(*mi,0,0), 2., QLA_elem_D(*di,1,2)); });
James C. Osborn - USQCD Software 2011 10
QOPQDP
! !"##$%&'($#)*+%',-./-0'123)+'*%'4()'5'6*&7'8 9*)4$332%$+")':;$)'2%<'*=>#+($=$%&)'
8 ?@AB')">>+#&'*%'>#+6#$))'1C3$;$*'D'!2#3$&+%7'
! E+))*F3$'4G2%6$)'8 H$$<)'=2I+#'"><2&$'
J ?2%<3$'=*;$<'>#$4*)*+%'6#24$K"33L'1#$)&#"4&"#$'G$2<$#)7'
J 9"3M>3$'32N4$)'14"##$%&3L'*%*M23*O$<'P*&G':;$<'32L+"&7'
J Q&G$#'H4'1R'2%<'2#F*#L7'8 C<<'="3M6#*<'
8 C<<'$*6$%)+3($#)'
James C. Osborn - USQCD Software 2011 11
(Q)MG
! !"#$%&'()*"+,#(-.*/'#%0(12345367(
! 8"(9:#/).*:#(;"-,(<"#()"0,(= >,?@#%)/"&(:&0(9#"*"&':/"&(A,*"&'?(%&(BCD(
= >,?@(9#"A:A*E(A,*"&'?(%&(BFDBCD(
! G"0,(H"#$?(:&0(%&(9#"0.)/"&I(A.@(?/**(-:&E("99"#@.&%/,?(<"#(%-9#"+,-,&@(
! J:#',(&.-A,#("<(9:#:-,@,#?(@"(@.&,(
! C"-:%&(H:**(1K:.*7(:&0(K@:'',#,0(1L:-,?7(%&(9#"'#,??(
James C. Osborn - USQCD Software 2011 12
QLUA
! !"#$%&'()*+',-"'.*)/$0'#$+'+*1*2-34*$.'
! 5-6'&#)''7 89:8;:')-21*")'<#)=.#+>'?@A8>'%2-1*">';BC'
7 D2-1*"'EF'
7 GH0*$)-21*")'<#)=.#+>'?@A8>';BC'
! EF'I'*H0*$)-21*")'#"*'.*43-"#"J'&#%K)'($/2'89:8;:')(33-".)'.&*4''
Pochinsky
Friday, May 6, 2011
QLUA: LQCD SCRIPTING
based on the Lua programming language
Integrated with QDP & QIO
seamless Level III extensions
platform-independent
in production use
QLUA FEATURES
Full programming language (variables, routines, &c)
automatic memory management (including QCD data)
Latent types
Available Level III packages
Möbius Domain Wall Fermions
Clover Wilson(including EigCG)
QDPOP
QLUA POINTERS
http://www.lua.org/
https://lattice.lns.mit.edu/trac/downloads
https://usqcd.lns.mit.edu/wiki/QLUA_tutorials
Babich
Friday, May 6, 2011
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD D
!"#"$%&'%"($%)*+,%-./0#01
/012$3"456&78$9"7:;*"<>8'7?&7@$AB=$ACDD
!"#$%&'()*%";5"#$,#(E87;(5@
"#$'8*&F4$"4$5*8$0,2G$H8E8F"<87;$I$)"#57('?5"7;J
!()*$%7"687=$K(:8$1F&7:=$L"8F$M(8H5=$358E8$M"55F(8'=$%NF(#5$L"O=$1F&?H("$!8''(=$M?")*?#$3*(=$&#H$GF8P8($3578F)*8#:"
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD E
!"##$%&'($)&"#$*
! /&58;5$78F8&;8G$H87;("#$(;$CIJIAI
! K#)F?G8;$;(#LF8MNO,$;"FH87;$4"7P
! 9(F;"#$&#G$9(F;"#M)F"H87
! Q6(;58G$R&;;
! KR<7"H8G$;5&LL878G$+&;S5>UK30-=$'?5$#"5$@85$#&VH8
! 2"R&(#$6&FF
! %"5*$1N$&#G$%(1N;5&'$&78$;?<<"758G=$(#)F?G(#L$R(W8G$<78)(;("#$6(5*$X78F(&'F8$?<G&58;IY
! 35&LL878G$)"G8$G"8;$R"78$'8;(G8;$+;88$?<G&58$47"R$358H8-I
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD E
!"#$%$&'$()*$&+,#$'
! FG&(H&'H8$(#$<?'H()$78<";(5"7@$'?5$#"5$5*"7"?I*H@$58;58JK
! L?H5(MNO,$9(H;"#$P$9(H;"#M)H"G87$?;(#I$0LO
! L?H5(MNO,$;5&II878J$?;(#I$LOQ
! 2;H&;*$&?5"M5?#(#I$&5$7?#5(R8$+(#$&JJ(5("#$5"$8S(;5(#I$)"R<(H8M5(R8$T%/F3U$&?5"M5?#(#I-
! L?H5(M;*(45$;"HG87$I8#87&H(V8J$'8@"#J$;5&II878J
! FG&(H&'H8$G87@$;""#$+688:;-K
! L?H5(MNO,$9(H;"#=$9(H;"#M)H"G87=$;5&II878J=$&#J$56(;58J$R&;;$5"$;?<<"75$'"5*$LOQ$&#J$0LO=$(#)H?J(#I$784878#)8$*";5$(R<H8R8#5&5("#;W
! 2"R&(#$6&HH$"<5(R(V&5("#;$+&#J$78;57?)5?7(#I$5"$4&)(H(5&58$R?H5(MNO,-
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DC
!"#$%&'()**%+,'-).,#/012'#-3.%-%+,4,#5+
6.#$%&'78.#+,'95:
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DD
!"#$%#&'()"*+,*-.$/
! EF&(G&'G8$;""#(;*$+DHI$J"#5*;-K
! L?G5(HMN,$<&7&GG8G(O&5("#$(#$J"78$5*&#$"#8$P(J8#;("#=$
P8F8G"<8P$;(J?G5"?;G@$4"7$9(G;"#HG(:8$&#P$
;5&QQ878P$487J("#;R
! L?G5(HMN,$P"J&(#$6&GGR
! S#5874&)8$"F87*&?GT$$+388$'8G"6R-
! /"#Q87H587JK
! EPP(5("#&G$7"?5(#8;K$4"7)8;=$U?&7:$;J8&7(#Q=$
8(Q8#;"GF87;=$85)R
! L?G5(Q7(P$4"7$9(G;"#$&#P$9(G;"#H)G"F87
! 1"#5(#?8P$8V<G"7&5("#$"4$&GQ"7(5*J;$+P"J&(#$
P8)"J<";(5("#=$85)R-
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DA
!"#$%#&'()"*+,-'."
! !8;?E5;$&78$4"7$5*8$8F8#G"HH$<78)"#H(5("#8H$)E"F87I(J<7"F8H$9(E;"#$J&57(KIF8)5"7$<7"H?)5$+L2;E&;*M-N
! !?#;$6878$H"#8$"#$&$O8>"7)8$OPQ$RSC$+)"#;?J87$L>87J(M$)&7H-$6(5*$;E(T*5E@$"?5I"4IH&58$0,2UV
! /&58;5$F87;("#$(#)E?H8;$2;E&;*$&?5"I5?#(#T$&#H$"5*87$8#*&#)8J8#5;=$T(F(#T$<87*&<;$&$DCW$'?J<V
! >"7$784878#)8=$&$;5&#H&7H$H?&EI;"):85$#"H8$6(5*$78)8#5$+98;5J878-$X?&HI)"78$Q8"#;$6"?EH$;?;5&(#$&7"?#H$/0*1$2%,3$(#$;(#TE8$<78)(;("#$4"7$&$68EEI"<5(J(Y8H$9(E;"#I)E"F87$2;E&;*V$$P*8$2;$)E?;587Z;$X?&HI;"):85$[<587"#$#"H8;$H"$'85587$5*&#$40*1$2%,35
! 3<&5(&E$F"E?J8$(;$*8EH$4(K8H$&5$AR\V
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"#$%&'(%&)#&*+,-%'./0,1"%'(&%-0/0#,2
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"#$"%&'()*+),$-.(&/%$#*&')(.0"0+-1
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DB
!"#$"%&'()*+),$-.(&/0+12#(&')(.3"3+-4
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"#$"%&'()*+),$-.(&"/,,$)0
! 3?FF&7(G8H$78;?I5;$&78$4"7$&$)"#;87J&5(J8$)&;8K$DAL
78)"#;57?)5$+#"#8$4"7$H"?'I8-$6(5*$#"$58F<"7&I$M&?M8L4(N(#MO
! 3(#MI8$&#H$*&I4$<874"7F&#)8$&78$&'"?5$AOEN$&#H$BODN$*(M*87$5*&#$H"?'I8O
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"#$%&$'%
! 0,2F$#"6$G(H8;$&5$*55<IJJG&55()8KL(5*?'K)"MJN?O&
! >?5?78$78G8&;8;$6(GG$'8$&##"?#)8O$"#$5*8$0,2FP&##"?#)8$M&(G(#L$G(;5K$$Q"(#$&5$5*8$68';(58$"7$'@$8M&(G(#L$$$$$$$$$$$$$$$$N?O&P&##"?#)8R;?';)7('8SL""LG8L7"?<;K)"M$K
! 3"?7)8$)"O8$78<";(5"7@$*&;$'88#$M"H8O$"H87$5"$T(5U?'K
! <7"H(O8;$#()8$48&5?78;$;?)*$&#$(;;?8$57&):87=$;"?7)8$)"O8$'7"6;(#L=$)"MM8#5(#L$"#$)"MM(5;=$85)K
! T(5$(5;8G4$&GG"6;$4"7$M"78$4G8V('G8$)"GG&'"7&5("#$&#O$W<&7&GG8GX$)"O8$O8H8G"<M8#5K
! Y"L85*87$5*8;8$;*"?GO$M&:8$(5$8&;(87$4"7$W"?5;(O87;X$5"$)"#57('?58K
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"#$%&'()*+,-)%./)*0,123
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD DE
!"##$%&'(%&$#)*+$'*%,'-(.(&*&(/%0
! 0,2FG;$(#5874&)8$(;$"H87I?8$4"7$;"J8$)*&#K8;L
! 1?778#5M@=$(#)M?I(#K$!"#$%&$K85;$@"?$&$*&#I4?M$"4$4?#)5("#;L$%8;(I8;$5*";8$4"7$(#(5(&M(N(#K$&#I$4(#&M(N(#K$5*8$M('7&7@=$5*8$J";5$)"JJ"#M@$)&MM8I$&78'()$#*$"+,-"#$.-$&#I$/01,23-"#$.4L
! ()$#*$"+,-"#$.+$"+,5'+$"+,67$2$8-$M"&I;$5*8$K&?K8$4(8MI$"#5"$5*8$OP,L
! /01,23-"#$.9)("3/)05'9)"2:,5'/0167$2$8-$M"&I;$5*8$;"?7)8$"#5"$5*8$OP,=$<874"7J;$5*8$(#H87;("#=$&#I$785?7#;$5*8$;"M?5("#L
! Q878$+$"+,=$9)("3/)0=$&#I$9)"2:,$&78$&MM$<"(#587;$5"$4(8MI;$"#$5*8$!"#$L
! R*878$(;$#"$<7"H(;("#$4"7$J&#(<?M&5(#K$4(8MI;$"#$5*8$OP,=$6(5*"?5$I(H(#K$(#5"$5*8$0,2FG;$(#587#&M;L
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AC
!"##$%&'($)*+",')'-.#
! /$#0&1$)'+2$E*(;$(;$F?;5$&$;57&6GH&#$I$<87;"#&J$6(;*$J(;5K$ $285&(J8L$H8)*&#();=$#&H(#M$)"#N8#5("#;=$85)K$*&N8$@85$5"$'8$L(;)?;;8L$'@$5*8$0,2O#&?5;K
! 98$<7"'&'J@$#88L$;"H85*(#M$J(:8$5*8$4"JJ"6(#MP
! Q878$;"?7)8=$;"J?5("#=$85)K$&78$"<&R?8$S"'F8)5;T$5*&5$:#"6$&'"?5$4(8JL;$"#$5*8$UV,K
!"#$%$&'()*+,-*!"##./,01.234$+1.24$!"#*5"6"2789$%$&'()*+,-*!.+:*;.,!10!"#4$<"9<,*5"6"2783=96/,$%$&'()*+,-*3.#,*;.,!10!"#4$35.+=6*5"6"2783=!9#.=+$%$&'()*+,-*3.#,*;.,!10!"#4$35.+=6*5"6"278&'()*!="1*!.+:*;.,!1094$>=3#*94$<"9<,*=61,678&'()*!="1*3.#,*;.,!103=96/,4$>=3#*3=96/,4$35.+=6*=61,678&'()*3=!?,03=!9#.=+4$3=96/,4$94$3=!?,678&'()*3"?,*3.#,*;.,!103=!9#.=+4$>=3#*3=!9#.=+4$35.+=6*=61,678&'()*1,3#6=@*3.#,*;.,!103=96/,78,#/AAA
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AD
!"#$%&"'(
! E*(;$F85;$5*8$?;87$<(<8F(#8$&$;87(8;$"4$"<87&5("#;$"#$5*8$GH,=$6(5*"?5$I"J(#K$L&5&$'&):$&#L$4"75*M$$>"7$8N&I<F8O
! 3?<<"75;$I?F5(<F8$F&55()8;$&#L$K8#87&F$PF(#:$4(8FL;Q$#88L8L$'@$I?F5(K7(L
! R"78$I"L?F&7=$I&:(#K$(5$<";;('F8$5"$)""<87&58$6(5*$?;87$GH,$)"L8$+#8N5$;F(L8-
!"#$%&'()%*+,-%.+-&)/*'012-345'*,%*'012-34*6+7'1%'1)-189!"#$%*:-(1/*:-(1-)%*'012-34*'012-34*:-(1%6(1(:89!"#$%*'&;-/*'&0,+'734*:-(1-)%*'012-34034*'&;-189<<<!"#$%*(;-%*+,-%.+-&)/*'&0,+'7345'*,%*'&0,+'734*6+7'1%'1)-189
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AA
!"#$%&'$%()*+*#,
! 0,2E$)"F<G858G@$(#;?G&58;$5*8$?;87$47"F$HI,$)"J8K$$L#$<7(#)(<G8=$(5$;*"?GJ$8M8#$'8$<";;('G8$5"$;?<<"75$'&):8#J;$'8;(J8;$1,2E$+8KNK=$O<8#1/-K
! O#8$)&#$(F&N(#8$)&;8;=$*"68M87=$6*878$&$;&MM@$?;87$6"?GJ$6$5"$&))8;;$HI,$4(8GJ;$"6#8J$'@$0,2EK$$3"F85*(#N$G(:8$5*8$4"GG"6(#N$F(N*5$6"7:P
!"#$%&'()*+&',-./!"#$%&'()*$&',01-./)22)3456)0"7()%"80,09!"#$%&'()*+&',:(;<=>(:$&',-./-)-)-
?%=,1)@<0=<ABCDE)22)1=).=%')<="#1(0>)1=)FG4)7(7=09H456:>=%I(J>=%&1"=#K)>7(,0(':>=&0$(K)&K)>=%I(0LE<0=<AMD)N)H456:(;<=>(:$&',:>"1(:?"(%'J>=%&1"=#LE-)-)-22)&>(0)3456)O(0#(%$=#10,$1***#8%=$O>K)#1.0(,'>///J#&$%(=#C<1K)<0=<LE-)-)-
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AE
!"#$$%&'%()*+),-$./0123(.4*&')(5#$/&'
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD
AE
!"#$%&'()*+,$%-.$%,/
! '()*+0+,123$>"7$5*7"?F*<?5$G"';$+8HFH=$)"I<?5(#F$
<7"<&F&5"7;-=$(5$;?44()8;$5"$?;8$5*8$;I&JJ8;5$#?I'87$"4$KL,;$
5*&5$6(JJ$4(5$5*8$G"'=$'?5$"458#$"#8$KL,$(;#M5$8#"?F*H
! 4,5$*+0+,123*N5M;$F8#87&JJ@$I";5$)";5O8448)5(P8$5"$<?5$I"78$
5*&#$"#8$KL,$(#$&$#"Q8H$$R*8;8$)&#$'8$?;8Q$(#$&#$
8I'&77&;;(#FJ@$<&7&JJ8J$4&;*("#$+'@$7?##(#F$I?J5(<J8$
;8<&7&58$G"';-=$'?5$5*8#$*";5$I8I"7@$'8)"I8;$&$)"#;57&(#5H
! 6.7.8%#%$23$98MQ$J(:8$5"$'7"&Q8#$5*8$7&#F8$"4$<7"'J8I;$5"$
6*()*$KL,;$&78$&<<J()&'J8$+8HFH=$F&?F8$F8#87&5("#-H$$S878$
!"#$%&'!()*+%&$(;$:8@H
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AB
!"#$$%&'%()*+)(,#$-&')./
! EF,G5"G*";5$&#H$(#587G#"H8$0#&12-1*"
! EF,G5"G*";5$&#H$(#587G#"H8$ $$#*%&,3
4)565)789(
:;<)=&>-&-0#&1)?#0@-,
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AE
!"#$%&'""$&#()*"+,-+#.
+FGH$'&#I6(I5*$&;$J8&;?78I$'@$K85GHGL$MNOPOD-
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AE
!"#$%&'()*(+,
+FG,$'&#H6(H5*$I8&;?78H$6(5*$1,2J$32K$LAMN-$$$$ $$$$$$$$$$$O!"#$%&$'()*+',--.*./0123&##*$4
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AE
!"#$%&$'()*+,-)./"+01
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD AE
!"#$%#&'()"*&%+",
F#;<(78G$'@$H"55I(8'$+ACCC-=$*55<JKK<*@;();L(#G(&#&L8G?KM;NK<)#85;K
$$$$$$$$$$O(&$P"IQN78#$+ACCR-=$&7S(OJ*8<TI&5KCRDCCRE
! >"7$5*8$9(I;"#$Q&57(UTO8)5"7$<7"G?)5=$68$*&O8J
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EC
!"#$%#&'()"*&%+",
! F*(;$G"H8I$(;$<8;;(G(;5()$(#$5*8$;8#;8$5*&5$(5$&;;?G8;$68$&78$J"(#J$5"$<&7&II8I(K8$(#$&II$L$H(G8#;("#;M$$>"7$;G&II$#?G'87;$"4$#"H8;=$5*(;$(;$#8N87$"<5(G&IM
! O5$(;$"<5(G(;5()$(#$&II$"5*87$78;<8)5;$+&;;?G(#J$<8748)5$"N87I&<<(#J$"4$)"GG?#()&5("#$&#H$)"G<?5&5("#=$4"7$8P&G<I8-=$58II(#J$?;$5*8$'8;5$68$)&#$<";;('I@$H"M
! >"7$5*(;$8P87)(;8=$68Q78$(#5878;58H$(#$5*8$!"#$%&'!()*+%&$78J(G8$+;G&II8;5$<";;('I8$;?'RN"I?G8;-M$$S"6$;G&II$)&#$68$J"$'84"78$5*8$;?74&)8TN"I?G8$7&5("$I(G(5;$?;U
! F*(;$(;$V?;5$&$G"H8I=$'?5$O$H"#Q5$;88$&#@$I""<*"I8;$6(5*"?5$78;"75(#J$5"$#86$&IJ"7(5*G;M
! 98QII$*&N8$&)5?&I$8P<87(8#)8$N87@$;""#M
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD ED
!"#$%&"'()*+',%'-.
$$$/$F$G=$$$$$$$H=$$$$I=$DC=$DA=$DG=$DH=$DI=$AC=$AG=$EA=$GI=$HG
AC$J4K"<;AC$J4K"<;
DCC$J4K"<;DCC$J4K"<;
BCC$J4K"<;BCC$J4K"<;
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EA
!"#$%$"&$'$(()("*'$+),-'.*-'"/01"-2"34567
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EE
!"#$$%&'%()*+,#"$-.)$#/
! F5$"458#$(;#G5$8#"?H*$5"$;<88I$?<$J?;5$5*8$(#K87587L
! M5*87$"<87&5("#;$+8LHL=$<7"<&H&5"7$)"#57&)5("#;=$N?&7:$4(8OI$;P8&7(#H-$6(OO$8K8#5?&OO@$'8)"P8$5*8$'"55O8#8):=$78N?(7(#H$8(5*87$P"78$QR,$)"I8$"7$&$)*&#H8$(#$6"7:4O"6$+8LHL=$67(5(#H$"?5$<7"<&H&5"7;$&;$&#$(#587P8I(&58$;58<-L
3"?7)8S$*55<STT8#L6(:(<8I(&L"7HT6(:(T>(O8SUPI&*O;/&6L;KH
QR,V&))8O87&58I
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EF
!"#$%&%'"(
! 3"G8$"<87&5("#;$H8;87I8$"<5(G(J8H$7"?5(#8;$"4$5*8(7$"6#=$5"$
'8$&HH8H$5"$0,2KL
! M5*87;$&78$<7"'N8GO;<8)(4()$"7$)"#;(;5$"4$;(G<N8$PQN?8R$)"H8L
! M#8$)"?NH$&N6&@;$67(58$5*8;8$<(8)8;$(#$1,2K=$'?5$)&#$68$
G&:8$5*(;$<7")8;;$8&;(87S
! T"58$5*&5$<874"7G&#)8$"458#$(;#U5$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
)7(5()&N$*878L$$V*8$(G<"75$5*(#Q$(;$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
5"$&I"(H$G"I(#Q$H&5&$"44$5*8$WX,L
! Y#$5*8$N&#Q?&Q8$"4$5*8$,3012$;"456&78$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
;5&):=$0,2K$(;$&$P/8I8N$ER$<&):&Q8L
! K;$"4$@85=$5*878$(;$#"$/8I8N$A$N('7&7@$4"7$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
WX,;L$$+0,2K$H"8;$)"#5&(#$)"G<"#8#5;$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
5*&5$G(Q*5$'8$?;84?N$4"7$'?(NH(#Q$;?)*$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
&$5*(#QL-
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EB
!"#$%&'()&*$+,-
! F$"#G@$H8&#$5"$<";8$5*8$I?8;5("#=$#"5$&#;687$(5J
! 3(#)8$FKH$H";5$4&H(G(&7$6(5*$02LM1=$5*8$4"GG"6(#N$5*"?N*5$8O<87(H8#5$;88H;$#&5?7&GP$9*&5$;"75$"4$G&@87$6"?GQ$68$#88Q$5"$7?#$"?7$02LM1$)"Q8$"#$RL,;$6(5*$H(#(H&G$)*&#N8;S
! 3(H<G@$<"75(#N$02L$5"$RL,;$<7"'&'G@$(;#K5$<";;('G8=$'?5$)&#$68$)"H8$)G";8S
! 02LM1$'&;()&GG@$<7"T(Q8;P
DJ$3(58UG")&G$"<87&5("#;$<874"7H8Q$(#$<&7&GG8GP$T8)5"7$&QQ(5("#=$3,+E-$H?G5(<G()&5("#=$;<(#$<7"V8)5("#=$85)J$ $W*(;$(;$Q"&'G8J
AJ$3?<<"75$4"7$&7'(57&7@$;?';85;$+Q"&'G8=$6(5*$&$<874"7H&#)8$<8#&G5@-J
EJ$X<87&5("#;$(#T"GT(#N$;*(45;J$$3?<<"75(#N$N8#87&G$;*(45;$6"?GQ$'8$*&7Q=$'?5$&$;?';85$H(N*5$'8$8#"?N*J
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EF
!"#$$%&'%()*(+"%(&,&-./0(1234
! G#$<7&)5()8=$78&H$)"I8$)"#;(;5;$"4$J"78$5*&#$K?;5$02L$)&HH;M$ $98$"458#$N'78&:$"?5O$"4$02L$&#I$<874"7J$;(58PH")&H$"<87&5("#;$?;(#Q$0/RM
! L"75(#Q$0/R$(#$&HH$(5;$Q8#87&H(5@$6"?HI$'8$!"#$%&&#I$0/R$)"I8$6"?HI$<874"7J$5877('H@$?#H8;;$"<87&5("#;$&78$4?;8I$5"Q85*87M
! !8&;"#&'H8$&H587#&5(S8$J(Q*5$'8$5"$78P67(58$5*8;8$<"75("#;$(#$1,2R$I(78)5H@=$<87*&<;$?;(#Q$&$)"HH8)5("#$"4$0/RPH(:8$!!"#$%&#!!$4?#)5("#;$"7$&$)"#S8#(8#)8$H('7&7@$;?)*$&;$'!#()*M
!"#$%&'()*$+%,-$.$/012$3"456&78$9"7:;*"<=$>8'7?&7@$AB=$ACDD EF
!"#$$%&'%()*(+%,%-.'%&%/,0
! GHI8$;"$4&7$&;;?J8K$5*&5$!""$5*8$6"7:$(;$K"#8$"#$5*8$LM,N
! O*(;$6"#H5$'8$<7&)5()&P$(#$J&#@$)&;8;=$K?8$5"$P(J(58K$LM,$J8J"7@$(4$#"5*(#Q$8P;8N
! 98$6"?PK$#88K$'"5*$&$*";5R;(K8$&#K$LM,R;(K8$02M=$6(5*$4?#)5("#;$4"7$57&#;4877(#Q$K&5&$'85688#$5*8J$+"458#$78"7K87(#Q$&77&@;$(#$5*8$<7")8;;-N
! S7$<87*&<;$68$)&#$T?;5$578&5$5*8$*";5$&;$&$;5"7&Q8$78;"?7)8U$ $VWX";5Y$J8J"7@$(;$5*8$#86$K(;:NZ$$O*(;$6&;58;$5*8$1M,=$*"68I87=$&#K$1M,;$&78$?#K8#(&'P@$'85587$&5$;"J8$5*(#Q;N
! [(5*87$6&@=$5*8$<7"Q7&JJ87$6(PP$'8$4"7)8K$5"$J&:8$)*"()8;=$J&#@$"4$6*()*$K8<8#K$"#$5*8$5&7Q85$J&)*(#8N
! S?7$;(J<P8$<7"Q7&JJ(#Q$J"K8P$(;$'8Q(##(#Q$5"$'78&:$K"6#N$ $M87*&<;$02M$(;$#"5$5*8$7(Q*5$<P&)8$5"$;5&75$N$N$N
Friday, May 6, 2011