spark advanced devops
TRANSCRIPT
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 1/219
!"#$%&
'!#'()"! )+'&&
,-./0 12345 &6-.7 &899:; "-<; 1234
0;;65==<>:?@<0-.@AB@;=?-;-C.:/7<
DDDA>:B7@?:BA/E9=:B=C>8@6>-<;:/
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 2/219
9-7:BF C:F ?-;- <:96>@
!-;-C.:/7< )>E8?5“A 8B:G:@? 6>-;GE.9 GE. C
– G.E9 "I+ ;E "J6>E.-;:EB '?L-B/@? 'B->M;:/< -B?
•
NE8B?@? :B >-;@ 123O• CM ;0@ /.@-;E.< EG '6-/0@ &6-.7
• $.:F:B-> ;@-9 G.E9 P) [email protected]@>@M ',%+-C
• Q-:<@? RST ,:>>:EB :B 1 .E8B?<
• U42 @96>EM@@<
• We’re hiring!
• +@L@> 1=O <866E.; 6-.;B@.<0:6< D:;0
• VE.;EBDE.7<
• ,-6Q
• !-;-&;-J
W0;;65==?-;-C.:/7<ADE.7-C>@A/E9 X
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 3/219
I0@ !-;-C.:/7< ;@-9 /EB;.:C8;@? 9E.@ ;0-B !"# EG ;0@ /E?@ -??@? ;E &6-.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 4/219
'Y"(!'
•
V:<;E.M EG &6-.7
• Q!! G8B?-9@B;-><
• &6-.7 Q8B;:9@ './0:;@/;8.@ZB;@F.-;:EB D:;0 Q@<E8./@ ,-B-F@.<W&;-B?->EB@K ['Q(X
• YPZ<
•
+-C5 !@L$6< 323
H@GE.@ +8B/0
•
,@9E.M -B? %@.<:<;@B/@
• \EC< ]^ &;-F@< ]^ I-<7<
• H.E-?/-<; #-.:-C>@< -B?'//898>-;E.<
• %M&6-.7
• !@L$6< 321
• &08GG>@
• &6-.7 &;.@-9:BF
'G;@. +8B/0
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 5/219
&E9@ <>:?@< D:>> C@ <7:66@?
%>@-<@ 7@@6 _`' >ED ?8.:BF />-<<
W469 – 45O269 GE. _`' D:;0 :B<;.8/;E.X
1 -BEBM9E8< <8.L@M<5 %.@ -B? %E<; />-<<
+8B/05 BEEB – 369
1 C.@-7< WC@GE.@ >8B/0 -B? -G;@. >8B/0X
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 6/219
a
• ',%+-C 6.Eb@/; D-< >-8B/0@? :B \-B 1233K c M@-. 6>-BB@? ?8.-;:EB
• %@.<EBB@>5 Uc4 <;8?@B;<K 6E<;?E/<K G-/8>;M ̀ <;-GG
• N8B?:BF G.E9 [email protected]@B;=ZB?8<;.M 6-.;B@.<0:6K (&N 'D-.?K !-.6-K !E"K12d /E96-B:@<
• ).@-;@? H!'&K ,@<E<K &('%A P6/E9:BF 6.Eb@/;<5 &8//:B/; ̀ #@>EJA
“Unknown to most of the world, the University of California, Berkeley’s ',%+0-< ->.@-?M >@G; -B :B?@>:C>@ 9-.7 EB ;0@ DE.>? EG :BGE.9-;:EB ;@/0BE>EFMK -even the web. But we haven’t yet experienced the full impact of thegroup[…] Not even close”
] !@..:/7 V-..:<K Y:F-$9K '8F 123S
'>FE.:;09<,-/0:B@<%@E6>@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 7/219
&/0@?8>:BF ,EB:;E.:BF !:<;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 8/219
!"#$%&'()
!*+
,#%-./
9'8"#':5"'2(8;< =90< 09>< ?%-@< 9!A
B%C.D2(
?+E':
9%"%6#%&$8 7>4
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 9/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 10/219
%&'&()* +),-. /(0-&112'3
/(&3&*
4(&5&*
657)*)%()7.8)9
%2()7.
4(2**:&;
<=<,0(5
<7&-2)*2;&> <?1,&51
W:;@.-;:L@K :B;@.-/;:L@K ,+K <;.@-9:BFK F.-60K &_+K @;/X%&'&
W122S – 123OX
W122T – 1234eX
C).0D,
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 11/219
ED3 FGGH
AAA:B \8B@ 123O
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 12/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 13/219
FGH –
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 14/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 15/219
\8B@ 1232
0;;65==DDDA/<[email protected]@>@MA@?8=U9-;@:=6-6@.<=1232=
“The 9-:B -C<;.-/;:EB :B &6-.7 :< ;0-; ;.:C8;@? ?-;-<@; WQ!!XK D0:/0 .@6.@<@/E>>@/;:EB EG ECb@/;< 6-.;:;:EB@? -/.E9-/0:B@< ;0-; /-B C@ .@C8:>; :G - 6-.;
P<@.< /-B @J6>:/:;>M /-/0@ -B Q!! :B 9-/0:B@< -B? .@8<@ :; :B 98>;:6>@ ,-66-.->>@> [email protected];:EB<A
Q!!< -/0:@L@ G-8>; ;E>@.-B/@ ;0.E8F0>:B@-F@5 :G - 6-.;:;:EB EG -B Q!! :< >E<;@BE8F0 :BGE.9-;:EB -CE8; 0ED :; D-<E;0@. Q!!< ;E C@ -C>@ ;E .@C8:>? b8<; ;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 16/219
'6.:> 1231
0;;65==DDDA/<[email protected]@>@MA@?8=U9-;@:=6-6@.<=1231
“We 6.@<@B; Q@<:>:@B; !:<;.:C8;@? !-;-?:<;.:C8;@? [email protected] -C<;.-/;:EB ;0-; >6.EF.-99@.< [email protected] :B][email protected] /E9>-.F@ />8<;@.< :B - G-8>;];E>@.-B; 9-BB
Q!!< -.@ 9E;:L-;@? CM ;DE ;M6@< EG -;0-; /8..@B; /E968;:BF [email protected]< 0:B@GG:/:@B;>M5 :;@.-;:L@ ->FE.:;09< -B? :9:B:BF ;EE><A
ZB CE;0 /-<@<K 7@@6:BF ?-;- :B 9@[email protected]/@ CM -B E.?@. EG 9-FB:;8
“Best %-6@. 'D-.? -B? VEBE.-C>@ ,@B;:EB GE. )
] ):;@? Og1 ;:9@<h
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 17/219
!"#$$%&'$#()*+&%,$%-$&%,.*1#($%&/2*3%$!%4$*+56*+5:6$;<=#6>5"/-%+56>
] 1 &;.@-9:BF %[email protected]<X 0-L@ C@@B /:;@?
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 18/219
)@(A$4 B 6%" D#E%A56$%4$/)&%):($) B )@(A$4*)@(/7-FGFA! H IJKL 8%58(%70
6,.%) B &%):($)*.,8/(,.M>,
Seemlessly mix SQL queries with Spark programs.
)E9:BF <EEBh
!"#$$ &' ()&$#*+', #- .+' )(/01#-23''4* 506 789:;< =>?@A
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 19/219
3&,8O B P&,8O/E%&$#+%)Q %.%)),3%) B )8,&9*$%4$I#(%3&,8OS B 3&,8O*T5#6U%&$#+
/#>Q E%&$%4Q .)30 BW **X
0;;6<5==-96>-CA/<[email protected]@>@MA@?8=D/EB;@B;=86>E-?<[email protected]:;0fGEB;<A6?G
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 20/219
0;;6<5==DDDA/<[email protected]@>@MA@?8=U<- [email protected]<M<3OA6?G
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 21/219
0;;65==<0E6AE.@:>>MA/E9=6.E?8/;=2
@HEE75 ROOAgg%.:B;5 ROgAgg
%!NK @%8
&0:66:BF
0;;65==DDDA-9-jEBA/[email protected]:BFN-<;]!-;-]'B->M<:<=?6=3SSgO4ic1S
RO2 a '9-jEB5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 22/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 23/219
0;;65==;:BM8.>A/E9=?
] 321 6-F@<
] !@L$6< <;M>@
] NE. /E96>@;@ C
] ZB/>8?@<5
] &6-.7 &;.@-] !-BF@.< E
Y.E86HMk@Q@?8/@HM
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 24/219
0;;65==;:BM8.>A/E9=/
] 32g 6-F@<
] !@L$6< <;M>@
] NE. /E96>@;@
] ZB/>8?@<5] %M&6-.7 ] &6-.7 &_+] &6-.7]<8C9
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 25/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 26/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 27/219
W&/->- ` %M;0EB EB>MX
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 28/219
!.:L@. %.EF.-9
!"!""
#
!""
I
I
!"
!""
#
!""II
lE.7@. ,-/0:B@
lE.7@. ,-/0:B@
# #
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 29/219
'"$&<F
'"$&<O'"$&<N'"$&<K'"$&<L
'"$&<P
'"$&<Q'"$&<R'"$&<S'"$&<FG
'"$&<FF
'"$&<FO'"$&<FN'"$&<FK'"$&<FL
'"$&<FP
'"$&<FQ'"$&<FR'"$&<FS'"$&<OG
'"$&<OF
'"$&<OO'"$&<ON'"$&<OK'"$&<OL
@99
!"!""
#
!""
!"!""
#
!""
!"!""
#
106' (B6.#.#0-* C
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 30/219
A##2#T "8T &8)FU%#(T "8T &8)OA##2#T "8T &8)F
@99 VW K -%#"'"'2(8
4(X2T "8T &8)RU%#(T "8T &8)O4(X2T "8T &8)R
A##2#T "8T &8)N4(X2T "8T &8)L4(X2T "8T &8)L
A##2#T "8T &8)KU%#(T "8T &8)SA##2#T "8T &8)F
'B Q!! /-B C@ /.@-;@? 1 D-M<5
] %-.->>@>:j@ - /E>>@/;:EB] Q@-? ?-;- G.E9 -B @J;@.B-> <E8./@ W&OK )mK V!N
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 31/219
Y Z,&,((%(#[% #6 Z<$O56"5&>)J\\ B )+*8,&,((%(#[%([“fish", “cats“, “dogs”])
RR Z,&,((%(#[% #6 -+,(,E,( "5&>)J\\B )+*8,&,((%(#[%/G#)$/71#)O7Q 7+,$)7Q 7>53)700
RR Z,&,((%(#[% #6 ],E,],E,J\\^-$?W "5&>)J\\ B )+*8,&,((%(#[%/_&&,<)*,)G#)$(“fish", “cats“, “dogs”)
] I-7@ -B/E>>@/;SparkC
9@;0E?
] (E; [email protected];E;M.@n8:.@9@9E.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 32/219
Y J%,> , (5+,( $4$ 1#(% #6 Z<$O56(#6%)J\\ B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70
RR J%,> , (5+,( $4$ 1#(% #6 -+,(,E,( (#6%)J\\ B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70
RR J%,> , (5+,( $4$ 1#(% #6 ],E,],E,J\\^-$?W (#6%) B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70`
] I0@.@;E .@-)mK &O
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 33/219
A##2#T "8T &8)FU%#(T "8T &8)OA##2#T "8T &8)F
4(X2T "8T &8)RU%#(T "8T &8)O4(X2T "8T &8)R
A##2#T "8T &8)N4(X2T "8T &8)L4(X2T "8T &8)L
A##2#T "8T &8)KU%#(T "8T &8)SA##2#T "8T &8)F
A##2#T "8T &8)F
A##2#T "8T &8)F
A##2#T "8T &8)N A##2#T "8T &8)K
A##2#T "8T &8)F
IX'E"$#Y Z
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 34/219
IC2%E$8C$Y O Z
A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F
A##2#T "8T &8)K
A##2#T "8T &8)F
CE$%($1@99
A##2#T "8T &8)F
A##2#T "8T &8)F
A##2#T "8T &8)N A##2#T "8T &8)K
A##2#T "8T &8)F
IC2EE$C"Y Z
!.:L@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 35/219
IC2EE$C"Y Z
"J@/8;@ !'Yh
!.:L@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 36/219
IC2EE$C"Y Z
!.:L@.
E2)+'($8@99
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 37/219
IC2EE$C"Y Z
E2)+'($8@99
$##2#8@99
CE$%($1@99
IX'E"$#Y Z
IC2%E$8C$Y O Z
!.:L@.
A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F
A##2#T "8T &8)K
A##2#T "8T &8)F
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 38/219
IC2EE$C"Y Z
!.:L@.
E2)+'($8@99
$##2#8@99
CE$%($1@99
1%"%
IX'E"$#Y Z
IC2%E$8C$Y OT 8.5XXE$[ 6%E8$Z
>'-!"%
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 39/219
!.:L@.
E2)+'($8@99
$##2#8@99
CE$%($1@99
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 40/219
!.:L@.
1%"%
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 41/219
E2)+'($8@99
$##2#8@99
A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F
A##2#T "8T &8)K
A##2#T "8T &8)F
CE$%($1@99
IX'E"$#Y Z
A##2#T "8T &8)F
A##2#T "8T &8)F A##2#T "8T &8)F
$##2#IC2EE$C"Y Z
I8%\$B2=%88%(1#%Y Z
IC25("Y Z
L
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 42/219
E2)+'($8@99
$##2#8@99
A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F
A##2#T "8T &8)K
A##2#T "8T &8)F
CE$%($1@99
IX'E"$#Y Z
A##2#T "8T &8)F
A##2#T "8T &8)F A##2#T "8T &8)F
$##2#IC2EE$C"Y Z
IC25("Y Z
I8%\$B2=%88%(1#%Y Z
L
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 43/219
><F E2)+'($8@99Y0%122-@99Z
><O ><N ><K
><F $##2#8@99YX'E"$#$1@99Z
><O ><N ><K
I-<7]3I-<7]1
I-<7]OI-<7
%-;0 o 0?G<5==A A A
G8B/ = _.contains(…)
<0E8>?)-/0@oG-><@
!-;-<@;]>@L@> L:@D5 %-.;:;:EB]>@L@> L:@D5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 44/219
3X ).@-;@ <E9@ :B68; Q!!< G.E9 @J;@.B-> ?-;- E. 6-.->>@
/E>>@/;:EB :B ME8. ?.:L@. 6.EF.-9A
1X +-j:>M ;.-B<GE.9 ;0@9 ;E ?@G:B@ B@D Q!!< 8<:BF;.-B<GE.9-;:EB< >:7@ 1#($%&/0 E. .,8/0
OX '<7 &6-.7 ;E +,+O%/0 -BM :B;@.9@?:-;@ Q!!< ;0-; D:>> B
C@ .@8<@?A
SX +-8B/0 -/;:EB< <8/0 -< +5:6$/0 -B? +5((%+$/0 ;E 7:/6-.->>@> /E968;-;:EBK D0:/0 :< ;0@B E6;:9:j@? -B? @J@CM &6-.7A
W>-jMX
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 45/219
.,8/0 #6$%&)%+$#56/0 +,&$%)#56/0
1(,$L,8/0 >#)$#6+$/0 8#8%/0
1#($%&/0 3&5:8;<a%</0 +5,(%)+%/0
.,8Z,&$#$#56)/0 &%>:+%;<a%</0 &%8,&$#$#56/0
.,8Z,&$#$#56)=#$Ob6>%4/0 )5&$;<a%</0 8,&$#$#56;</0
),.8(%/0 T5#6/0 ***
:6#56/0 +53&5:8/0 ***
] ,E<; ;.-B<GE.9-;:EB< -.@ @>@9@B;]D:<@ W;0@M DE.7 EB EB@ @>@9@B; -; - ;:9@XK C;.8@ GE. ->> ;.-B<GE.9-;:EB<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 46/219
&%>:+%/0 $,9%K&>%&%>/0
+5((%+$/0 ),E%_)!%4$I#(%/0
+5:6$/0 ),E%_)-%@:%6+%I#(%/0
1#&)$/0 ),E%_)KMT%+$I#(%/0
$,9%/0 +5:6$;<a%</0
$,9%-,.8(%/0 15&%,+O/0
),E%!5A,)),6>&,/0 ***
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 47/219
• #$%&&'!""
• ()*+,-,%!""
• .$'',%!""
• /$)-!""
• 01233*,%!""
• 45)&5!""
• /6+1&5!""
• "&27*,!""
• 8%79!""
• 8:&5!""
• 091,;$!""
• <,-+,=!""
• >%?,!""
• @$::$5%-$!""
• A,&!"" !(%)*'
• >:0'$-B !(+#
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 48/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 49/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 50/219
3X &@; EG 6-.;:;:EB< (“splits”)
1X +:<; EG ?@6@B?@B/:@< EB 6-.@B; Q!!<
OX N8B/;:EB ;E /E968;@ - 6-.;:;:EB F:L@B 6-.@B;<
SX $6;:EB-> 6.@G@..@? >E/-;:EB<
4X $6;:EB-> 6-.;:;:EB:BF :BGE GE. 7=L Q!!< W%-.;:;:[email protected]
:.21 -)7,D(&1 )** -D((&', <7)(O 07&(),20'1P
m
m
m
m
m
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 51/219
%-.;:;:EB< o EB@ 6@. V!N& C>E/7
!@6@B?@B/:@< o BEB@
)E968;@ W6-.;:;:EBX o .@-? /E..@<6EB?:BF C>E/7
6.@G@..@?+E/-;:EB< W6-.;X o V!N& C>E/7 >E/-;:EB
%-.;:;:EB@. o BEB@
m
m
m
m
m
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 52/219
%-.;:;:EB< o <-9@ -< 6-.@B; Q!!
Dependencies = “one];E]one” on parent
)E968;@ W6-.;:;:EBX o /E968;@ 6-.@B; -B? G:>;@. :;
6.@G@..@?+E/-;:EB< W6-.;X o BEB@ W-<7 6-.@B;X
%-.;:;:EB@. o BEB@
m
m
m
m
m
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 53/219
%-.;:;:EB< o $B@ 6@. .@?8/@ ;-<7
Dependencies = “shuffle” on each parent
)E968;@ W6-.;:;:EBX o .@-? -B? bE:B <08GG>@? ?-;-
6.@G@..@?+E/-;:EB< W6-.;X o BEB@
%-.;:;:EB@. o V-<0%-.;:;:[email protected]<7<X
m
m
m
m
m
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 54/219
E,( +,)),6>&,J\\ B )+*+,)),6>&,!,M(%(“9)”, “.<$,M(%”).select(“colc1”, “colc3”).where(“colc5 = ?”, “blue”)
k@M<6-/@ I-C>@
p&@.L@. <:?@ /E>89B` .ED <@>@/;:EB
WGE. ?@->:BF D:;0 D:?@ .ED<X
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 55/219
!"#$" "&' !(#$) *&'++ ,- (#**./0 ./ # 12*"34 1#**#/5$#6./(2"6*(+."6*.7'8
:M:6$:d#8cefcfc?gcShNij 5*' *(#$) –9*(#$)61#**#/5$#6./(2"6*(+."6*.7':;<<<
=%(+5.% $52222 22R 22R22 222 22222R R22
2k kR 2 kR 2 lR 22R m2RR222R *22Rk2Q2R2R R2Rk2k E%&)#56 f*n*e
R2R
')#63 -+,(, E%&)#56 S*ef*g /],E, D5$-85$/!L0 ohc;#$ -%&E%& ULQ ],E,e*p*f2?e0!<8% #6 %48&%))#56) $5 O,E% $O%. %E,(:,$%>*!<8% NO%(8 15& .5&% #615&.,$#56*A&%,$#63 -8,&9A56$%4$***A&%,$%> )8,&9 +56$%4$**-8,&9 +56$%4$ ,E,#(,M(% ,) )+*!<8% #6 %48&%))#56) $5 O,E% $O%. %E,(:,$%>*
!<8% NO%(8 15& .5&% #615&.,$#56*
)+,(,W
I0@ +,)),6>&,*#68:$*)8(#$*)#[% 6-.-9@;@. ?@G-8>;< ;E 322K222A I0:< :< ;0@ -66.EJ:9-;B89C@. EG 60M<:/-> .ED< :B - <:BF>@ &6-.7 6-.;:;:EBA ZG ME8 0-L@ .@->>M D:?@ .ED<W;0E8<-B?< EG /E>89B<XK ME8 9-M B@@? ;E >ED@. ;0:< L->8@A I0@ 0:F0@. ;0@ L->8@K ;0@G@D@. &6-.7 ;-<7< -.@ /.@-;@?A ZB/.@-<:BF ;0@ L->8@ ;EE 98/0 9-M >:9:; ;0@ 6-.->>@>:<9level.”
0;;6<5==F:;08CA/E9=?-;-<;-J=<6-.7]/-<<-B?.-]/EBB@/;E.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 56/219
<7)(O BQ&-D
<7)(ORII0''&-,0
IS T)U) 4(
] $6@B &E8./@] Z96>@9@B;@? 9E<;>M :B &/->-] &/->- d \-L- '%Z<] !E@< -8;E9-;:/ ;M6@ /EBL@.<:EB<
0;;6<5==F:;08CA/E9=?-;-<;-J=<6-.7]/-<<-B?.-]/EBB@/;E.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 57/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 58/219
“Simple ;0:BF<<0E8>? C@ <:96>@K
/E96>@J ;0:BF<<0E8>? C@ possible”
] '>-B k-M
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 59/219
!",$5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 60/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 61/219
] +E/->
] &;-B?->EB@ &/0@?8>@.
] ['Q(
] ,@<E<
&;-;:/ %-.;:;:
!MB-9:/ %-.;:;:EB:BF
!"#$%&'(
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 62/219
8&7C-$9B,-
"DCC
.
.! .
.! .
.! .
.!
.
.
.
.
..
!! !!
$&$&$&$&
8C
"D "DCC "DCC CC
D$;,D&%,
DD
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 63/219
I/@1J
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 64/219
$%&' !" ) *+,-.+
421O
/**0 12 :)1O
/**0 13
/**0 12
/**0 13
/**0 14
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
:)1O
6',&(')*
:.(&)>1
E$* 9&53 F 5,H 0'$-B@&53IJ
K:,+.$:+,-IL*&9$*M
K:,+Q''D$;,I“.6()-
K:,+IL:'$-BK,=,92+
E$* :9 F 5,H 0'$-B@&5+,=+I9&53J
S KT7)5T:'$-BU:1,** UU;$:+,- *&
S KT7)5T:'$-BU:27;)+ UU5$;, L.6()-:+Q'
UU;$:+,- *&9$*MNO
lE.7@. ,-/0:B@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 65/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 66/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 67/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 68/219
#
*+,-.+
&&!V< 421O
#
&&!V< 421O
#
V< 421O
#
&&!V< 421O
!" !" !"
*+,-.+
!"!" !" !"
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 69/219
*+,-.+
&&!V< 421O &&!V< 421O V< 421O
#
&&!V< 421O
!"
#
!"
#
!"
#
!"
#
!"
#
!"
#
!"
0/Q!WXYZ!W>!X]D0CQD@>0[ M%,3$2*+[ NP ^ &3 H&-B,- )5:+$59,: +& -25 &5 ,$91 ;$91)5,
0/Q!WXYZ!W>!X@Z!>0[ M%,3$2*+[ Q\\P ^ &3 9&-,: +& $**&H 0'$-B $''*)9$+)&5: +& 2:, &5 +1, ;$91
0/Q!WXYZ!W>!X.>.Z!_[ M%,3$2*+[ CZCQ\ !Q. – N A`P C&+$* ;,;&-6 +& $**&H 0'$-B $''*)9$+)&5: +&/EBG=<6-.7]@BLA<0
0/Q!WX"Q>.ZDX.>.Z!_[ M%,3$2*+[ aNO .`P .,;&-6 +& $**&9$+, +& +1, 0'$-B ;$:+,- $5% H&-B,- %$
&;-B?->EB@ <@;;:BF<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 70/219
&;-B?->EB@ <@;;:BF<
] '66< <8C9:;;@? D:>> .8B :B NZN$ 9E?@ CM ?@G-8>;
)8,&9*+5&%)*.,4N 9-J:989 -9E8B; EG )%P /E.@< ;E .@n8@<; GE. ;0@-66>:/-;:EB G.E9 -/.E<< ;0@ />8<;@.
)8,&9*%4%+:$5&*.%.5&<N ,@9E.M GE. @-/0 @J@/8;E.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 71/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 72/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 73/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 74/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 75/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 76/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 77/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 78/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 79/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 80/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 81/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 82/219
Q@<E8./@,-B-F@.
I*2&', YK 3
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 83/219
(E?@,-B-F@. (E?@,-B-F@.
I0',)2'&(
(E?@,-B-
E77 C)1,&(
1 O S 4
I0',)2'
c c
T T
i
Q@<E8./@,-B-F@.
I*2&', YK
I’m HA via
W00X&&7&(
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 84/219
(E?@,-B-F@. (E?@,-B-F@.
I0',)2'&(
(E?@,-B-
E77 C)1,&( I0',)2'E77 C)1,&(
I0',)2'&( I0',)2'
I*2&', YF<-.&>D*&(
E771 C)1,&(
Q@<E8./@,-B-F@.
I*2&', YK
!.:L@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 85/219
(E?@,-B-F@. (E?@,-B-F@.
I0',)2'&(
(E?@,-B-
E77 C)1,&(
BQ&-D,0(
/** :
I0',)2'
BQ&-D
/** :
Q@<E8./@,-B-F@.
I*2&', YK
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 86/219
(E?@,-B-F@. (E?@,-B-F@.
I0',)2'&(
(E?@,-B-
E77 C)1,&(
BQ&-D,0(
/** :
I0',)2'
BQ&-D
/** :
4(2U&(I0',)2'&(
BQ&-D,0(
/** :
] !E@< BE; <
['Q( <@;;:BF<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 87/219
cc6:.c%4%+:$5&)N /EB;.E>< 0ED 9-BM @J@/8;E.< D:>> C@ ->>E/-;@?
cc%4%+:$5&c.%.5&<N Q', GE. @-/0 @J@/8;E.
cc%4%+:$5&c+5&%)N )%P /E.@< GE. @-/0 @J@/8;E.
)8,&9*><6,.#+_((5+,$#56*%6,M(%>
)8,&9*><6,.#+_((5+,$#56*.#6F4%+:$5&))8,&9*><6,.#+_((5+,$#56*.,4F4%+:$5&))8,&9*><6,.#+_((5+,$#56*):)$,#6%>-+O%>:(%&;,+9(53!#.%5:$ /r0)8,&9*><6,.#+_((5+,$#56*)+O%>:(%&;,+9(53!#.%5:$ /L0)8,&9*><6,.#+_((5+,$#56*%4%+:$5&b>(%!#.%5:$ /a0
!MB-9:/ '>>E/-;:EB5
0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=C>EC=9-<;@.=/E.@=<./=9-:B=</->-=E.F=-6-/0@=<6-.7="J@/8;E.'>>E/-;:EB,[email protected]<
YARN resource manager UI: O$$8NRR^#8 ,>>&%))WNtftt
W(E -66< .8BB:BFX
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 88/219
W(E -66< .8BB:BFX
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 89/219
M,9OU2:,-b)'UNcUcUdOUef gPh :'$-BU:27;)+ UU9*$::
&-?K$'$91,K:'$-BK,=$;'*,:K0'$-B/) UU%,'*&6U;&%, 9*),5+ UU;$:+
T&'+T9*&2%,-$T'$-9,*:T@"#UaKOKNUNK9%1aKOKNK'cKNOTV$-:T:'$-BU
,=$;'*,:UNKNKcU9%1aKOKNU1$%&&'OKaKcU9%1aKOKNKV$- Nc
'66 .8BB:BF :B />:@B;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 90/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 91/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 92/219
M,9OU2:,-b)'UNcUcUdOUef gPh :'$-BU:27;)+ UU9*$::
&-?K$'$91,K:'$-BK,=$;'*,:K0'$-B/) UU%,'*&6U;&%, 9*2:+,- UU;$:
6$-5 T&'+T9*&2%,-$T'$-9,*:T@"#UaKOKNUNK9%1aKOKNK'cKNOTV$-:T:'
,=$;'*,:UNKNKcU9%1aKOKNU1$%&&'OKaKcU9%1aKOKNKV$- Nc
'66 .8BB:BF :B />8<;@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 93/219
'66 .8BB:BF :B />8<;@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 94/219
'66 .8BB:BF :B />8<;@. 9E?@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 95/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 96/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 97/219
<7)(O I&',()* C)1,&( Z.0 1,)(,1 BQ&-D,0(1[ :)1O1 (D' 2'
80-)* qBEB@r V89-B C@:BF "J@/8;E.
<,)'>)*0'& &;-B?->EB@ ,-<;@. lE.7@. \#, "J@/8;E.
\E]^ ['Q( '66 ,-<;@. (E?@ ,-B-F@. "J@/8;E.
C&101 ,@<E< ,-<;@. ,@<E< &>-L@ "J@/8;E.
)8,&9c):M.#$ 6.EL:?@< - 8B:GE.9 :B;@.G-/@ GE.<8C9:;;:BF bEC< -/.E<< ->> />8<;@. 9-B-F@.<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 98/219
M#6R)8,&9c):M.#$ cc.,)$%& )8,&9NRRO5)$Npfppcc%4%+:$5&c.%.5&< ef3
.<2)+$*8<
&E8
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 99/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 100/219
!"/**0 12
/**0 13
/**0 12
: :
: :: :
6',&(')*:.(&)>1
Recommended to use at most only 75% of a macGE. &6-.7
,:B:989 "J@/8;E. 0@-6 <:j@ <0E8>? C@ i YH
Max Executor heap size depends… maybe 40 YH
,@9E.M 8<-F@ :< F.@-;>M -GG@/;@? CM <;E.-F@ >@L@<@.:->:j-;:EB GE.9-;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 101/219
_#<A
1$8$#'%E'_$1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 102/219
J\\*+,+O%/0 BB J\\*8%&)#)$/LFLKJu2KrGu0
]^?
1$8$#'%E'_$1
most CPU-efficient option
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 103/219
8$#'%E'_$1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 104/219
J\\*8%&)#)$/LFLKJu2KrGu2-FJ0
]^?
8$#'%E'_$1
_
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 105/219
*8%&)#)$/LFLKJu2_r\2\b-a0
]^?
1$8$#'%E'_$1
V< 42
_
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 106/219
*8%&)#)$/LFLKJu2_r\2\b-a2-FJ0
]^?
8$#'%E'_$1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 107/219
*8%&)#)$/\b-a2KrGu0
]^?
1$8$#'%E'_$1
1$8$#'%E'_$1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 108/219
J\\*8%&)#)$/LFLKJu2KrGu2S0
]^? 2( `21$ / ]^? 2( `
_
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 109/219
*8%&)#)$/LFLKJu2_r\2\b-a2S0
]^?
1$8$#'%E'_$1 1$8$#'%E'_$1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 110/219
*8%&)#)$/KII2DF_Z0
]^?<F W 7--<F
8$#'%E'_$1
B%C.D2(
]^?<O W 7--<F
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 111/219
*:68%&)#)$/0
]^?
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 112/219
]^?
?
b
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 113/219
] ZG Q!! G:;< :B [email protected] /0EE<@ ,",$Q[f$(+[
] ZG BE;K 8<@ ,",$Q[f$(+[f&"Q D= G-<; <@.:->:j-;:EB >:C.-.M
] Don’t spill to disk unless functions that computed the datase-.@ [email protected] @J6@B<:L@ E. ;0@M G:>;@. - >-.F@ -9E8B; EG ?-;-AW.@/E968;:BF 9-M C@ -< G-<; -< .@-?:BF G.E9 ?:<7X
] P<@ .@6>:/-;@? <;E.-F@ >@L@>< <6-.:BF>M -B? EB>M :G ME8 D-B; G
G-8>; .@/[email protected] W9-MC@ ;E <@.L@ .@n8@<;< G.E9 - D@C -66X
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 114/219
ZB;@.9@?:-;@ ?-;- :< -8;E9-;:/->>M 6@.<:<;@? ?8.:BF <08GG>@ [email protected];:EB<
o
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 115/219
%M&6-.75 <;E.@? ECb@/;< D:>> ->D-M< C@ <@.:->:j@? D:;0 %:/7>@ >:C.-.MK <E :; ?BE; 9-;;@. D0@;0@. ME8 /0EE<@ - <@.:->:j@? >@L@>A
!@G-8>; ,@9E.M '>>E/-;:EB :B "J@/8;E. \#,
@ /)8,&9*)$5&,3%*.%.5&
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 116/219
MG#FG#
FG#
I)-.&> ]441
@1&( /(03()51!6'1B#-,'6A
<.DAA*& 5&50(?
8 3
Ibv !Db-
Q!! &;E.-F@5 D0@B ME8 /->> A6@.<:<;WX E. A/-/0@WXA &6-.7 D:>> >:9:; ;0@ -9E8B; EGmemory used when caching to a certain fraction of the JVM’s overall heap se
&6-.7 8<@< [email protected] GE.5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 117/219
memory used when caching to a certain fraction of the JVM s overall heap, se)8,&9*)$5&,3%*.%.5&<I&,+$#56
&08GG>@ -B? -FF.@F-;:EB C8GG@.<5 l0@B [email protected]:BF <08GG>@ [email protected];:EB<K &6-.7 D:>>/.@-;@ :B;@.9@?:-;@ C8GG@.< GE. <;E.:BF <08GG>@ E8;68; ?-;-A I0@<@ C8GG@.< -.@ 8<@<;E.@ :B;@.9@?:-;@ .@<8>;< EG -FF.@F-;:EB< :B -??:;:EB ;E C8GG@.:BF ?-;- ;0-; :< F;E C@ ?:.@/;>M E8;68; -< 6-.; EG ;0@ <08GG>@A
P<@. /E?@5 &6-.7 @J@/8;@< -.C:;.-.M 8<@. /E?@K <E 8<@. G8B/;:EB< /-B ;0@9<@>L@.@n8:.@ <8C<;-B;:-> [email protected] NE. :B<;-B/@K :G - 8<@. -66>:/-;:EB ->>E/-;@< >-.F@ -E. E;0@. ECb@/;<K ;0@<@ D:>> /EB;@B; GE. [email protected]>> [email protected] 8<-F@A P<@. /E?@ 0-< -/to everything “left” in the JVM heap after the space for Q!! <;E.-F@ -B? <08GG><;E.-F@ -.@ ->>E/-;@?A
3A ).@-;@ -B Q!!
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 118/219
1A %8; :; :B;E /-/0@
OA +EE7 -; 0'$-B@&5+,=+ >EF<EB ;0@ ?.:L@. 6.EF.-9 E.&6-.7 PZ
]D(Z `*&9B.$5$?,-.$:+,-Q9+&-[ Q%%,% -%%XcXN )5 ;,;&-6 &5 ;7BK*&9$*[aceNN I:)i,[ dNdKa W`R 3-
E2)8 V'EE "$EE D25 .2V &5C. &$&2#D $%C.-%#"'"'2( '8 C2(85&'()T V.'C. D25 C%(%))#$)%"$ "2 )$" ".$ "2"%E 8'_$ 2X ".$ @99
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 119/219
&@.:->:j-;:EB :< 8<@? D0@B5
I.-B<G@..:BF ?-;- EL@. ;0@ B@;DE.7
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 120/219
&6:>>:BF ?-;- ;E ?:<7
)-/0:BF ;E [email protected] <@.:->:j@?
H.E-?/-<;:BF L-.:-C>@<
\-L- <@.:->:j-;:EB k.ME <@.:->:j-;:EBL<A
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 121/219
• P<@< Java’s Z7V,9+Z2+'2+0+-,$; [email protected]
• lE.7< D:;0 -BM />-<< ME8 /.@-;@ ;0-; :96>@9@B;<V$E$K)&K0,-)$*)i$7*,
• [E8 /-B /EB;.E> ;0@ [email protected]/@ EG <@.:->:j-;:EB9E.@ />E<@>M CM @J;@B?:BF V$E$K)&K>=+,-5$*)i$7*,
• N>@J:C>@K C8; n8:;@ <>ED
• +@-?< ;E >-.F@ <@.:->:j@? GE.9-;< GE. 9-BM />-<<@<
• Q@/E99@B?@? <@.:->:j-;:EB GE. 6.E?8/
• P<@ kM.E L@.<:EB 1 GE. <6@@?M <@.:->:j-;:E9E.@ /E96-/;B@<<
• !E@< BE; <866E.; ->> 0,-)$*)i$7*, ;M6@
• Q@n8:.@< ME8 ;E 6'2#*.'6 the classes you-?L-B/@
• ZG <@;K D:>> C@ 8<@? GE. <@.:->:j:BF <08GG>@ ?BE?@< -B? -><E <@.:->:j:BF Q!!< ;E ?:<7
C2(XI8$"(“8-%#cI8$#'%E'_$# ”, d2#)I%-%C.$I8-%#cI8$#'%
IE .@F:<;@. ME8. EDB /8<;E9 />-<<@< D:;0 k.ME 8<@ ;0@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 122/219
IE .@F:<;@. ME8. EDB /8<;E9 />-<<@< D:;0 k.MEK 8<@ ;0@-,?):+,-W-6&@*$::,: 9@;0E?5
E$* 9&53 F 5,H 0'$-B@&53IJK:,+.$:+,-IKKKJK:,+Q''D$;,IKKKJ
9&53K-,?):+,-W-6&@*$::,:I0,jI9*$::Z3M.6@*$::NPR 9*$::Z3M.6@*$::OP
E$* :9 F 5,H 0'$-B@&5+,=+I9&53J
] ZG ME8. ECb@/;< -.@ >-.F@K ME8 9-M B@@? ;E :B/.@-<@ :'$-BKB-6&:,-)$*)i,-K7233,-K;7 /EBG:F 6.E6@.;M
] I0@ ?@G-8>; :< 1K C8; ;0:< L->8@ B@@?< ;E C@ >-.F@ @BE8F0 ;E0E>? ;0@ $B62'*. ECb@/; ME8 D:>> <@.:->:j@A
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 123/219
A A A !"!""
#
!""
!"!""
#
!""
V:F0 /08.B +ED /08.B
)E<; EG Y) :< 6.E6E.;:EB-> ;E ;0@ s\-L- ECb@/;<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 124/219
A A A !"!""
#
!""
!""
V:F0 /08.B
\-L- ECb@/;<
W<E 8<@ -B -..-M EG ]5+: :B<;@-? E\)5B,%\):+X
UE,-7&:,[?9 Ukk[l/-)5+A@",+$)*: Ukk[l/-)
IE 9@-<8.@ Y) :96-/;5
%-.->>@> $>? Y) ),& Y) Y%-.->>@> Y)
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 125/219
<//;fg8$>%#%EE$EhE1,= <//;fg8$=2(C?%#c!V$$-,= <//;fg
] P<@< 98>;:6>@ ;0.@-?<;E ?E CE;0 ME8BF F@B-B? E>? F@B Y)
] '><E - 98>;:;0.@-?:BF/E96-/;:BF /E>>@/;E.
] VE;&6E; ?E@<
/E96-/;:EB EB>M :BE>? F@B
<//;fg8$>%#%EE$E,=
] P<@< 98>;:6>@ ;0.@-?< ;E?E ME8BF F@B Y)
] l:>> ?@G-8>; ;E &@.:-> EB<:BF>@ /E.@ 9-/0:B@<
] Aka “throughputcollector”
] YEE? GE. D0@B - >E; EGDE.7 :< B@@?@? -B?>EBF 6-8<@< -.@-//@6;-C>@
] P<@ /-<@<5 C-;/06.E/@<<:BF
<//;>%#%EE$E,=B.#$%18[ijk <//;>%#%EE$E=?!B.#$%18[ijk
] )EB/8..@B; ,-.7&D@@6 -7-“Concurrent lowpause collector”
] I.:@< ;E 9:B:9:j@6-8<@< ?8@ ;E Y) CM?E:BF 9E<; EG ;0@ DE.7
/EB/8..@B;>M D:;0-66>:/-;:EB ;0.@-?<
] P<@< <-9@ ->FE.:;09EB ME8BF F@B -<6-.->>@> /E>>@/;E.
] P<@ /-<@<5
] Y-.<;-.
] !@<.@6
] Z< - -B?
/E9Y)
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 126/219
m
&;-F@ 3
IC2EE$C"Y ZI-<7 s3I-<7 s1I-<7 sO
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 127/219
&;-F@ 1
&;-F@ O
&;-F@ 4
AA
\EC s3
AA
&;-F@ S
Q!! $Cb@/;< !'Y &/0@?8>@. I-<7 &/0@?8>@.
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 128/219
I-<7&/0@?8>@.
H
!%%NKV&)5I-%%OJ
K ?-&2'`6I…J
K3)*+,-I…J
] H8:>? [email protected];E. !'Y
] &6>:; F.-60 :B;E
*.B2'* EG ;-<7<
] &8C9:; @-/0 <;-F@ -<.@-?M
]
]
!'Y I-<7&@; I-<7
E3'01,2- ,007&(),0(1
Doesn’t know
)90D, 1,)3&1&;-F@G-:>@?
] +-8B/0@<
:B?:L:?8-> ;-<7<
] Q@;.M G-:>@? E.<;.-FF>:BF ;-<7<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 129/219
“One EG ;0@ /0->>@BF@< :B 6.EL:?:BF Q!!< -< -B -C<;.-/;:EB :</0EE<:BF - .@6.@<@B;-;:EB GE. ;0@9 ;0-; /-B ;.-/7 >:B@-F@ -/.E<< -D:?@ .-BF@ EG ;.-B<GE.9-;:EB<.”
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 130/219
“The 9E<; :B;@.@<;:BF n8@<;:EB :B ?@<:FB:BF ;0:< :B;@.G-/@ :< 0ED ;E.@6.@<@B; ?@6@B?@B/:@< C@;D@@B Q!!<.”
“We GE8B? :; CE;0 <8GG:/:@B; -B? 8<@G8> ;E />-<<:GM ?@6@B?@B/:@<:B;E ;DE ;M6@<5• B-..ED ?@6@B?@B/:@<K D0@.@ @-/0 6-.;:;:EB EG ;0@ 6-.@B; Q!!
:< 8<@? CM -; 9E<; EB@ 6-.;:;:EB EG ;0@ /0:>? Q!!• D:?@ ?@6@B?@B/:@<K D0@.@ 98>;:6>@ /0:>? 6-.;:;:EB< 9-M
?@6@B? EB :;.”
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 131/219
"J-96>@< EG B-..ED -B? D:?@ ?@6@B?@B/:@<A
"-/0 CEJ :< -B Q!!K D:;0 6-.;:;:EB< <0EDB -< <0-?@? .@/;-BF>@<A
;< 5<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 132/219
! #$#%&' ($)*+*+,-
! .//
0,+-
1+2*&)
3),4(56
7*$3& 8
7*$3& 9
7*$3& :
=< /< ><
?<
@$(
! 2,A* ($)*+*+,-
“This ?:<;:B/;:EB :< 8<@G8> GE. ;DE .@-<EB<5
!@6@B?@B/:@<5 (-..ED L< l:?@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 133/219
3X (-..ED ?@6@B?@B/:@< ->>ED GE. 6:6@>:B@? @J@/8;:EB EB EB@ />8<;@. BE?@K
D0:/0 /-B /E968;@ ->> ;0@ 6-.@B; 6-.;:;:EB<A NE. @J-96>@K EB@ /-B -66>M -9-6 GE>>ED@? CM - G:>;@. EB -B @>@9@B;]CM]@>@9@B; C-<:<A
ZB /EB;.-<;K D:?@ ?@6@B?@B/:@< .@n8:.@ ?-;- G.E9 ->> 6-.@B; 6-.;:;:EB< ;E C@-L-:>-C>@ -B? ;E C@ <08GG>@? -/.E<< ;0@ BE?@< 8<:BF - ,-6Q@?8/@]>:7@[email protected];:EBA
1X Q@/[email protected] -G;@. - BE?@ G-:>8.@ :< 9E.@ @GG:/:@B; D:;0 - B-..ED ?@6@B?@B/MK -
EB>M ;0@ >E<; 6-.@B; 6-.;:;:EB< B@@? ;E C@ .@/E968;@?K -B? ;0@M /-B C@.@/E968;@? :B 6-.->>@> EB ?:GG@.@B; BE?@<A ZB /EB;.-<;K :B - >:B@-F@ F.-60 D:;0D:?@ ?@6@B?@B/:@<K - <:BF>@ G-:>@? BE?@ 9:F0; /-8<@ ;0@ >E<< EG <E9@ 6-.;:;:EBG.E9 ->> ;0@ -B/@<;E.< EG -B Q!!K .@n8:.:BF - /E96>@;@ .@]@J@/8;:EB.”
IE ?:<6>-M ;0@ >:B@-F@ EG -B Q!!K &6-.7 6.EL:?@< - $5\%M:3-$? 9@;0E?5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 134/219
)+,(,W #68:$*$5\%M:3-$?
&%)t?N -$? B/S0 >,$,*$%4$ L,88%>J\\qSnSs ,$ $%4$I#(% ,$ ^+5w >,$,*$%4$ D,>558J\\qSnes ,$ $%4$I#(% ,$ ^+56
)+,(,W +5:6$)*$5\%M:3-$?&%)thN -$? B/S0 -O:11(%>J\\qSnos ,$ &%>:+%;<a%< ,$ ^+56)5(%
xc/S0 L,88%>J\\qSn?s ,$ .,8 ,$ ^+56)5(%WNepw I#($%&%>J\\qSnhs ,$ 1#($%& ,$ ^+56)5(%WNew L,88%>J\\qSngs ,$ .,8 ,$ ^+56)5(%WNe?w >,$,*$%4$ L,88%>J\\qSnSs ,$ $%4$I#(% ,$ ^w >,$,*$%4$ D,>558J\\qSnes ,$ $%4$I#(% ,$ ^
VED ?E ME8 7BED :G - <08GG>@ D:>> C@ /->>@? EB - I.-
.@6-.;:;:EB bE:B /EF.E86 -B? -BM EG ;0@ mHM E. mHMk@M ;.-B<GE
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 135/219
D0.' .+B. 6'(B6.#.#0- E)*. /B$$* /0B$'*' 3F G6)'H
] .@6-.;:;:EB K bE:BK /EF.E86K -B? -BM EG ;0@ mHM E. mHMk@M ;.-B<GE/-B .@<8>; :B <08GG>@<
] ZG ME8 ?@/>-.@ - B89%-.;:;:EB< parameter, it’ll probably shuffle
] ZG - ;.-B<GE.9-;:EB /EB<;.8/;< - <08GG>@?Q!!, it’ll probably shuf
] /E9C:B@HMk@M /->>< - <08GG>@ W<E ?E E;0@. ;.-B<GE.9-;:EB< >:[email protected]@MK D0:/0 -/;8->>M @B? 86 /->>:BF /E9C:B@HMk@MX
%,3 -,'$-+)+)&5I52;/$-+)+)&5:[ ]5+JI);'*)9)+
&-%[ Z-%,-)5?MCP F 52**J[ !""MCP F n
9&$*,:9,I52;/$-+)+)&5:R :1233*, F +-2,J
o
Q!!A</->-
VED ?E ME8 7BED :G - <08GG>@ D:>> C@ /->>@? EB - I.-
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 136/219
Transformations that use “B89%-.;:;:EB<” like distinct will probably shuffle:
%,3 %):+)59+I52;/$-+)+)&5:[ ]5+JI);'*)9)+ &-%[ Z-%,-)5?MCP F
52**J[ !""MCP F
;$'I= FS I=R 52**JJK-,%29,`6W,6II=R 6J FS =R
52;/$-+)+)&5:JK;$'IXKXNJ
] 'B @J;.- 6-.-9@;@. ME8 /-B 6-<< - 7=L ;.-B<GE.9-;:EB ;E >@; &6-.7 ;0-; ME8 D:>> BE; C@ 9@<<:BF D:;0 ;0@ 7@M< -; ->>
] '>> [email protected];:EB< ;0-; <08GG>@ ?-;- EL@. B@;DE.7 D:>> C@B@G:; G.E9 6-.;:;:
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 137/219
6 6
] [email protected];:EB< ;0-; C@B@G:; G.E9 6-.;:;:EB:BF59&?-&2'R ?-&2'Y)+1R V&)5R *,3+Z2+,-8&)5R -)?1+Z2+,-8&)5R ?-&2'`6W,6R
-,%29,`6W,6R 9&;7)5,`6W,6R *&&B2'R K K K
0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=C>EC=9-<;@.=/E.@=<./=9-:B=</->-=E.F=-6-/0@=<6-.7=.??=Q!!A</->-s+O21
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 138/219
+:
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 139/219
VED 9-BM &;-F@< D:>> ;0:< /E?@ .@n8:.@e
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 140/219
:9K+,=+()*,IL:&;,()*,K+=+LJK
;$'I;$'(259JK
3*$+.$'I3*$+.$'(259JK
3)*+,-I3)*+,-(259JK
9&25+IJ
VED 9-BM &;-F@< D:>> ;0:< !'Y .@n8:.@e
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 141/219
VED 9-BM &;-F@< D:>> ;0:< !'Y .@n8:.@e
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 142/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 143/219
dd d
!"
:
J o 4
:
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 144/219
!"
!"
J o 4
:
J o 4 J o 4
J o 4
:
• H.E-?/-<; L-.:-C>@< – &@B? - >-.F@ .@-?]EB>M >EE786 ;-C>@ ;E ->> ;0<@B? - >-.F@ G@-;8.@ L@/;E. :B - ,+ ->FE.:;09 ;E ->> BE?@<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 145/219
• '//898>-;E.< – /E8B; @L@B;< ;0-; E//8. ?8.:BF bEC @J@/8;:EB GE. ?68.6E<@<A "J-96>@5 VED 9-BM >:B@< EG ;0@ :B68; G:>@ D@.@ C>-B7e $/E..86; .@/E.?< D@.@ :B ;0@ :B68; ?-;-<@;e
dd d
<7)(O 1D770(,1 F ,?7&1 0A 1.)(&> U)(2)9*&1J
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 146/219
•
H.E-?/-<; L-.:-C>@< – ->>ED< ME8. 6.EF.-9 ;E @GG:/:@B;>M <@B? - >-.FL->8@ ;E ->> ;0@ DE.7@. BE?@< GE. 8<@ :B EB@ E. 9E.@ &6-.7 [email protected];:E<@B?:BF - >-.F@K .@-?]EB>M >EE786 ;-C>@ ;E ->> ;0@ BE?@<A
• '//898>-;E.< – ->>ED< ME8 ;E -FF.@F-;@ L->8@< G.E9 DE.7@. BE?@;0@ ?.:L@. 6.EF.-9A )-B C@ 8<@? ;E /E8B; ;0@ s EG @..E.< <@@B :B ->:B@< <6.@-? -/.E<< 322< EG BE?@<A $B>M ;0@ ?.:L@. /-B -//@<< ;0@ L
-//898>-;E.K ;-<7< /-BBE;A NE. ;-<7<K -//898>-;E.< -.@ D.:;@]EB>MA
dd d
H.E-?/-<; L-.:-C>@< >@; 6.EF.-99@. 7@@6 - .@-?]EB>M L-.:-C>@ /-/0@? EB @-/0 9-/0:B@ .-;0@. ;0-B<0:66:BF - /E6M EG :; D:;0 ;-<7<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 147/219
<0:66:BF - /E6M EG :; D:;0 ;-<7<
NE. @J-96>@K ;E F:L@ @[email protected] BE?@ - /E6M EG - >-.F@:B68; ?-;-<@; @GG:/:@B;>M
&6-.7 -><E -;;@96;< ;E ?:<;.:C8;@ C.E-?/-<; L-.:-C>@<8<:BF @GG:/:@B; C.E-?/-<; ->FE.:;09< ;E .@?8/@/E998B:/-;:EB /E<;
E,( M&5,>+,)$U,& B )+*M&5,>+,)$/_&&,</eQ SQ g00
&/->-5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 148/219
M&5,>+,)$U,&*E,(:%
M&5,>+,)$U,& B )+*M&5,>+,)$/(#)$/&,63%/eQ h000
M&5,>+,)$U,&*E,(:%
%M;0EB5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 149/219
+:B7
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 150/219
!"
!"#$%&'(
P<@< VII%
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 151/219
!"
!"
12 ,H G:>@
!"
P<@< C:;;E..@B;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 152/219
!"
!"
12 ,H G:>@
A A A! #$ ! #$ ! #$ ! #$
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 153/219
!"!" !"
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 154/219
!"!"
!" !"!"
A l t i bl th t l b “ dd d” t th h
dd d
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 155/219
Accumulators are variables that can only be “added” to through
-B B**0/#B.#I' [email protected];:EB
P<@? ;E :96>@9@B; /E8B;@.< -B? <89<K @GG:/:@B;>M :B 6-.->>@>
&6-.7 B-;:L@>M <866E.;< -//898>-;E.< EG B89@.:/ L->8@ ;M6@< -B?<;-B?-.? 98;-C>@ /E>>@/;:EB<K -B? 6.EF.-99@.< /-B @J;@B?GE. B@D ;M6@<
Only the driver program can read an accumulator’s value, not the;-<7<
!"# !""#$ $ %"%"&&'('#")*+,&-
%"%'!(!))*)+,*,.++"/,-0 .0 /0 0--%12(*!"3,4 $1 !""#$ 2$
&/->-5dd d
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 156/219
!""#$%!"#'3
,++:. : )+6#11242+#"3$=f>&>> : )+68,&,((%(#[%/qeQ SQ gQ hs05'? ?/40N
0+3,#+ ,++:.,++:. @: 4
&>>615&%,+O/10
,++:.6A#+2'
%M;0EB5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 157/219
%M&6-.7 -; - Y>-B/@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 158/219
l.:;@ &6-.7 bEC<:B %M;0EB
Q8B :B;@.-/;:L@ bEC< :B ;0@ <0@>>
&866@J;@
%M&6-.7
52 @,A.=
802BB ACD
604BB DCEE.F>=
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 159/219
&6-.7 )E.@ "BF:B@W&/->-X
&;-B?->EB@ &/0@?8>@. ['Q( ,@<E+E/->
\-L- '%Z
%M&6-.7
9:;+<
GCF>.">
GCF>+CAA.+
9:;+<
GCF>.">
1H5I
9CD<.>
1,:.]44
,+>:CK &_+K <08GG>@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 160/219
+E/-> !:<7
!.:L@. \#,
"J@/8;E. \#,
"J@/8;E. \#,
1,:.
Z0(O&( C)-.4(2U&( C)-.2'&
(I=J
]44
,+>:CK &_+K <08GG>@
!-;- :< <;E.@? -< %:/7>@? ECb@/;< :B -B Q!!q'..-MqHM;@`)>007]44
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 161/219
C)77&>]44
/?,.0']44 Q!!q'..-Mq r rK K K
W322 kH – 3,H @-/0 6:/7@? ECb@/;X
)%M;0EBW?@G-8>; 6M;0EBX
)0EE<@ [E8. %M;0EB Z96>@9@B;-;:EB
9:;+<
GCF>.">
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 162/219
6M6M• \ZIK <E G-<;@. • >@<< [email protected]• )NNZ <866E.;
GCF>.">
4(2U&( C)-.2'&Z0(O&( C)
h /_0/Q!WX"!]<>!X/_C#ZDF'6'6 /_0/Q!WX/_C#ZDF'6'6 KT7
h /_0/Q!WX"!]<>!X/_C#ZDF'6'6 /_0/Q!WX/_C#ZDF'6'6 KT7)5T:'$-BU:27;)+
$Q
T09 I/?,.0' FN! /?/? FNaNK <7&&> D7
lE.? )E8B; S3 < 34 < 1AT J
&E.; Sc < SS < 3A24 J
&; ; 3TS O c Si
I0@ [email protected]/@ <6@@? 86 D:>> ?@6@B? EB DE.7 >E-? WG.E9 12t ;E O2
V@.@ -.@ <E9@ C@B/09-.7<5
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 163/219
&;-;< 3TS < OAc < Si J
V@.@ :< ;0@ /E?@ 8<@? GE. C@B/09-.75
-%% F :9K+,=+()*,IL+,=+LJ
%,3 H&-%9&25+IJ[
-%%K3*$+.$'I*$;7%$ =[=K:'*)+IpTpJJq
K;$'I*$;7%$ =[I=RNJJK-,%29,`6W,6I*$;7%$ =R6[=l6JK9&**,9+Q:.
%,3 :&-+IJ[-%%K:&-+`6I*$;7%$ =[=R NJK9&25+IJ
%,3 :+$+:IJ[
:9K'$-$**,*)i,I-$5?,INcOrJR OcJK3*$+.$'I*$;7%$ =[ =-$5?,IacOrJJ
0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=68>>=13SS
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 164/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 165/219
&6-.7 <E.;@? ;0@ <-9@ ?-;- ab A)1,&(8<:BF KGb A&c&
;0-B
322IH !-M;EB- &E.; )E96@;:;:EB 123S
'>> ;0@ <E.;:BF ;EE7 6>-/@ EB ?:<7 WV!
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 166/219
lE.7 CM !-;-C.:/7< @BF:B@@.<5 Q@MBE>? u:BK %-.L:j !@M0:9K u:-BF.8: ,@BFK '>: Y0E?<:K ,-;@
,E.@ :BGE5
0;;65==<E.;C@B/09-.7AE.F
0;;65==?-;-C.:/7<A/E9=C>EF=123S=33EGG:/:->>M]<@;<]-]B@D].@/E.?]:B]>-.F@]<
using Spark’s in][email protected] /-/0@h
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 167/219
] &;.@<<@< “shuffle” which underpins everything from SQL to ,>>:C
] &E.;:BF :< /0->>@BF:BF C=/ ;0@.@ :< BE .@?8/;:EB :B ?-;-
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 168/219
] &E.; 322 IH o 422 IH ?:<7 Z=$ -B? 122 IH B@;DE.7
B'32'&&(2'3 6'U&1,5&', 2' <7)(OJ
] &E.;]C-<@? <08GG>@ W&%'Qk]12S4X] (@;;M B-;:L@ B@;DE.7 ;.-B<6E.; W&%'Qk]1SciX] "J;@.B-> <08GG>@ <@.L:/@ W&%'Qk]OTgcX
I*&U&( E77*2-),20' *&U&* :&-.'2dD&1J
] Y) -B? /-/0@ G.:@B?>M [email protected] >-ME8;] %:6@>:B:BF
!"!""
#
!""
I
I
] ZB;@> u@EB )%P "4 1cT2 a 1A4 YVj D= O1 /E.@<] 1SS YH EG Q',] i J i22 YH &&! -B? Q'Z! 2 <@;86 GE.9-;;@? D:;0 =@J;S] UgA4 YC6< W?J? 9K(*X C-B?D:?;0 C@;D@@B 1 .-B?E9 BE?@<
] "-/0 .@/E.?5 322 CM;@< W32 CM;@ 7@M ` g2 CM
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 169/219
")15 :1AiJ>-.F@
W12c DE.7@.<X
] $6@B\!k 3AT
] V!N& 1ASA3 D= <0E.; /:./8:; >E/-> .@-?< @B-C>@
] '6-/0@ &6-.7 3A1A2
] &6@/8>-;:L@ "J@/8;:EB EGG
] ZB/.@-<@? +E/->:;M l-:; ;E :BG:B:;@
] )E96.@<<:EB ;8.B@? EGG GE. :B68;K E8;68; ` B
] P<@? PB<-G@ ;E 68; ->> ;0@ ?-;- EGG]0@-6 -B:; 9-B8->>M W:A@A B@L@. ;.:FF@.@? ;0@ Y)X
] O1 <>E;< 6@. 9-/0:B@] cK4g1 <>E;< ;E;->
?-&2'`6W,6
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 170/219
o :&-+`6W,6
-,%29,`6W,6
:'$-BK:1233*,K:')**F3$*:,
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 171/219
W'GG@/;< .@?8/@. <:?@ -B? 7@@6< ->> ;0@ ?-;- :B [email protected]
] ,8<; ;8.B ;0:< EB GE. ?MB-9:/
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 172/219
] lE.7@. \#, <@.L@< G:>@<] (E?@ ,-B-F@. <@.L@< G:>@<
] l-< <>ED C@/-8<@ :; 0-? ;E /E6M ;0@ ?-;- O ;:9@<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 173/219
,-6 E8;68; G:>@EB >E/-> ?:.
+:[email protected]@>C8GG@.
!"
(Z)C8GG@
] P<@< - ;@/0B:n8@ /->>@? [email protected]]/E6M
] Z< - 9-6]<:?@ E6;:9:j-;:EB ;E <@.L@ ?-;- [email protected]:/7>M ;E .@n8@<;:BF .@?8/@.<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 174/219
,-6 E8;68; G:>@EB >E/-> ?:.
(Z)C8GG@.
&;:JK &;:JK &;:JK &;:JK
] "B;:CM Z=$V!N&
w 32K222 .
] (E;:/@ ;0-; 9-60-< ;E 7@@6 O G:>@0-B?>@< E6@B
[ L :E2Cc8
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 175/219
/.LMD.JK /.LMD.JK /.LMD.JK
>E/-
] ,E<
0-B?>@< E6@B
I:9&E.;
&;:JK &;:JK &;:JK &;:JK
W1iK222 8B:n8@ C>E/7<X
QN o 1
1
o OAc YH
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 176/219
] $B>M EB@ G:>@ 0-B?>@E6@B -; - ;:9@
&;:JK &;:JK &;:JK &;:JK
] 4 D-] 4 D-
1
W1iK222 8B:n8@ C>E/7<X
QN o 1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 177/219
/.LMD.JK /.LMD.JK /.LMD.JK
QN o 1
C&(3&<0
I:9&E.;
]
]
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 178/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 179/219
@1&(64 ^)5& E3& 80-),20' /&,
1iSg1gS1 \E0B Y->; O1 (@D [E.7 &@- VE.<@
g4i1gO1S l:B<;EB &9:;0 S3 $/@-B:- 'B;
g1iT3Tc3 IE9 &-DM@. 3T ,:<<:<<:66: Q-//EEB
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 180/219
OT4iSgO1 )-.>E< V:BEbE<- OO $.>-B?E )-;
TOcSi1TS +8:< QE?.:F8@j OS $.>-B?E !EF<
\!H)=$!H) [E8. '66
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 181/219
091,;$!""] Q!! EG QED ECb@/;<K @-/0 .@6.@<@B;:BF - .@/E.?
] QED ECb@/;< o ;M6@ d /E>A B-9@ EG @-/0
] &;E.@< ?-;- [email protected] @GG:/:@B;>M CM ;-7:BF -?L-B;-F@ EG ;0@ </0@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 182/219
] &/0@9-Q!!< -.@ -><E .@F8>-. Q!!<K <E ME8 /-B .8B;.-B<GE.9-;:EB< >:7@ 9-6WX E. G:>;@.WX
] '>>ED< B@D [email protected];:EB<K >:7@ .8BB:BF &_+ EB ECb@/;<
0;;6<5==?-;-C.:/7<A/E9=C>E
F=1234=21=3T=:B;.E?8/:BF]?-;-G.-9@<]:B]<6-.7]GE.]>-.F@]</->@]?-;-]</:@B
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 183/219
l : h
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 184/219
l-.B:BFh
$B>M >EE7< -; G:.<; .ED
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 185/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 186/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 187/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 188/219
!"#$$%&'$#()*+&%,$%-$&%,./***0*1#($%&/2*3%$!%4$*+56$,#6)/7-8,&9700*+5:6$;<=#6>5"/-%+56>)/?00
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 189/219
k-G7-
N>89@
I)% <E/7@;] &/->-C>@] V:F0];0.E8F068;] N-8>;];E>@.-B;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 190/219
V!N&
&O
k:B@<:<
ID:;;@. )E96>@J ->FE.:;09< /-B C@ @J6.@<<@? 8<:BF5] &6-.7 ;.-B<GE.9-;:EB<5 ;$'IJR -,%29,IJR V&)5IJR ,+9] ,+>:C d Y.-60u] &_+
H-;/0 Q@->;:9@
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 191/219
$B@ 8B:G:@? '%Z
I-;0-F-;- !-< WI!X
] +@-? ?@L@>E6@. EG &6-.7 &;.@-9:BF d )E99:;;@.EB '6-/0@ &6-.7 /E.@
] V@>6@? .@]D.:;@ &6-.7 )E.@ :B;@.B->< :B 1231 ;E9-7@ :; 32J G-<;@. ;E <866E.; &;.@-9:BF 8<@ /-<@<
] $B >@-L@ G.E9 P) [email protected]@>@M %0! 6.EF.-9
] "J5 ZB;@.B a '9-jEB ZB;@.B a )EBL:L- Q@<@-./0
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 192/219
"J5 ZB;@.B a '9-jEBK ZB;@.B a )EBL:L-K Q@<@-./0'<<:<;-B; a ,:/.E<EG; Q@<@-./0 ZB?:-
] 3 F8Mx ?E@< BE; </->@
] &/->@< ;E 322< EG BE?@<
] H-;/0 <:j@< -< <9->> -; 0->G - <@/EB?
] %.E/@<<:BF >-;@B/M -< >ED -< 3 <@/EB?
] "J-/;>M]EB/@ <@9-B;:/< BE 9-;;@. D0-; G-:><
W>:L@ <;-;:<;:/<X
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 193/219
%-F@ L:@D< k-G7- GE. C8GG@.:BF &6-.7 G
&9-.; 9@;@. .@-?:BF<W'BE9->M !@;@/;:EBX
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 194/219
+:L@ D@-;0@. ?-;-
\E:B 1 >:L@ ?-;-<E8./@<
ZB68; ?-;- <;.@-9
H-;/0@< @[email protected] u <@/EB?<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 195/219
W!:</.@;:j@? &;.@-9XH-;/0 :B;@.L-> o 4 <@/EB?<
I o 2 I o d
ZB68;
!&;.@-9
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 196/219
H>E/7 s3
Q!! a Io2
H>E/7 s1 H>E/7 sO H>E/7 s3
Q!! a Iod4
H>E/7 s1 H>E!&;.@-9
$B@ Q!! :< /.@-;@? @[email protected] 4 <@/EB?<
H>E/7 s3 H>E/7 s1 H>E/7 sO
4 <@/ ,-;@.:->:j@h
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 197/219
%-.;A s3 %-.;A s1 %-.;A sO
%-.;A s3 %-.;A s1 %-.;A sO
H&-%:!""
3*$+.$'IJ
*)5,:!""
3-&; '6:'$-B );'&-+ 0'$-B@&5+,=+
3-&; '6:'$-BK:+-,$;)5? );'&-+ 0+-,$;)5?@&5+,=+
2 40/#$/ # +5.#+ %$0/#6-78457$/&$ 9-$1 $95 950:-78 $10/#; #7; <#$.1 -7$/0=#+ 5> ? ,/.57;
:9 F 0'$-B@&5+,=+IL*&9$*MOPLR LD,+H&-BY&-%@&25+LJ
::9 F 0+-,$;)5?@&5+,=+I:9R aJ
2 40/#$/ # "%$0/#6 $1#$ 9-++ .577/.$ $5 15,$7#6/@A50$B +-:/ +5.#+15,$@CCCC
*)5,:"0+-,$; F ::9K:&9B,+C,=+0+-,$;IL*&9$*1&:+LR ssssJ
% +-$ 1 +- - $ ;
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 198/219
2 %A+-$ /#.1 +-7/ -7$5 950;,
H&-%:"0+-,$; F *)5,:"0+-,$;K3*$+.$'I*$;7%$ *)5,[ *)5,K:'*)+IL LJJ
2 45D7$ /#.1 950; -7 /#.1 <#$.1
'$)-:"0+-,$; F H&-%:"0+-,$;K;$'I*$;7%$ H&-%[ IH&-%R NJJ
H&-%@&25+:"0+-,$; F '$)-:"0+-,$;K-,%29,`6W,6I*$;7%$ =R 6[ = l 6J
2 E0-7$ $1/ >-0,$ $/7 /+/6/7$, 5> /#.1 )"" 8/7/0#$/; -7 $1-, "%$0/#6 $5 $1/ .57,5+/
H&-%@&25+:"0+-,$;K''-)5+IJ
::9K:+$-+IJ 2 %$#0$ $1/ .56AD$#$-57
::9K$H$)+C,-;)5$+)&5IJ 2 F#-$ >50 $1/ .56AD$#$-57 $5 $/06-7#$/
[email protected]:B-> s3 [email protected]:B-> s1
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 199/219
h 59 U*B ssss
1,**& H&-*%
h KT5,+H&-BXH&-%9&25+K'6 *&9$
K K K
UUUUUUUUUUUUUUUUUUUUUUUUUU
C);,[ OcNaUcrUOa Na[Oa[ON
UUUUUUUUUUUUUUUUUUUUUUUUUU
I1,**&R OJIH&-*%R NJ
!"
/**0 12
#
/**0 13
NACD<0 12
:
: :
: : !"
/**0 14
#
: :
H-;/0 :B;@.L-> o c22 9<
]
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 200/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 15
NACD<0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
!"
/**0 12
#
/**0 13
NACD<0 12
: ]
: :
: :
!"
/**0 14
#
/** 15
: :
!
#
NACD<0 13
NACD<0 13
H-;/0 :B;@.L-> o c22 9<
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 201/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 15
NACD<0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
!"
/**0 12
#
/**0 13
NACD<0 12
: ]
: :
: : !"
/**0 12
#
/** 13
: :
!
#
NACD<0 13
NACD<0 13
H-;/0 :B;@.L-> o c22 9<
NACD<0 14
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 202/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 13
NACD<0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
NACD<0 14
!"
/**0 12
#
/**0 13
/**0 12
: ]
: :
: : !"
/**0 12
#
/** 13
:
: :
:
!
#
/**0 13
/**0 13
H-;/0 :B;@.L-> o c22 9<
/**0 14
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 203/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 13
/**0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
/**0 14
!"
/**0 12
#
/**0 13
/**0 12
: ]
: :
: : !"
/**0 12
#
/** 13
:
: :
:
!
#
/**0 13
/**0 13
H-;/0 :B;@.L-> o c22 9<
/**0 14
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 204/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 13
/**0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
/**0 14
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 205/219
!"
#
NACD<0 12
: ]
6 , *
: :
: : !"
#
:
: :
:
H-;/0 :B;@.L-> o c22 9<
!
#
NACD<0 12
1 :B
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 206/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
NACD<0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
NACD<0 12
!"
#
NACD<0 12
: ]
6',&(')*
: :
: : !"
#
:
: :
:
!
#
NACD<0 12
6 ,
H-;/0 :B;@.L-> o c22 9<
NACD<0 13
NACD<0 14
NACD<0 13
NACD<0 14
NACD<0 13
NACD<0 14
NACD<0 13 NACD<0 14
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 207/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
NACD<0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
NACD<0 12
!"
#
/**0 12
: ]
6',&(')*
: :
: : !"
#
:
: :
:
!
#
/**0 12
6',&
H-;/0 :B;@.L-> o c22 9<
/**0 13
/**0 14
/**0 13
/**0 14
/**0 13
/**0 14
/**0 13
/**0 14
,-;@.:->:j@h
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 208/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 12
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
: :
6',&:.(&
V< 421O
/**0 12
!"
#
/**0 14
: ]
6',&(')*
: :
: : !"
#
:
: :
:
!
#
/**0 14
6',&
H-;/0 :B;@.L-> o c22 9<
/**0 15
/**0 1?
/**0 13
/**0 13
/**0 1?
/**0 12
/**0 12
/**0 16
PB:EBh
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 209/219
*+,-.+
6',&(')*:.(&)>1
&&! &&!V< 421O
/**0 15
6',&(')*:.(&)>1
&&! &&!V< 421O
: :
6',&:.(&
V< 421O
/**0 16
] N:>@ <M<;@9<] &E/7@; )EBB@/;:EB<
] '77- '/;E.<
] k-G7-] N>89@
] ID:;;@.
] 'BMD0
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 210/219
&E8./@< ?:.@/;>M -L-:>-C>@:B 0+-,$;)5?@&5+,=+ '%Z
Q@n8:.@< >:B7:BF -F-:B<;@J;.- ?@6@B?@B/:@<
Q@n8:.@< :968<@.]?@G:B@?
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 211/219
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 212/219
;$'I J
3*$+.$'I J
3)*+,-I J
25)&5I &+1,
-,%29,I J
V&)5I 0 M PJ
2'%$+,0+$+,`6W,6I Jm
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 213/219
3)*+,-I J
-,'$-+)+)&5I52;/$-+)+)&5:J
9&25+I
9&25+`6<$*2,IJ
-,%29,Q`6W,6I RM52;C$:B:PJ
V&)5I &+1,-0+-,$;RM52;C$:B:PJ9&?-&2'
+-$5:3&-;I J
/**
/**
2'%$+,0+$+,`6W,6I J
IE 8<@5
5 ->>ED< ME8 ;E 9-:B;-:B -.C:;.-.M <;-;@ D0:>@/EB;:B8E8<>M 86?-;:BF :; D:;0 B@D :BGE.9-;:EBA
IE 9-:B;-:B - .8BB:BF /E8B; EG @-/0 D:B - ;@J; ?-;- <;.@-9 !+'6' 6)--#-2 /0
'$)-: F
m
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 214/219
3X !@G:B@ ;0@ <;-;@Y%( %#:'"#%#D 1%"% "D-$Z
1X !@G:B@ ;0@ <;-;@ 86?-;@ G8B/;:EBY8-$C'XD V'". % X5(C"'2( .2V "2 5-1%"$ ".$ 8"%"$ 58'() ".$
-#$\'258 8"%"$ %(1 ($V \%E5$8 X#2& ".$ '(-5" 8"#$%&Z
%,3 2'%$+,(259+)&5I5,H<$*2,:R -255)5
)3 -255)5?@&25+ ): D&5,[
-255)5?@&25+ F c
-,+2-5 :2;I5,H<$*2,:R -255)5?@&2
^ 5,H E$*2,: H)+1 +1, '-,E
^ +& ?,+ +1, 5,H 9&25+
:B - ;@J; ?-;- <;.@-9 !+'6' 6)--#-2 /0#-.'2'6 .L(' 05 *.B.'A5
-255)5?@&25+: F '$)-:K2'%$+,0+$+,`6W
m Q@n8:.@< - /0@/76E:B; ?:.@/;E.M ;E C@/EBG:F8.@?
NE. @J-96>@5
5 /-B C@ 8<@? ;E -66>M -BM Q!! [email protected];:EB ;0-;:< BE; @J6E<@? :B ;0@ !&;.@-9 '%ZA
:'$;]53&!"" F :9K')9B*,()*,IKKKJ ^ !"" 9&5+$)5)5? :'$;
^ V&)5 %$+$ :+-,$; H)+1 :'$; )53&-;$+)&5 +& %& %$+$ 9*,9*,$5,%"0+-,$; F H&-%@&25+:K+-$5:3&-;I*$;7%$ -%%[
-%%KV&
+-$5:3&-;I J
/**
/**
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 215/219
6
] N8B/;:EB->:;M ;E bE:B @[email protected] C-;/0 :B -?-;- <;.@-9 D:;0 -BE;0@. ?-;-<@; :< BE;?:.@/;>M @J6E<@? :B ;0@ !&;.@-9 '%ZA
] ZG ME8 D-B; ;E ?E .@->];:9@ ?-;-
/>@-B:BF CM bE:B:BF ;0@ :B68; ?-;-<;.@-9 D:;0 6.@]/E968;@? <6-9:BGE.9-;:EB -B? ;0@B G:>;@.:BF C-<@? EB :;A
V
E.
,+>:C Y
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 216/219
H)5%&HI H)5%&H\,5?+1R :*)%,]5+,-E$*J
9&25+`6Y)5%&HI H)5%&H\,5?+1R :*)%,]5+,-E$*J
9&25+`6<$*2,Q5%Y)5%&HI H)5%&H\,5?+1R
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 217/219
-,%29,`6Y)5%&HI R H)5%&H\,5?+1R :*)%,]5+,-E$*J
-,%29,`6W,6Q5%Y)5%&HI R H)5%&H\,5?+1
R :*)%,]5+,-E$*
RM52;C$:B:P
J
-,%29,`6W,6Q5%Y)5%&HI R R H)5%&H\,5?+1R :*)%,]5+,-E$*RM52;C$:B:PJ
'-)5+IJ
:$E,Q:C,=+()*,I'-,3)=R M:233)=PJ
3&-,$9
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 218/219
:$E,Q:Z7V,9+()*,:I'-,3)=R M:233)=PJ
:$E,Q:#$%&&'()*,:I'-,3)=R M:233)=PJ
7/23/2019 Spark Advanced DevOps
http://slidepdf.com/reader/full/spark-advanced-devops 219/219