7/25/2019 Spark Ops Final

1/45

Patrick Wendell

Databricks

Deploying andAdministering Spark

7/25/2019 Spark Ops Final

2/45

Outline

Spark components

Cluster managers

Hardware & confguration

Linking wit Spark

!onitoring and measuring

7/25/2019 Spark Ops Final

3/45

Outline

Spark components

Cluster managers

Hardware & confguration

Linking wit Spark

!onitoring and measuring

7/25/2019 Spark Ops Final

4/45

Spark application

Dri"er program

#a"a program tat creates a

SparkConte$t

%$ecutors

Worker processes tat e$ecutetasks and store data

7/25/2019 Spark Ops Final

5/45

Cluster managerCluster manager grantse$ecutors to a Spark application

7/25/2019 Spark Ops Final

6/45

Dri"er programDri"er program decides wen tolaunc tasks on wic e$ecutor

eeds 'ull networkconnecti"ity to workers

7/25/2019 Spark Ops Final

7/45

(ypes o' ApplicationsLong li"ed)sared applications Sark Spark Streaming #ob Ser"er *Ooyala+

Sort li"ed applications

Standalone apps Sell sessions

May do mutli-user

scheduling within

allocation from

cluster manger

7/25/2019 Spark Ops Final

8/45

Outline

Spark components

Cluster managers

Hardware & confguration

Linking wit Spark

!onitoring and measuring

7/25/2019 Spark Ops Final

9/45

Cluster !anagers

Se"eral ways to deploy Spark

,- Standalone mode *on.site+

/- Standalone mode *%C/+

0- 1A2

3- !esos

4- S5!2 6not co"ered in tis talk7

7/25/2019 Spark Ops Final

10/45

Standalone !ode8undled wit Spark

9reat 'or :uick ;dedicated