flink in a container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name...
TRANSCRIPT
![Page 1: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/1.jpg)
Flink in a Container
Learn it the Caipi-Way
Nico Kreiling Karlsruhe, 21.2.2017
![Page 2: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/2.jpg)
2
Flink in a Container
![Page 3: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/3.jpg)
3
The Ingridients
![Page 4: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/4.jpg)
4
If you just mix it all...
![Page 5: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/5.jpg)
5
But bundled in a nice container...
![Page 6: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/6.jpg)
6
And Containers can be used for many drinks
![Page 7: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/7.jpg)
7
Why not use a VM? Source:http://stackoverflow.com/a/33132462/1806392
![Page 8: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/8.jpg)
8
Rezept = Dockerfile
![Page 9: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/9.jpg)
9
Let’s make a it…
Create Dockerfile
docker build –t <image> .
![Page 10: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/10.jpg)
10
If you are to lazy to build it yourself...
docker pull <image>
![Page 11: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/11.jpg)
11
Let’s make a it…
Create Dockerfile
docker build –t <image> .
docker run <image>
![Page 12: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/12.jpg)
12
Thats all?
![Page 13: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/13.jpg)
13
Scale out!
But how?
![Page 14: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/14.jpg)
14https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/cluster_setup.html
Flink Cluster: Standalone Mode
![Page 15: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/15.jpg)
› docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager
› docker run --expose 6121,6122 --link jobmanager1 krlng/flink taskmanager
› docker run --expose 6121,6122 --link jobmanager2 krlng/flink taskmanager
15
Create the pure way
![Page 16: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/16.jpg)
› YAML Syntax
› Similar to classic Docker Commands
› Custom Networking
› Easy to re-run
16
Docker-Compose
![Page 17: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/17.jpg)
version: "2"services:flink-jobmanager:image: krlng/flinkports:- "8081:8081"- "6123:6123"
command: jobmanager
17
Docker-Compose
flink-taskmanager:image: krlng/flinkexpose:- "6121"- "6122”
links:- flink-jobmanager
command: taskmanager
![Page 18: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/18.jpg)
18
It‘s kind of a messBut wait, something is missing...
There we are!
![Page 19: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/19.jpg)
19
Don‘t try that...
![Page 20: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/20.jpg)
20
We need something to consume it
![Page 21: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/21.jpg)
› Service discovery: Let’s talk containers to each other
› Monitoring: track containers’ and cluster health
› Configuration scripting: Load application configuration into containers
› Establish container policies: Configure resources limits
› Provisioning: spin up containers depending on their requirements
› Rolling upgrades and rollback
21Source: https://insights.hpe.com/content/hpe-nxt/en/articles/2017/02/the-basics-explaining-kubernetes-mesosphere-and-docker-swarm.html
Container Management
![Page 22: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/22.jpg)
22
Container Orchestration
![Page 23: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/23.jpg)
23
Kubernetes Terminoligy
Pod(==Container)
ReplicationController
Service
Namespace
![Page 24: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/24.jpg)
24
Namespace: flink
RC: Jobmanager
Pod: Jobmanager
Pod: Taskmanager
RC: Taskmanager
Service: WebUI
Service: jobmanager
![Page 25: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/25.jpg)
› More Applications: Using real-world problems
› More Tools: Spark, Kafka, Cassandra, ElasticSearch…
› More Automation: Build-Pipeline
25
Whats next?
![Page 26: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/26.jpg)
26
Bringing it all together in an automated way
Kubernetes
ReplicationController
Pod
Service(+LB)
PodGitlab
WebGUI
DockerImage
RESTAPIgit push
kubectl
dockerpull
![Page 27: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/27.jpg)
Thanks for the Attention!Time for Questions!
27
![Page 28: Flink in a Container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager › docker run --expose 6121,6122 --link jobmanager1](https://reader035.vdocument.in/reader035/viewer/2022062602/5ec56484040010522219d16d/html5/thumbnails/28.jpg)
Vielen Dank
Nico Kreiling
Big Data Scientist
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
0173 3181 168