pursuit of stability growing aws ecs in production · pursuit of stability growing aws ecs in...
TRANSCRIPT
Pursuit of stability
Growing AWS ECS in production
Alexander Köhler Frankfurt, September 2018
2
Alexander Köhler
DevOps EngineerSystems Engineer
Big Data Engineer
Application Developer
@la3mmchen
inovex is an IT project house
with a focus on digital transformation
‣ Product Discovery · Product Ownership
‣ Web · UI/UX · Replatforming · Microservices
‣ Mobile · Apps · Smart Devices · Robotics
‣ Big Data & Business Intelligence Platforms
‣ Data Science · Data Products · Search · Deep Learning
‣ Data Center Automation · DevOps · Cloud · Hosting
‣ Agile Training · Technology Training · Coaching
Using technology to inspire our clients. And ourselves.
Karlsruhe · Pforzheim · Stuttgart · München · Köln · Hamburg
www.inovex.de/en
The startWhat the customer told us to do.
4
host web applications that our SCRUM team develops
The startWhat the customer didn’t told us right on the spot.
5
host web applications that our SCRUM team develops
different languages (e.g.
java, go)
SSO solution Multi-
stage setup
Cloud based
Need for an android build
server
Distribute APKs
Different maintainers
The startWhat we told the customer we could do.
6
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
The startWhat we told the customer we could do.
7
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
different languages (e.g.
java, go)
SSO solution Multi-
stage setup
Cloud based
Android Build Server
Distribute APKs
Different maintainers
The startWhat we told the customer we could do.
8
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
SSO solution Multi-
stage setup
Cloud based
Android Build Server
Distribute APKs
Different maintainers
Wrap the apps in containers
The startWhat we told the customer we could do.
9
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
SSO solution Multi-
stage setup
Cloud based
Android Build Server
Distribute APKs
Wrap the apps in containers Add CI/CD
pipelines
The startWhat we told the customer we could do.
10
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
SSO solution Multi-
stage setup
Android Build Server
Distribute APKs
Wrap the apps in containers Add CI/CD
pipelinesAWS
The startWhat we told the customer we could do.
11
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Multi-stage setup
Android Build Server
Distribute APKs
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
The startWhat we told the customer we could do.
12
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Android Build Server
Distribute APKs
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
Infrastructure-as-Code to deploy in
multiple AWS accounts
The startWhat we told the customer we could do.
13
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Distribute APKs
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
Infrastructure-as-Code to deploy in
multiple AWS accounts
Ec2 instance with planned builds
The startWhat we told the customer we could do.
14
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
Infrastructure-as-Code to deploy in
multiple AWS accounts
Ec2 instance with planned builds Provide S3 buckets for
upload/download
Round 1
15
Round 1“Containers in production.”
16
Back in 2017: No EKS.But Kops with AWS support.
Round 1“Containers in production.”
17
Back in 2017: No AWS Fargate. (as long as i remember)But still Kops with AWS support.
Round 1“Let‘s bring containers in production“, they said.
18
AWS ECS to the rescue.Easy setup, I’ve heard.
Round 1“Let‘s bring containers in production“, they said.
19
Round 1“Let‘s bring containers in production“, they said.
20
ECS itself is pretty easy, but there are some more components involved.
Round 1How exactly bring container now into production?
21
ECS Cluster
ECS Service ECS Service ...
Round 1How exactly bring container now into production?
22
ECS Cluster
ECS Service ECS Service ...
task. (id 42) task. (id 41) …
Round 1How exactly bring container now into production?
23
ECS Cluster
EC2 node
ECS Service ECS Service ...
EC2 node ...VPC
ECS Ami
Round 1How exactly bring container now into production?
24
ECS Cluster
EC2 node
ECS Service ECS Service ...
EC2 node ...
Docker Registry
VPC
ECS Ami
Round 1How exactly bring container now into production?
25
ECS Cluster
EC2 node
Load balancer
ECS Service ECS Service ...
EC2 node ...
DNS Certs
Docker Registry
VPC
ECS Ami
Round 2
26
Round 2So, replace ECS Ami
27
What we knew good: Ubuntu 16.04 LTS. Should be easy.
Round 2Move away from ECS AMI
28
ECS Cluster
EC2 node
Load balancer
ECS Service ECS Service ...
EC2 node ...
DNS Certs
Docker Registry
VPC
ECS Ami Ubuntu AMI
ECS AMI
Round 2Move away from ECS AMI
29
Docker
ECS agent
Ubuntu
ECS AMI
Round 2Move away from ECS AMI
30
Docker
ECS agent
Ubuntu
Docker
ECS agent
ECS AMI
Round 2Move away from ECS AMI
31
Docker
ECS agent
Ubuntu
EBS Volume
Docker
Docker-Root at EBS
ECS agent
32
Round 3
Round 3Bring up a monitoring solution
33
It’s AWS. There is cloudwatch?Or do prometheus like everyone else?
Round 3Bring up a monitoring solution
34
We did prometheus. And ended up with something like this.
Round 3Let prometheus collect all the metrics from all the sources.
35
Monitoring stack
ECS cluster
Container
EC2 instancesA
WS A
pi
Round 3Let prometheus collect all the metrics from all the sources.
36
Monitoring stack
ECS cluster
Container
EC2 instancesA
WS A
pi
Prometheus
ECS Discovery
EC2 discovery
AlertmanagerGrafana
BlackboxExporter
Round 3Let prometheus collect all the metrics from all the sources.
37
Monitoring stack
ECS cluster
Container
EC2 instancesA
WS A
pi
Prometheus
ECS Discovery
EC2 discovery
AlertmanagerGrafana
BlackboxExporter
Node Exporter
Docker Daemon Metrics
ContainerStarts
CAdvisor
Round 4
38
Ever tried to debug Keycloak in a container not starting properly on a distributed infrastructure?
We did. We ended up somehow like this.
Read all the shells.
Round 4
39
Lets collect the logs.
Read all the shells. Nay.
Round 4Setup a log pipeline.
40
ECS cluster
Container
EC2 instances
Docker
Elasticsearch Service on AWS
Round 4Setup a log pipeline.
41
ECS cluster
Container
EC2 instances
Docker json files
filebeat
processor “add_docker_metadata” Elasticsearch
Service on AWS
42
Round 5Add this magic thing called CI/CD.
gitlab-runner
make
gitlab-runner
docker build
gitlab-runner
docker push
gitlab-runner
terraform apply
gitlab repository gitlab repositorydeploy/
gitlab registry(docker registry)
gitlab registry(docker registry)
AWS ECR(docker registry)
build artifact(e.g. dist/)
build artifact(e.g. dist/)
ECS cluster(running service)
Executed in both the Dev and the Test environment.
Round 5The pipeline
43
Actually, pretty easy.Some different kind of “easy”.
Round 5Toolstack
44
Gitlab
ECS Service
ECS Cluster
EC2
Round 5Toolstack
45
Gitlab
ECS Service
ECS ClusterGitlab Runner
EC2
Terraform
Round 5Toolstack
46
Gitlab
ECS Service
ECS ClusterGitlab Runner
EC2
Terraform
Terraform (aws-
infrastructure)
ECS Service
ECS ClusterGitlabRunner
EC2
Gitlab-CI
Round 5The pipeline itself
47
Push to ECR-Dev
Build App Wrap in Docker
docker login ecr registrydocker push
Add to GitlabDocker
Registry
Push to next-Stage
Deploy ECS services
1.Pull from gitlab2.Push to ECR
48
Final round.
ConclusionWhat I’ve talked about
49
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
Infrastructure-as-Code to deploy in
multiple AWS accounts
Ec2 instance with planned builds Provide S3 buckets for
upload/download
ConclusionWhat we build but wasn’t mentioned.
50
Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar
on top and host your web applications in AWS.
Wrap the apps in containers Add CI/CD
pipelinesAWS
Keycloak
Infrastructure-as-Code to deploy in
multiple AWS accounts
Ec2 instance with planned builds Provide S3 buckets for
upload/download
Final wordsBring container in production.
51
ECS itself: easy and straightforward.The complete stack with supporting tools: can be tricky.
Final wordsProven tool chain
52
Provision with terraform.Gitlab-CI as CI/CD base.
Gitlab-Runner as workhorse.
Final wordsWhat we might do in another way the second time.
53
Take a clearer decision about the usage of additional AWS service.
Final wordsWhat we might do in another way the second time.
54
Re-check if fargate isn’t the better solution for most of the services
Vielen DankAlexander Köhler
DevOps Engineer
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
https://github.com/inovex/aws-ecs-kickstart