Migration to Docker with ECS

ECS in Data Services

Change Drivers● Per-Project ops code (CMS2, SB2, Oracle, Mumford, Catalog Services,

Banyan, CIS, CSI, CLS, External Syndication, PRSyndication, etc.)

● Local/Cloud deployments are different

● Low cost efficiency for non-prod environments

● Slow deployments

Main Goals● Improve Development Process

● Improve Deployment Process

● Reduce Costs

● Transparent Migration

Development Process Improvements● Airplane Mode

● Fast Iterations

● Pre-committed Tests

● Local ~= Cloud

Deployment Process Improvements● Single Process for Multiple Projects

● Better EC2 Resources Utilization

● Blue/Green Deployment

● Rapid Deployment

Cost Reduction Areas● Low CI/CERT efficient ratio

● CI is mostly for cloud-adopted tests

● Instance types not optimal (CPU, Memory, Storage)

Exploration● Generalize current deployment (CloudFormation + Puppet + CLI) across all

projects (Common ops Git project + Project-specific ones)

● Custom AMIs (Packer)

● Flynn

● Docker Swarm

● EC2 Container Service (ECS)

EC2 Container Service

ECS Benefits● IAM Support

● Dynamic Resources Allocation

● Pay Only for EC2 Resources

● Scaling Based on CloudWatch

● Blue/Green Deployment

● ECR (EC2 Container Registry) Integration

● Deployment Configuration Percentage

ECS Components● Cluster

● Container Instance

● Task Definition

● Scheduler

● Service

● Task

● Container

ECS Limitations● Fixed ELB to Container Relationship (One Instance - One Service)

● Limited ELB Features

● Lack of ECS Support in CloudFormation

Dynamic Services● Dynamic private IP

addresses

● Dynamic Ports

Service Discovery● Etcd

● Consul

● Zookeeper

● Airbnb Synapse

● SkyDNS

● ...

Service Registration● Registrator

● Netflix Prana

● Airbnb Nerve

● ...

Dynamic Balancing● Consul-template

● Traefik

● Tutum

● etc

Consul + Registrator + Consul-template(HAProxy)

Container Balancing Modes Port-to-Port Balancing

APP_PORT 9080

ENV SERVICE_${APP_PORT}_TAGS PTP

Name-based Balancing

APP_PORT 9080

ENV SERVICE_${APP_PORT}_TAGS ci-banyan, ci-banyan-int

Zookeeper/Badger RegistrationProblem

● Docker private socket: 172.17.0.2:9081 cat not be reachable from outside

Solution

● Consul lookup for external socket. Socket of EC2 instance: 10.0.33.156:32457

● Application config change to allow override ZK/Badger socket

ECS CloudFormation: Missing Features● Logging support not implemented

● Deployment Configuration not implemented

● Only TCP ports can be exposed

● Task Revision update not implemented

● Inactive tasks flood Web interface

ECS Toolset● Thor - Ruby-based CLI Toolkit

● CloudFormation for ECS EC2 Resources

● Ruby AWS SDK v2 for ECS Resources

Demo

Dev Side

Dev Process Changes● Use Docker

● Provide your jar and build an image

● Mount your code (for not-compiling frameworks)

● Test cloud setup locally instead of CI

● Push your image to ECR

Demo

To Do List ● Proceed with migration

● Related services in containers (EmoDB, Polloi, etc)

● CLI improvements

● Registrator for ZK and Badger

● Better scaling

Questions?

Linkshttps://github.com/bazaarvoice/data-service-deploy

https://bits.bazaarvoice.com/confluence/display/DEV/Data+Services+Docker+Deployment+Study

https://bits.bazaarvoice.com/confluence/display/DEV/Data+Services+Container+Platform+Requirements

https://www.consul.io

https://github.com/gliderlabs/registrator

https://github.com/hashicorp/consul-template