ecs and ecr deep dive

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Apr 21 2016

Amazon EC2 Container Service Deep Dive

Shiva N, Solution Architect, AWS

Agenda

The BasicsInfrastructure SetupInfrastructure ManagementDeploying ApplicationsPaaS on ECSUsing the CLI

TaskDefinitionsContainers

ClustersContainer Instances

Key Components

Amazon ECS Infrastructure Setup

Amazon ECS Infrastructure Setup

Amazon ECS Cluster SetupAmazon ECR Setup

Amazon ECS Cluster Setup

Amazon ECS Cluster Setup

There are many ways to provision cluster infrastructure

v AWS – CloudFormation, Simple Systems Manager, Autoscale Groups, OpsWorks, ECS-CLI

v Others - Terraform, PaaS, Partners

Let’s talk about CloudFormation

Cluster Setup with AWS CloudFormation

CloudFormation supports ECS cluster, service and task definition resourcesUse AWS::IAM::Role to create ECS service role and container instances roleLaunch container instances using AWS:AutoScaling::LaunchConfiguation and AWS:AutoScaling::AutoScalingGroup


"Resources" : {

"ECSCluster": {

"Type": "AWS::ECS::Cluster"

},

"ECSAutoScalingGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"VPCZoneIdentifier" : { "Ref" : "SubnetID" },

"LaunchConfigurationName" : { "Ref" : "ContainerInstances" },

"MinSize" : "1",

"MaxSize" : { "Ref" : "MaxSize" },

"DesiredCapacity" : { "Ref" : "DesiredCapacity" }

},

[…]

},


"ContainerInstances": {

"Type": "AWS::AutoScaling::LaunchConfiguration",

"Metadata" : {

"AWS::CloudFormation::Init" : {

"config" : {

"commands" : {

"01_add_instance_to_cluster" : {

"command" : { "Fn::Join": [ "", [ "#!/bin/bash\n", "echo ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] }

}

},

[…]

}

}

}

Amazon ECR Setup

Amazon ECR Setup

You have read and write access to the repositories you create in your default registry, i.e. <aws_account_id>.dkr.ecr.us-east-1.amazonaws.comRepository names can support namespaces, e.g. team-a/web-app.Repositories can be controlled with both IAM user access policies and repository policies.

Amazon ECR Setup

# Authenticate Docker to your Amazon ECR registry

> aws ecr get-login

docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

# Create a repository called ecr-demo

> aws ecr create-repository --repository-name ecr-demo

# Push an image to your repository

> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

Amazon ECS Infrastructure Management

Amazon ECS Infrastructure Management

Monitoring & LoggingScaling ECSService Discovery & Configuration ManagementSecurity

Monitoring & Logging

Monitoring and Logging on Amazon ECS

Monitoring with Amazon CloudWatchConfiguring logging in Task DefinitionAmazon CloudTrailMonitoring Amazon ECS with DatadogMonitoring Amazon ECS with Sysdig Cloud

Monitoring with Amazon CloudWatch

Metric data sent to CloudWatch in 1-minute periods and recorded for a period of two weeksAvailable metrics: CPUReservation, MemoryReservation, CPUUtilization, MemoryUtilizationAvailable dimensions: ClusterName, ServiceName


Use the Amazon CloudWatch Monitoring Scripts to monitor additional metrics, e.g. disk space:

# Edit crontab

> crontab -e

# Add command to report disk space utilization to CloudWatch every five minutes

*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --from-cron

Configuring Logging in Task Definition

logConfiguration task definition parameterRequires version 1.18 or greater of the Docker Remote API Maps to docker run --log-driver option Log drivers: json-file, syslog, journald, gelf, fluentd

Logging with Amazon CloudWatch Logs

• Logging container with syslogd and CloudWatch Logs Agent

• Attach /var/log Volume to Logging container (Sidecar pattern)

• Link other containerssyslogd

CloudWatch Logs Agent

CloudWatch Logs

Container instance

ECS Cluster

ECS Agent Logs

Docker Logs

Logging Amazon ECS API with AWS CloudTrail

{

"eventVersion": "1.03",

"userIdentity": {…},

"eventTime": "2015-10-12T13:57:33Z",

"eventSource": "ecs.amazonaws.com",

"eventName": "CreateCluster",

"awsRegion": "eu-west-1",

"sourceIPAddress": "54.240.197.227",

"userAgent": "console.amazonaws.com",

"requestParameters": {

"clusterName": "ecs-cli"

},

Logging Amazon ECS API with AWS CloudTrail

"responseElements": {

"cluster": {

"clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli",

"pendingTasksCount": 0,

"registeredContainerInstancesCount": 0,

"status": "ACTIVE",

"runningTasksCount": 0,

"clusterName": "ecs-cli",

"activeServicesCount": 0

}

},

[…]

Monitoring Amazon ECS with Datadog

Monitoring Amazon ECS with Sysdig Cloud

Scaling Amazon ECS

Scaling Amazon ECS

AutoScaling your Amazon ECS clusterScaling your Services with Lambda

Setup ECS Cluster with AutoScaling

Create LaunchConfiguration• Pick instance type

depending on resource requirements, e.g. memory or CPU

• Use latest Amazon Linux ECS-optimized AMI, other distros available

Create AutoScaling group and set to cluster initial size

AutoScaling your Amazon ECS Cluster

• Create CloudWatch alarm on a metric, e.g. MemoryReservation

• Configure scaling policies to increase and decrease the size of your cluster

Scaling your Services with Lambda

• Cloudwatch metrics tied to SNS

• SNS triggers Lambda Container Scaling function

• Lambda scales task count on cluster

• Bonus - Extensible ‘cluster intelligence’ layer

Service Discovery & Configuration Management

Service Discovery on Amazon ECS

Service Discovery with ECS Services & Route 53Service Discovery with WeaveworksService Discovery and Configuration Management with ConsulService Discovery and Configuration Management with etcd

Service Discovery with ECS Services & Route 53

• Route 53 private hosted zone• Set search path on hosts with DHCP option sets• Define ECS services with ELB• Create CNAMEs for each ELB

Service Discovery with ECS Services & Route 53

Task

Task TaskTask

ECS Service

Application router, e.g.

nginx

Internal ELB with CNAME, e.g.

api.example.com

Route 53 private zone,

e.g. example.com

Service Discovery with Weaveworks

DNS interface for cross-host container communicationGossip protocol to share grouped updatesOverlay network between hosts

Service Discovery and Configuration Management with ConsulThree main components:• Consul agent - Runs on each node, responsible for

checking the health of the services and of the node itself.

• One or more Consul servers - Store and replicate data, leader elected using the Raft consensus algorithm

• Registrator agent - Automatically register/deregisters services based on published ports and metadata from the container environment variables defined in the ECS task definition

Service Discovery and Configuration Management with Consul

ECS

Clus

ter

consul-server

ECS Instance

consul-agent

registrator

ECS Instance

Back end 1

Back end 2

consul-agent

registrator

ECS Instance

Front end

ECS

Clus

ter

Service Discovery and Configuration Management with etcd

etcd

registrator

ECS Instance

Container 1

Container 2

confd etcd

registrator

ECS Instance

Container 1

Container 2

confd etcd

registrator

ECS Instance

Container 1

Container 2

confd

Security

Security

ECS IAM Policies and RolesECR IAM Policies and RolesImage Vulnerability Scanning with Twistlock

ECS IAM Policies and Roles

The ECS agent calls the ECS APIs on your behalf, so container instances require an IAM policy and role that allows these calls.The ECS service scheduler calls the EC2 and ELB APIs on your behalf to register and deregister container instances with your load balancers. Use AmazonEC2ContainerServiceforEC2Role and AmazonEC2ContainerServiceRole managed policies (respectively)

ECR IAM Policies and Roles

ECR uses resource-based permissions to control access.By default, only the repository owner has access to a repository. You can apply a policy document that allows others to access your repository.Use managed policies for IAM users or roles that allow differing levels of control: AmazonEC2ContainerRegistryFullAccess, AmazonEC2ContainerRegistryPowerUser or AmazonEC2ContainerRegistryReadOnly

Image Vulnerability Scanning with Twistlock

Deploying Applications

Deploying Applications

Scheduling ContainersAutomating Deployments

Scheduling Containers

Scheduling Containers on ECS

Batch Jobs

ECS Task schedulerRun tasks once

Batch jobsRunTask (random) StartTask (placed)

Long-Running Apps

ECS Service schedulerHealth managementScale-up and scale-downAZ awareGrouped Containers

Scheduling Containers: Long-running App

Optionally run your service behind a load balancer.One load balancer per service.ELB currently supports a fixed relationship between the load balancer port and the container instance port.If a task fails the ELB health check, the task is killed and restarted (until service reaches desired capacity).


Update service’s task definition (rolling update)Specify a deployment configuration for your service:• minimumHealthyPercent: lower limit (as a percentage of

the service's desiredCount) of the number of running tasks that must remain running in a service during a deployment.

• maximumPercent: upper limit (as a percentage of the service's desiredCount) of the number of running tasks that can be running in a service during a deployment.

Scheduling Containers: Long-running app

Deploy using the least space: minimumHealthyPercent = 50%, maximumPercent = 100%


Deploy quickly without reducing service capacity: minimumHealthyPercent = 100%, maximumPercent = 200%


Blue-Green Deployments

• Define two ECS services• Each service is associated w/ ELB• Both ELBs in Route 53 record set

with weighted routing policy, 100% Primary, 0% Secondary

• Deploy to Blue or Green service and switch weights

TaskTask

Route 53 record set

with weighted routing policy

0%100%

Automating Deployments

Automating Deployments

Continuous Delivery to ECS with JenkinsContinuous Delivery to ECS with Shippable

Continuous Delivery to ECS with Jenkins

4. Push image to Docker registry

2. Build image from sources 3. Run test on image

1. Code push triggers build

5. Update Service

6. Pull image

Continuous Delivery to ECS with Jenkins

Easy DeploymentDevelopers – Merge into master, done!

Jenkins Build StepsTrigger via Webhooks, Monitoring, LambdaBuild Docker image via Build and Publish plugin Push Docker image into RegistryRegister Updated Job with ECS API

Continuous Delivery to ECS with Shippable

ECS CI/CD Partners

PaaS on ECS

PaaS on ECS

AWS Elastic BeanstalkConvoxRemind Empire

AWS Elastic Beanstalk

Uses Amazon ECS to coordinate deployments to multicontainer Docker environmentsTakes care of tasks including cluster creation, task definition and execution

AWS Elastic Beanstalk

Elastic Beanstalk uses a Dockerrun.aws.json file that describes how to deploy containers.The Dockerrun.aws.json file includes three sections:• AWSEBDockerrunVersion: Set to "2" for multicontainer

Docker environments.• containerDefinitions: An array of container definitions.• volumes: Creates mount points in the container instance

that a container can use.

Convox

Convox

# Initialize your app and create default manifest

> convox init

# Locally build and run your app as declared in the manifest

> convox start

# Create app

> convox apps create my_app

# Deploy app, output ELB DNS name

> convox deploy

[...]

web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

Remind Empire

Control layer on top of Amazon ECS that provides a Heroku like workflowAny tagged Docker image can be deployed to Empire as an app • When you deploy a Docker image to Empire, it will

extract a Procfile from the WORKDIR• Each process type in the Procfile maps directly to an

ECS Service

Remind Empire

Routing layer backed by internal ELBs• An application that specifies a web process will get an

internal ELB attached to its ECS Service • When a new internal ELB is created, an associated

CNAME record is created in Route53 under the internal TLD, enabling service discovery via DNS

Using the CLI

Using the CLI

Configuring the ECS CLICluster Setup with the ECS CLIDeploy Compose App with ECS CLIScaling with ECS CLI

Configuring the ECS CLI

Easily create Amazon ECS clusters & supporting resources such as EC2 instancesRun Docker Compose configuration files on Amazon ECSAvailable today – http://amzn.to/1jBf45a

Configuring the ECS CLI

# Configure the CLI using environment variables

> export AWS_ACCESS_KEY_ID=<my_access_key>

> export AWS_SECRET_ACCESS_KEY=<my_secret_key>

> ecs-cli configure --region us-east-1 --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --cluster ecs-cli-demo

# Configure the CLI using an existing AWS CLI profile

> ecs-cli configure --region us-west-2 --profile ecs-profile --cluster ecs-cli-demo

Cluster Setup with the ECS CLI

# Creates a new ECS cluster with two container instances in an existing VPC

> ecs-cli up --capability-iam --keypair my_ecs_keypair --size 2 --security-group sg-a12bc34d --vpc vpc-0e9dc8b7 --subnets subnet-12ab34cd,subnet-56ef78ab --instance-type t2.medium

# Creates a new ECS cluster with one container instance in a new VPC

> ecs-cli up --capability-iam --keypair my_ecs_keypair --azs us-east-1a,us-east-1c --cidr 192.169.0.0/24 --port 22 --instance-type t2.medium

Deploy Compose App with ECS CLI

Docker Compose lets you define and run multi-container applications:1. Define app environment with Dockerfile2. Define services that make up your app in docker-

compose.yml3. Run docker-compose up to start and run entire app


proxy:

build: ./proxy

ports:

- "80:80"

links:

- web

web:

build: ./web

command: bundle exec rails server -b 0.0.0.0

environment:

- SECRET_KEY_BASE=secretkey

expose:

- "3000"


> ecs-cli compose up

> ecs-cli compose ps

> ecs-cli compose service create

> ecs-cli compose service start

Scaling with ECS CLI

> ecs-cli scale n

> ecs-cli compose scale n

> ecs-cli compose service scale n

Thank you!

ecs and ecr deep dive

Internet