aws august webinar series - running batch jobs on ecs

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Chris BarclayPrincipal Product Manager

Running Batch Jobs on Amazon EC2 Container Service

Agenda

Containers

EC2 Container Service

Common patterns

Demo

Q&A

Containers

What are Containers?

OS virtualization

Process isolation

Images

Automation Server

Guest OS

Bins/Libs Bins/Libs

App2App1

Container Advantages

PortableServer

Guest OS

Bins/Libs Bins/Libs

App2App1

Container Advantages

FlexibleServer

Guest OS

Bins/Libs Bins/Libs

App2App1

Container Advantages

FastServer

Guest OS

Bins/Libs Bins/Libs

App2App1

Container Advantages

EfficientServer

Guest OS

Bins/Libs Bins/Libs

App2App1

Server

Guest OS

Bins/Libs Bins/Libs

App2App1

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

Server

Guest OS

EC2 Container Service Benefits

Easily Manage Clusters for Any Scale

Nothing to run

Complete state

Control and monitoring

Scale

Flexible Container Placement

Applications

Batch jobs

Multiple schedulers

Designed for Use with Other AWS Services

Elastic Load Balancing

Amazon Elastic Block Store

Amazon Virtual Private Cloud

AWS Identity and Access Management

AWS CloudTrail

Extensible

Comprehensive APIs

Open source agent

Custom schedulers

EC2 Container Service Terminology

Amazon EC2 instances

Docker daemon

Amazon ECS agent

Key Components: Container Instances

Regional

Resource pool

Grouping of Container Instances

Start empty, dynamically scalable

Key Components: Clusters

Key Components: Task Definitions

Volume Definitions

Container Definitions

Key Components: Task Definitions

Shared Data Volume

PHP App Time of day App

Key Components: Task Definitions { "environment": [], "name": "simple-demo", "image": "my-demo", "cpu": 10, "memory": 500, "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "mountPoints": [ { "sourceVolume": "my-vol", "containerPath": "/var/www/my-vol" } ], "entryPoint": [ "/usr/sbin/apache2", "-D", "FOREGROUND" ], "essential": true },

{ "name": "busybox", "image": "busybox", "cpu": 10, "memory": 500, "volumesFrom": [ { "sourceContainer": "simple-demo" } ], "entryPoint": [ "sh", "-c" ], "command": [ "/bin/sh -c \"while true; do /bin/date > /var/www/my-vol/date; sleep 1; done\"" ], "essential": false }

{ "environment": [], "name": "simple-demo", "image": "my-demo", "cpu": 10, "memory": 500, "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "mountPoints": [ { "sourceVolume": "my-vol", "containerPath": "/var/www/my-vol" } ], "entryPoint": [ "/usr/sbin/apache2", "-D", "FOREGROUND" ], "essential": true },

Key Components: Task Definitions[ { "image": "mysql", "name": "db", "cpu": 10, "memory": 500, "essential": true, "entryPoint": [ "/entrypoint.sh" ], "environment": [ { "name": "MYSQL_ROOT_PASSWORD", "value": "pass" } ], "portMappings": [] }]

Essential to our Task

Create and mount volumes

Expose port 80 in containerto port 80 on host

10 CPU Units (1024 is full CPU),500 Megabytes of Memory

{ "name": "busybox", "image": "busybox", "cpu": 10, "memory": 500, "volumesFrom": [ { "sourceContainer": "simple-demo" } ], "entryPoint": [ "sh", "-c" ], "command": [ "/bin/sh -c \"while true; do /bin/date > /var/www/my-vol/date; sleep 1; done\"" ], "essential": false }

Key Components: Task Definitions[ { "image": "tutum/wordpress-stackable", "name": "wordpress", "cpu": 10, "memory": 500, "essential": true, "links": [ "db" ], "entryPoint": [ "/bin/sh", "-c" ], "environment": [ … ], "portMappings": [ { "containerPort": 80, "hostPort": 80 } ] }, ]

From Docker Hub

Mount volume from other container

Command to exec

Key Components: Tasks

Container Instance

Schedule

Shared Data Volume

PHP App Time of day App

Shared Data Volume

PHP App Time of day App

Unit of work

Grouping of related Containers

Run on Container Instances

Key Components: Tasks

Key Components: Run a Task

Good for short-lived containers, e.g.

batch jobs

Key Components: Create a Service

Good for long-running applications

and services

Key Components: Create Service

Load Balance traffic across containersAutomatically recover unhealthy containersDiscover services

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Elastic Load Balancing

Key Components: Update Service

Scale upScale down

Elastic Load Balancing

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Key Components: Update Service

Deploy new versionDrain connections

Elastic Load Balancing

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Key Components: Update Service

Deploy new versionDrain connections

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Elastic Load Balancing

Key Components: Update Service

Deploy new versionDrain connections

Shared Data Volume

PHP App

Time of day App

Shared Data Volume

PHP App

Time of day App

Elastic Load Balancing

Shared Data Volume

PHP App

Time of day App

Batch jobs

Containers and Batch Jobs

Easy to update

Any app, any language

Includes all dependencies

Image is the version

Test & deploy same artifact

When to Use ECS and Lambda

ECSLong-running jobs

Manage your event triggering

Any language, any dependency

Resources are your own - use

Spot, RIs

LambdaShort-lived jobs

Triggered on specific events

Supports specific environments

No infrastructure to manage

Demo: Batch Jobs with ECS

.zip

s3://bucket/scene.zip

ecs:RunTask

ECS Cluster

.png

Demo

Q&A

AWS re:Invent 2015 – October 6-9AWS re:Invent is the largest annual gathering of the global cloud community. Whether you are an existing customer or new to the cloud, AWS re:Invent will provide you with the knowledge and skills to refine your cloud strategy, improve developer productivity, increase application performance and security, and reduce infrastructure costs.

Though AWS re:Invent tickets are sold out, you can still register to view the Live Stream Broadcasts of the keynote addresses and select technical sessions on October 7 and October 8. Register now.

Details:Wednesday, October 79:00am - 10:30am PT: Andrew Jassy, Sr. Vice President, AWS11:00am - 5:15pm PT: 5 of the most popular breakout sessions (to be announced)

Thursday, October 89:00am - 10:30am PT: Dr. Werner Vogels, CTO, Amazon11:00am - 6:15pm PT: 6 of the most popular breakout sessions (to be announced)

Register now for the Live Stream Broadcast by submitting your email where prompted on the AWS re:Invent home page.

Stay Connected: Follow event activities on Twitter @awsreinvent (#reinvent), or like us on Facebook.

Thank you!