aws re:invent 2016: service integration delivery and automation using amazon ecs (con308)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Tom Fuller, Sr. Solutions Architect

Pascal Hahn, HERE Director of Cloud Architecture

Jeremy Brown, HERE Lead Engineer

December 1, 2016

Service Integration Delivery and

Automation Using Amazon ECS

CON308

What to Expect from the Session

1. Overview of services being deployed on ECS

and techniques for managing those services

2. An overview of HERE and their Open Location

Platform and Delivery Practices

3. A detailed explanation of the lessons HERE

has learned automating their deployments on

top of ECS

Disclaimer

• Best practices are often relative and

time bound, these examples

represent experiences over the last

year

• Docker and the surrounding

ecosystem is moving fast and we are

only scratching the surface

• Innovations are not frozen in time!

DevOps - PSA

Why Is This Hard?

• Configuration

• Service discovery

• Complexity of cluster types running on

containers

• Volume of clusters

• Scalability

• High availability

Apache Foundation Services

Kafka

• Scalable and efficient

streaming ingestion

• Dependency on

Zookeeper

• Tight integration with

Storm

• Scales up with ease

• Works best with locally

attached storage

• Has different modes of

operation (high

throughput vs. high

resiliency)

Storm

• Scalable and efficient

distributed real-time

computation system

• Dependency on

Zookeeper

• You deploy topologies

that are made up of

spouts and bolts

• Spouts are the

abstraction that enables

integration

Zookeeper

• Centralized service for

distributed sync and

configuration

• Draws initial inspiration

from Chubby

• Very sensitive to latency

• Typical configurations

involve 3 or 5 nodes

• Uses Zab for consensus

What is Service Discovery?

• The world of microservices is

dynamic

• Containers bring about a layer of

routing

• You need a way to find these

services

• You need to know if the service is

healthy

• How do we find things on the

internet today? Why wouldn’t that

still work?

Service Registry

Service A Service B

publishlookup

bind

Service Health

health checkhealth check

delete

Service Discovery on AWS

• A closed DNS solution

(Amazon Route 53) to the

rescue

• Isolation benefits with VPC

and private zones

• How do you maintain the

entries

• Amazon CloudWatch +

AWS Lambda

• ecssd_agent.go

• What about health checks?

Listening to Docker Events

endpoint := "unix:///var/run/docker.sock”startFn := func(event *docker.APIEvents) error {

var err errorcontainer, err := dockerClient.InspectContainer(event.ID)logErrorAndFail(err)port, service := getNetworkPortAndServiceName(container, true)if port != "" && service != "" {

sum = 1for {

if err = createDNSRecord(service, event.ID, port); err == nil {break

}if sum > 8 {

log.Error("Error creating DNS record")break

}time.Sleep(time.Duration(sum) * time.Second)sum += 2

}}fmt.Println("Docker " + event.ID + " started")return nil

}

Configuration Data on Amazon ECS

• AWS Identity & Access Management (IAM) roles

for tasks to the rescue

• Limit access to specific configuration sources

with policy

• Requires version 1.11 of the container agent

and a supported SDK/CLI version

• VPC S3 endpoints provide an additional layer of

security

• The rest is basically standard Amazon EC2

goodness

• Temporary rotated credentials

• Strict policy enforcement on Amazon S3

• Encryption at rest with AWS KMS

HERE Open Location PlatformAnalytics Execution Environment

Pascal Hahn & Jeremy Brown

2016

The Open Location Platform will be the epicenter of an ecosystem

that creates value through innovation and collaboration

Automotive

Enterprise IoT

Consumer IoT

Location is the glue that will connect disparate data sources.

Core components critical to realizing the value of location data

IngestionBring your own data

all while maintaining

control of privacy and

access controls

EnrichmentCombine map

content with historic

and real-time data to

add location context

to your data and

services

PublishPromote and share

your enriched data

and services with

ecosystem partners

or through your own

channels

ProcessingAccess off-the-shelf

algorithms while

applying historic,

real-time and

predictive analytics

to your data

Analytics Execution Environment

• Analytics data and

processing• Data buckets

• Processing clusters

• Processing chains

• Users

• Operational services• Monitoring

• Cost

• Performance

• Availability

• Log management

• Network connectivity (user

VPN)

AEE – Architectural overview

AEE Console (Web-UI)

AEE RESTFul API

AEE UI Framework

Service Deployment APIResource Management

API

Deployment Orchestration

Framework

HERE

AAAIAM

ECS

EMR

S3 ELB

HERE

Frontend

Service Delivery Design

Service Delivery Architecture

CI/CD Servers

Image Management

Deploy StrategyAmazon ECS Amazon ECS

Service Discovery and

Registration

Overview of Consul and Vault

Consul• Distributed HA Service Registry

• Service Discovery (DNS)

• KV storage

• Health checks

Vault• Distributed HA secrets management

• Encrypted K/V storage

• Secrets and Certificate generation and

revocation

Service Registration and Discovery at HERE

For HEREs Service Delivery Platform, we organize the running services

into namespaces.

Basenamespace

• "/team/env/app”

Service Name

• tomcat

• mysql

team1-dev-app1-mysql

team1-dev-app1-tomcat

…

team2-prd-app4-mysql

team2-prd-app4-tomcat

Namespace Hierarchy

Root /

team1 team2

prddevprddev

app2app1app2app1 app4app3app4app3

mysql tomcatmysql tomcat

Namespaces help make policies sensible

IAM Policy

Namespaces help make policies cohesive

Consul Policy

How Zookeeper/Kafka/Storm discover each other

The trio always discover each other by base-namespace

team1-dev-app1-

zookeeper*

team1-dev-app1-

storm*

team1-dev-app1-

kafka*

Container Management

Configuring containers and services at run time

When configuring containers at run time, there a couple of

choices for how to do it.

Configurable containers

Containers are configured by the

scheduler at launch usually via envars

(-e flags)

Self-configuring containers

Containers are configured at runtime

by a templating mechanism usually

reading config data from a data-store

Configurable Containers

How to implement it?

• Store configuration in a repo or bucket available to the CI/CD server

that triggers ECS task updates.

Pros

• Easy to implement

• Containers are immutable

• Easier to reason about

Cons

• Envars are not secure

• Config Update requires restart

• Your scheduler is also your

configuration manager

Self-Configuring Containers

How to implement it? • Base Containers embed code that retrieves app configuration from distributed

• Data-stores like Consul K/V or etcd and/or a secrets store like Vault or Keywhiz.

• Values and secrets can be written out to config files via templates.

Pros

• Containers and processes can

be updated on the fly• Nginx use case

• Secrets are stored securely

• Configs can be easily viewed

Cons

• It takes a lot more work up front

• Local development is more

complex• Requires Consul and Vault on

dev machines

Managing container releases

and Hierarchies at scale

Register Docker Image for Updates

Discover versions and relationships from the Git

Repository

here/centos-7

FROM centos/centos-7:v4

here/java-8

FROM here/centos-7:v3

here/python:v8


here/flask:v10

FROM here/python:v8

here/tomcat-8:v8

FROM here/java-8:v12

here/tomcat-7:v12


Discover versions and relationships from the

Docker Registry

here/centos-7:v3

here/java-8:v12 here/python:v8

here/flask:v10here/tomcat-8:v8here/tomcat-7:v12

here/centos-7:v3here/centos-7:v3

here/java-8:v12here/java-8:v12

here/java-8:v12

here/python:v8here/python:v8

here/python:v8

here/tomcat-7:v12here/tomcat-7:v12


here/tomcat-7:v12




here/flask:v10here/flask:v10



Update to the latest Parent

New Image

here/centos-7:v4

java-8 Dockerfile

- FROM here/centos-7:v4

+FROM here/centos-7:v5

Python Dockerfile

- FROM here/centos-7:v4

+FROM here/centos-7:v5

tomcat-7 Dockerfile


tomcat-8 Dockerfile


flask Dockerfile

FROM here/python:v8

Update to the latest Parent

New Image

here/centos-7:v4

java-8 Dockerfile

FROM here/centos-7:v5Python Dockerfile


tomcat-7 Dockerfile

- FROM here/java-8:v12

+FROM here/java-8:v13

tomcat-8 Dockerfile

- FROM here/java-8:v12

+FROM here/java-8:v13

flask Dockerfile

- FROM here/python:v8

+FROM here/python:v9

How a Docker container get versioned

Each master verify build results in a Docker image that is tagged as

testing.

testing-$symanticVersion-$timestamp-$gitrev

testing-1.6.5-1477661204-d871ef9

This results in a clean natural order and all tags can be traced back to

their origins in code or build.

Once a Docker image passes all acceptance tests, it gets retagged

stable.

stable-1.6.5-1477661204-d871ef9

Managing Service releases at

scale

How Docker containers get promoted

Collections of images that pass acceptance testing together get released

by pushing a bill of materials to an artifact repository.

ZOOKEEPER=here/zookeeper:stable-1.6.5-1477661204-d871ef9

KAFKA=here/kafka:stable-0.10.1.0-1476105328-79c3c0c

STORM=here/storm:stable-1.0.2-1476193847-7fe4563

Now this BOM can be used by any other environment based on its own

specific configs stored in Consul and Vault.

Releasing Services through CI/CD pipelines

HERE separates CI into distinct stages; all development happens on the

master.

Pre-submit

verify

Master

verify

Product

acceptance

test

System

acceptance

test

Customer

integration

test

Production

Some statistics on speed of deployment

• ECS has taken one of the slowest parts of our CI/CD pipeline and

made it one of the fastest.

ECS

12 to 15 minutes

95% success rate30 to 60 seconds

99%+ success rate

Thank you!

HERE makes it quick and easy to enable location based

solutions

Visit Booth 2316

Geocoding – New on the

AWS Marketplace

With HERE Geocoder, plan to

to add geocoding services to

to your app. Each plan allows

allows you to use its features

features for a single app or

website. To use HERE

platform for more than one

app or website, simply

register for an additional plan

plan for each additional app

app or website.

REST APIs

Add geocoding, positioning,

routing and more to any

connected device. The HERE

Location Platform brings

advanced features to low-power

hardware through a set of flexible

services accessible via REST APIs.

Positioning via REST APIs

Find the location of your apps and hardware

around the work, indoors and outdoors.

HERE has network positioning with global

cell and Wi-Fi coverage, including China,

for low-cost location use-cases, and

advanced assistance features to increase

accuracy and decreased time-to-first-fix.

High accuracy network positioning allows

you to detect location even if your hardware

doesn’t support GNSS.

Up for a Challenge?

HERE Top 50 Coding Challenge

Let us prove how the HERE cloud makes developing location-based services

fast and easy. The first 50 developers to participate in this contest receive a

premium miniature drone giveaway at the HERE booth during Amazon

Re:invent. Stop by our booth to find out more!

Remember to complete

your evaluations!

aws re:invent 2016: service integration delivery and automation using amazon ecs (con308)

Technology