puppetconf 2016: scaling puppet on aws ecs with terraform and docker – maxime visonneau, trainline

Puppet at Scaleusing Terraform & ECS

2

Maxime VISONNEAUPrincipal DevOps Engineer

@mvisonneau

3

The story behind this talk..

Puppet at Scale using Terraform & ECS 4

Agenda

Terraform and ECS?

Why?

How?

Beyond!

5

TERRAFORM Infrastructure as a Code

Execution plans

Resource graph

Change automation

Open Source

Strong community

Enterprise support

https://github.com/hashicorp/terraform

Puppet at Scale using Terraform & ECS 6

Alternatives to Terraform

CloudFormation

Tied to AWS Not made for that

7

ECS Container management service

Highly scalable

Highly available

Managed through AWS SDK/API/CLI

Maintained by AWS

Puppet at Scale using Terraform & ECS 8

Alternatives to ECS

GO AHEAD!

9

Why?

10

Use case

Client / Server architecture

Certificate Authority

Server workloads

PuppetDB for reporting, with a human friendly interface on top of it

Lets keep it opensource for now

Puppet at Scale using Terraform & ECS

11

Why not masterless ?

12

So why then ?

13

Because we can!

14

The love of the sport!

15

SPEED!

16

ECS EC20

100

200

300

400

500

600

seco

nds

Puppet at Scale using Terraform & ECS 17

Pros Speed of scale

Scalability Reliability / Availability Additionnal level of abstraction

Enhanced compute resources sharing

No chicken/egg issue at Puppet is not configuring itself anymore

Cons Complexity

Operability

The TCO will depend on where you currently are regarding your

implementation of containers at scale.

18

How?

19

First things first!

20

The Puppet Code

Puppet at Scale using Terraform & ECS

21

One step at a time!

22

The Certificate Authority

Puppet at Scale using Terraform & ECS

23

The Puppet Server

Puppet at Scale using Terraform & ECS

24

The Puppet DB

Puppet at Scale using Terraform & ECS

25

The puppetboard

Puppet at Scale using Terraform & ECS

PuppetDB

26

Assembling the pieces together

27

28

And there it is..

29

SHOWTIME!(time to pray the demo gods)

30

31

Room for improvement(a big one!)

Puppet at Scale using Terraform & ECS 32

How could we make it better ?

Service discovery

Move away from load balancers and enhance portability

Use a unikernel

Improve immutability of the

base image we use

Monitoring

What we don’t monitor never

fails

Abstract data persistence

Will enable the portability of the

solution onto another provider

Change scheduler

Improve operability and be cloud

provider agnostic

Puppet at Scale using Terraform & ECS 33

How could we make it better ?

34

That’s it!

35

Time for questions(and answers if possible..!)