it automation: make the server great again - toulouse devops fev 2017

IT AUTOMATION :

MAKE THE SERVER

GREAT AGAIN

@WAXZCE

QUENTIN ADAM AT TOULOUSE DEVOPS

Quentin ADAM from the Clever Cloud

@waxzce on twitter – github- soundcloud – instagram ….

WHO AM I ?

MY DAY TO DAY WORK :

CLEVER CLOUD,

THE IT AUTOMATION COMPANY

KEEP YOUR APPS ONLINE. MADE WITH

NODE.JS, SCALA, JAVA, RUBY, PHP,

PYTHON, GO…

Cloud & on premise ;-)

AND LEARN A LOT OF THINGS ABOUT

YOUR CODE, APPS, AND GOOD/BAD

DESIGN…

GIVE BACK TO THE COMMUNITY

NEVER

GONNA

LET YOU

DOWN

clever-cloud.com

LET’S TALK ABOUT

INFRASTRUCTURE

40 times in 5 years price drop of AWS

SERVERS COSTS ARE GOING DOWN

Google I/O 2015

“RACE TO ZERO”

I.E THE SERVER IS FREE

In fact, all you have is…

COOL.

YOU HAVE A SERVER.

HOSTING

Backup Updates Certification

Agreement Process Provisioning

SecurityHardware

maintenance

VALUE

= SERVER

MANAGEMENT

ONCE UPON A TIME, MY BELOVED

SERVERS

THEN, VIRTUALIZED ONES

CREATE HISTORY

DOCUMENTATION

SPECIFIC

TAYLOR MADE

NOT ABLE TO SCALE

KNOWLEDGE

!=

AUTOMATION

Automate

SPLIT COMPLEX PROBLEMS INTO A

COLLECTION OF SMALL ONES

POSSIBLE SERVERS

STATES

…

Updated to V43

New configuration

files

Install new service

History …

AVOID HISTORY

FIX STATE OF

THE DATA TO

ENSURE

ACCESSIBILITY

POSSIBLE STATE IN

AN IMMUTABLE STATE

working

Not working

Very simple to predicate

FIX THE

HISTORY IN A

IMMUTABLE

STATE

BTW,

REAL WORLD

INSIGHT

“WE LIVE IN A MUTABLE WORLD, DEAL WITH IT”

WORLD IS

SUCCESSION

OF

IMMUTABLE

INSTANT

EPHEMERAL

IMMUTABLE

INSTANCES

PET

!=

CATTLE

SERVICE

UPTIME

!=

SERVER

UPTIME

NO MORE OLYMPIC CONTEST ON

UPTIME

And why it’s a problem to manage technical product and decision with book keepers point of view https://www.youtube.com/watch?v=0ip1FoBsLB4

(AWS PRICING HYPOCRISY LEAD TO

BAD PRACTICE)

Summon Yog-Sothoth

HUMANLESS

POWERTOOLS

PRODUCTION HAS TO BE PERFECT

PEOPLE ARE NOT PERFECT

PEOPLE HAVE NOTHING TO DO ON

PRODUCTION STUFF

AT CLEVER CLOUD SSH CONNECTION

IS A RED ALERT

CAPACITY PLANNING

SPLIT PROCESS AND

STORAGE

Storage

• Databases

• Files

• Sessions

• Events

• …

Code

• Can be replicated

• Stateless

• Process

RUNTIME INSTANCES

VM, docker, kubernetes, puppet, ansible, vmware… This is not the problem

TECHNOLOGIES?

Search youtube quentin adam docker

DOCKER

THERE IS ONE PROCESS:

BUILD

THEN, REGISTER

AND LIVE WITHOUT DATA

AND DIE WHEN MORE

UPDATED VERSION IS THERE

TRASH OLD VERSION

FULLY AUTOMATED HOSTING FACTORY

AN INSTANCE IS DEAD OR ALIVE AND

THAT’S ALL

APP FAIL?

REDEPLOY

NEW

INSTANCE

OPERATION

SYSTEM

UPDATE ?

REDEPLOY

NEW

INSTANCE

REBALANCE

HARDWARE

ALLOCATION?

REDEPLOY

NEW

INSTANCE

PREDICABLE

FAIL OF THE

APPLICATION?

REDEPLOY

NEW

INSTANCE

Picking one instance or another doesn’t matter

STATELESSNESS IS THE KEY

BECAUSE YOU WILL NOT RELY

ON A SERVER ANYMORE

CREATE A FACTORY OF INSTANCE

PROXY

AND

ROUTING

Consul, etcd, Clever Cloud service dependencies

SERVICE DISCOVERY

CONFIGURATION

=> ENVIRONMENT VARIABLE

HOW TO MANAGE DATA?

SPLIT DATA AND PROCESS:

FS COMPOSITION

Runtime

• local

• replicable

• updated

Data

• file system

• block storage

• replicated

• copy on write?

DATA

STORAGE

WAYS

Not part of this list

DATABASE (ACID OR NOT, SQL OR NOT)

FILE SYSTEM

BLOC STORAGE

OBJECT STORAGE

(COPY ON WRITE)

(COPY ON WRITE)

DATA MANAGEMENT IS BIND

TO DB NEEDS

ACID :

DUPLICATE & DATA EVOLUTION

INSTANCE FOR ONE

ORGANIZATION

ACID

• Atomicity

• Consistency

• Isolation

• Durability

Powerful data

management

• Transaction

• User management

• One above one

Take advantage of ACID

database

BUT IT DESIGNS A

BOTTLENECK

EXAMPLE : E-SHOP ON

CLASSIC MODE

EXAMPLE : E-SHOP ON

CLASSIC MODE

User A buy a

hdd

Database Transaction :

• Stock management

• Order management

• Invoice generation

• Customer Account reward

• …

Transaction

user A is

processed

Stock &

Order are just

perfectly

synchronize

EXAMPLE : E-SHOP ON

CLASSIC MODE

User A buy a

hdd

Database Transaction :

• Stock management

• Order management

• Invoice generation

• Customer Account reward

• …

Transaction

user A is

processed

Stock &

Order are just

perfectly

synchronize

User B buy a

hdd Transaction

user B is

processed

then

EXAMPLE : E-SHOP ON

MULTI-TENANT MODE

i.e. : Multiple shop of various sellers on the same instance

EXAMPLE : E-SHOP ON

MULTI-TENANT MODE

User A buy a

hdd on seller A

Database Transaction :

• Stock management

• Order management

• Invoice generation

• Customer Account reward

• …

Transaction

user A is

processed

Stock &

Order are just

perfectly

synchronize

User B buy a

book on seller B Transaction

user B is

processed

then

HIGH AVAILABILITY:

DISTRIBUTED SYSTEM

DISTRIBUTED

DB AND

HARDWARE

DAFUCKNESS

FORGET THE SAN, GO FULL

DISTRIBUTED

NO SILVER

BULLET

STORAGE

BE CONSISTENT AND PREDICTABLE

MONITORING

IS

MANDATORY

EXPOSE METRICS

IMMUTABILITY

AS A PATTERN

ANOTHER TALK FROM ME ;-)

FR version : https://www.youtube.com/watch?v=y1BR7Rh7FSY&t=15s

SECURITY

VOODOO

LIKE

SECURITY IS EVOLVING

https://www.clever-cloud.com/blog/company/2016/04/04/security-is-a-process/

https://www.clever-cloud.com/blog/guests/2015/06/16/the-end-of-the-fortress-metaphor/

FROM SERVER TO

SERVERLESS:

CODE

LOCKED?

F*CK 12

FACTORS?

GO BACK ON

APPLICATION

SERVER?

I’m on twitter : @waxzce

quentin.adam@clever-cloud.com

THANK YOU !

ANY QUESTIONS ?

Clever Cloud gift coupon:

toulouseDEVOPS17