10 YEARS OF CLOUD10 YEARS OF CLOUDPIERRE-YVES RITSCHARD (PIERRE-YVES RITSCHARD ( ))@PYR@PYR

👋👋👋👋👋👋: Three-line Bio

CTO & Co-founder at Distributed systems and monitoring enthusiastOpen-Source developer

@pyr

Exoscale

10 YEARS OF CLOUD10 YEARS OF CLOUDBuilding better infrastructure with parentheses

EXOSCALEEXOSCALEInfrastructure as a serviceZones in Frankfurt, Vienna, Zürich, Geneva, Münich, Sofia

EXOSCALEEXOSCALE

EXOSCALEEXOSCALEprovider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" }

resource "exoscale_instance" "web" { template = "Ubuntu 21.04" disk_size = "50g" profile = "medium" ssh_key = "production" }

WHAT'S IN A CLOUD PROVIDERWHAT'S IN A CLOUD PROVIDERDatacenter operationsSo�ware development

SOFTWARE AT EXOSCALESOFTWARE AT EXOSCALEAPI GatewayOrchestrators (VM instance, Load-Balancer, Kubernetes)Object storage controllerNetwork controller (SDN)Customer managementMetering systemBillingWeb portal

ENGINEERING AT EXOSCALEENGINEERING AT EXOSCALE

ISN'T ALL OF THIS BASH, PERL, ANDISN'T ALL OF THIS BASH, PERL, ANDYAML?YAML?

ENGINEERING AT EXOSCALE: A TIMELINEENGINEERING AT EXOSCALE: A TIMELINE

2012: THE EARLY DAYS2012: THE EARLY DAYSWe started with

3 peopleA bit of timeA product idea

A DIFFERENT CLOUD PROVIDERA DIFFERENT CLOUD PROVIDERNot yet another virtual datacenter productIntegration with automation toolingIntegration in language-specific librariesFocus on horizontally-scalable applications

Local storageSecurity groups

THINGS THAT DIDN'T EXIST IN 2012THINGS THAT DIDN'T EXIST IN 2012AnsibleTerraformDocker

THINGS THAT DIDN'T EXIST IN 2012THINGS THAT DIDN'T EXIST IN 2012TelevisionWifi

OUR MINIMAL STACKOUR MINIMAL STACKApache CloudstackPuppetGood old MySQLA third-party customer management toolPython + AngularJSRiemann

OUR MINIMAL STACKOUR MINIMAL STACK

RIEMANNRIEMANNThe common saying back then was monitoring sucksPush-based model was a great fit for our use caseA great opportunity to contribute, as Riemann was in earlystages

2013: GOING LIVE2013: GOING LIVE

BACKEND DEVELOPERS DOINGBACKEND DEVELOPERS DOINGFRONTENDFRONTEND

THINGS OUR EARLY ADOPTERS ENJOYEDTHINGS OUR EARLY ADOPTERS ENJOYEDVagrant supportSecurity groups instead of firewallingA public IP per instance

IMPROVING RELEASE AUTOMATIONIMPROVING RELEASE AUTOMATION

WARPWARP

WARPWARP

WARPWARPOpen SourceTLS client certificate-based authenticationIRC supportHaskell Go agentPrefigured our inclination for Clojure at the orchestration layer

TROUBLE KICKS INTROUBLE KICKS INLate paymentsBitcoin mining on free credit

SOLVING ABUSESOLVING ABUSENeed to pull data from a bunch of placesStandard FSM type of problem

FUNCTIONAL PROGRAMMING TO THEFUNCTIONAL PROGRAMMING TO THERESCUE!RESCUE!

(match [state new-state unpaid-invoices?] [:ok :warning _ ] :warn! [:ok :critical _ ] :suspend! [:warning :critical _ ] :suspend! [:warning :ok _ ] :active! [:critical :ok false ] :active! [:critical :warning false ] :active! [_ _ _ ] nil)

2014: THE YEAR OF STORAGE2014: THE YEAR OF STORAGE

OBJECT STORAGEOBJECT STORAGEThe obvious choice for our crowdArchitecturally simpler than distributed block storageA good complement to our local storage backed instances

OBJECT STORAGE NEEDSOBJECT STORAGE NEEDSS3 is the sole player in that field: we need API compatibilityThe only alternative at the time was bad HTTP extensions

OBJECT STORAGE IN THE WILDOBJECT STORAGE IN THE WILDCephRiak-CSSwi�Costly vendor-backed solutions

WRITING AN OBJECT STOREWRITING AN OBJECT STOREWe focused on how to store large objectsTempted by a description of the (non-OpenSource) approach byDatastax on top of Cassandra

CHOOSING CASSANDRACHOOSING CASSANDRAGreat library supportSimple for us to operateVery few moving partsOur implementation could remain fully stateless

WE WERE (ALMOST) YOUNG AND (WAYWE WERE (ALMOST) YOUNG AND (WAYTOO) NAIVETOO) NAIVE

How are could it be?

WHAT WE DIDN'T ANTICIPATEWHAT WE DIDN'T ANTICIPATEIt's not all about actual data storage

The S3 API is a beastThe S3 API is under specifiedThe S3 API is not versionedThe S3 API client landscape is a mess

A QUICK DIGRESSION: S3 REQUESTSA QUICK DIGRESSION: S3 REQUESTSOperation: put object foo in bucket bar:

PUT /foo

Host bar.sos-ch-dk-2.exo.io

Authorization: AWS ....

<...>

A QUICK DIGRESSION: S3 REQUESTSA QUICK DIGRESSION: S3 REQUESTSOperation: update acl for object foo in bucket bar:

PUT /foo?acl

Host bar.sos-ch-dk-2.exo.io

Authorization: AWS ....

X-Amz-ACL: bucket-owner-full-control

A QUICK DIGRESSION: S3 REQUESTSA QUICK DIGRESSION: S3 REQUESTSOperation: Copy object bim from bucket bam to object foo in

bucket bar:PUT /foo

Host bar.sos-ch-dk-2.exo.io

Authorization: AWS ....

X-Amz-Copy-Source: /bim/bam

X-Amz-Copy-Source-If-Unmodified-Since: ARE YOU KIDDING ME?

BY THE WAYBY THE WAYStoring terrabytes of data on off-the-shelf hardware doesn'tcome by easy eitherInput and output payloads of arbitrary lengths aren't easyThe standard web stack doesn't cut it

2015: SCALING UP2015: SCALING UP

BILLING ISSUESBILLING ISSUESThe cron based approach to billing is showing its limitHard to keep it at a hourly rate because it takes too long

AT A CROSSROADSAT A CROSSROADS

AT A CROSSROADSAT A CROSSROADS

AT A CROSSROADSAT A CROSSROADS

INTRODUCING STREAM PROCESSINGINTRODUCING STREAM PROCESSINGYou can't do everything with cron

OUR CANDIDATE: APACHE KAFKAOUR CANDIDATE: APACHE KAFKAPartition-isolated consistencyDisaggregating memory

WHY KAFKA?WHY KAFKA?

A FIRST CANDIDATE: BANDWIDTHA FIRST CANDIDATE: BANDWIDTHMETERINGMETERING

Traffic accounting on hypervisors, with a small C agent30 second aggregates sent over to KafkaA Clojure Kafka consumer on the other end

KEY TAKEWAYSKEY TAKEWAYSIncredible reliabilitySoon expanded to the rest of our billing infrastructureThe system can weather temporary failures with no billingimpact

2017: TOO MUCH DATA2017: TOO MUCH DATA

SUDDEN S3 PICKUP IN USAGESUDDEN S3 PICKUP IN USAGEOur initial implementation limits the throughputTail latencies go through the roofCassandra is just not great at doing dense nodes

We knew this going inWe hit the wall hard

WE NEED A NUMBER OF NEW APIWE NEED A NUMBER OF NEW APICAPABILITIESCAPABILITIES

V4 signatures are becoming the norm for S3Better ACL support is neededThe docker registry exercises all weird properties of the API

WE FIND A GOOD PAPERWE FIND A GOOD PAPERAmbry attacks the same problem spaceThe paper lays out a great strategy

LET'S WRITE A DISTRIBUTED SYSTEMLET'S WRITE A DISTRIBUTED SYSTEMFROM SCRATCHFROM SCRATCH

What could go wrong?

KEY DECISIONSKEY DECISIONSA storage agent in COrchestration in ClojureZookeeper for agent discoveryCassandra for metadata storage

UIUI

2018: SECURITY, API, AND KUBERNETES2018: SECURITY, API, AND KUBERNETES

SPECTRE AND MELTDOWNSPECTRE AND MELTDOWNEstablished crisis communication channels with other keyprovidersLarge scale automation of Linux Kernel roll-outs

BUILDING ON KUBERNETESBUILDING ON KUBERNETESWe previously bet on MesosTraction seemed to be stronger around KubernetesOther players were still relevant (notably Swarm)Need to gain knowledge before being ready to sell

LOOKING BACKLOOKING BACK

OPEN-SOURCE AT EXOSCALEOPEN-SOURCE AT EXOSCALEApache Cloudstack was a great way to bootstrap our serviceHigh dependence on Open Source DatabasesSome key low level components (Qemu, Libvirt)

OPEN-SOURCE INVOLVEMENTOPEN-SOURCE INVOLVEMENTSeveral Apache developersKey contributions to major projects

THANKSTHANKSQuestions