Gentoo Linux developerCTO at Numberly

@ultrabug

© Eric Fischer / Flickr

Designing a scalable and distributed application

EuroPython 2015

© Eric Fischer / Flickr

USA ⇄ EUROPE

Geo distributed page hit counter web application

© Highways England / Flickr

spans over multiple and distant datacenters

#1Multi Datacenter

© Catherine Holmes

sum of the page hits from all the datacenters

#2A unique page hit count

© Catherine Holmes

no manual application (re)configuration

#3Automatic resizing

© Catherine Holmes

immediately available to all web services

#4Distributed configuration

© Catherine Holmes

...that’s some kind of contract !

Hell...

© C. K. Koay / Flickr

be lazy !

Rely on your stack

© Tripp / Flickr

Zen of pythonif the technology is hard to explain,

it's a bad idea

© HumanOS

UNIX tools philosophykeep it small and simple

© Jason Scott / Flickr

Isolated componentsbreaking it down to pieces

© Blake Thornberry / Flickr

distributed architecture style

Microservices

© Martin Grandjean / wikipedia

micromanagement = high operational cost

Microservices

© Martin Grandjean / wikipedia

Communication reliabilitynetwork latency and failure

© Steven Shorrock / Flickr

Locality vs consistencydata access lag or outage

© R2 HOX / Flickr

feature rich application stack

nginx + uWSGI

© Terabass / wikipedia

⇨ gets HTTP requests ⇨ produces jobs

⇦ returns the total hit count

collector

consumes jobs ⇨ increments a counter

processor

simple and reliable job queue server

beanstalkd

© Mooms

automated scaling and fault tolerance

Service discovery

© Cambridge University / wikipedia

multi datacenter + key / value storage

Consul

seamless service registration / health check

uWSGI consul plugin

consul-register = url=http://localhost:8500,name=my_service

to victory

Let’s build this up

© Brian Rincker / Flickr

(hopefully)

Live demo !

© shoutmeloud / Pinterest

source code : github.com/ultrabug/ep2015@ultrabug

Thanks

© Eric Fischer / Flickr