![Page 1: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/1.jpg)
Erlang as a cloud citizenPaolo Negri @hungryblank
![Page 2: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/2.jpg)
• 10 millions players
• 2 billions game server requests (http)
• 20 devops people
1 day at
![Page 3: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/3.jpg)
Cloud
“A cloud is made of billows upon billows upon billows that look like clouds.
As you come closer to a cloud you don't get something smooth, but irregularities at a smaller scale.”
Benoît B. Mandelbrot
http://www.flickr.com/photos/nirak/644336486
![Page 4: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/4.jpg)
AWS Cloud
![Page 5: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/5.jpg)
This talk will answer
• Why building a system targeting the cloud?
• How many EC2 instances do you need to respond to 0.25 billion uncacheable game reqs/day?
![Page 6: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/6.jpg)
15 months ago...
http://www.flickr.com/photos/wheatfields/515068701
![Page 7: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/7.jpg)
1st cloud hosted project, lessons learned
Pushing live the 60th application server?
not different from adding the 6th!
push a button1
![Page 8: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/8.jpg)
1st cloud hosted project, lessons learned
local network/local disk are low performance general purpose tools
(nothing to do with ad hoc data center solutions)2
![Page 9: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/9.jpg)
1st cloud hosted project, lessons learned
Complete automation is cool
Ease of adding hosts/automation can lead to bloated infrastructure3
![Page 10: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/10.jpg)
1st cloud hosted project, points of pain
• A lot of inefficient app servers (as per tweets)
• Much effort to scale up/maintain databases (mySQL & Redis)
• Expensive, not crazy expensive, but expensive
![Page 11: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/11.jpg)
Why trying again?
http://www.flickr.com/photos/kky/704056791/
![Page 12: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/12.jpg)
Uncertainity
• will we reach100K or 3 millions users?
• 3 million users in 2 weeks or 12 months?
• cheat tool released 1h ago => single game call up 5000%
• weekly releases, new feature performance impact?
![Page 13: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/13.jpg)
the cloud
• standard units (instances) of computing capacity
• a network connecting all instances
• an API to provision/dismiss instances
![Page 14: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/14.jpg)
the cloud
Sounds like a good framework to compose computing capacity
Why didn’t work as a framework to compose throughput?
![Page 15: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/15.jpg)
Scaling in the cloudthe recipe
CLOUD: composable units of computing capacity
+DEVELOPER: turn a unit of computing capacity in
a unit of throughput
=composable throughput, a plan for scaling BIG
![Page 16: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/16.jpg)
turn a unit of computing capacity
in a unit of throughput
http://www.flickr.com/photos/pasukaru76
![Page 17: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/17.jpg)
Unit of throughput,where?
Database
App server
![Page 18: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/18.jpg)
Unit of throughput,joke?
Database
App serverApp server App server App server
Database
Cache
![Page 19: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/19.jpg)
Unit of throughput?
Database
App server
No unit!
![Page 20: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/20.jpg)
Unit of throughput?
Database
App server
Tightly coupled throughput?
![Page 21: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/21.jpg)
Unit of throughput?
Database
App server
Monolithic throughput?
![Page 22: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/22.jpg)
Monolithic throughput
![Page 23: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/23.jpg)
Monolithic throughput
• likes monolithic infrastructure!
• scales well vertically
• wants screaming fast stack (network, disks...)
• any performance glitch impacts the whole system
![Page 24: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/24.jpg)
Tightly coupled throughput+
loosely coupled hardware(like cloud)
=frustration
![Page 25: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/25.jpg)
Who leads the tightly coupled dance?
Database
App server
![Page 26: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/26.jpg)
Who leads the tightly coupled dance?
Database
App server
The stateless application server!
![Page 27: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/27.jpg)
Stateless application servers guarantee
one thing...
which?
![Page 28: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/28.jpg)
Data is neverwhere you need it
![Page 29: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/29.jpg)
And another one...
![Page 30: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/30.jpg)
If you can feed them data fast enough...
they’ll choke on garbage collection
![Page 31: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/31.jpg)
We measure memcacheHIT / MISS
why app servers need to be 100% MISS?
![Page 32: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/32.jpg)
Where’s the best knowledge about hot/
cold data?
![Page 33: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/33.jpg)
Even the reverse makes more sense
Database
App server
1. pick your data up
2. go in the stateless app server
![Page 34: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/34.jpg)
WhatWentWrong?
![Page 35: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/35.jpg)
He can tell you!
• Rich Hickey
• Clojure author
“...If not in Erlang which I think has a complete story for how they do state”[1][1] Value Identity State @0.27
http://goo.gl/Zdjv0
http://www.flickr.com/photos/ghoseb/5120173586
![Page 36: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/36.jpg)
Most languages and runtimes don’t have a safe solution for concurrent, long lived state
Erlang stands out as an exception in this panorama
![Page 37: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/37.jpg)
Erlang...
Processes are the primary means to structure an Erlang
application.
wikipedia
![Page 38: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/38.jpg)
Erlang + OTPGeneric Server Behaviour
A generic server process (gen_server) implemented
using this module...
otp documentation
![Page 39: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/39.jpg)
Erlang + OTPGeneric Server Behaviour
handle_call(_Request, _From, State) -> {reply, ignored, State}.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVsn, State, _Extra) -> {ok, State}.
![Page 40: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/40.jpg)
gen_server
• An erlang process
• With LOCAL state
• responding to requests from clients
A unit of throughput!
![Page 41: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/41.jpg)
Composing throughput with erlang
EC2 instancegen_server
+
![Page 42: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/42.jpg)
Composing throughput with erlang
EC2 instance
1 EC2 instance + 1 erlang VM=
N kilo gen_servers (N kilo units of throughput)
![Page 43: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/43.jpg)
Composing throughput with erlang
EC2 instance
1 EC2 instance + 1 erlang VM=
N kilo gen_servers (N kilo units of throughput)
![Page 44: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/44.jpg)
Composing throughput with erlang
EC2 instance
1 EC2 instance + 1 erlang VM=
N kilo gen_servers (N kilo units of throughput)
![Page 45: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/45.jpg)
Composing throughput with erlang
EC2 instance
1 EC2 instance + 1 erlang VM=
N kilo gen_servers (N kilo units of throughput)
![Page 46: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/46.jpg)
Composing throughput with erlang
EC2 instance
1 EC2 instance + 1 erlang VM=
N kilo gen_servers (N kilo units of throughput)
![Page 47: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/47.jpg)
Scale by adding units instances
![Page 48: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/48.jpg)
Scale up adding units instances
![Page 49: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/49.jpg)
Erlang distribution
![Page 50: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/50.jpg)
Throughput complexity
• Losely coupled peers• Independent throughput
VS.
• Tightly coupled roles• Dependent throughput
![Page 51: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/51.jpg)
Where does the state come from?
gen_serverDatabase
Start
Stop
![Page 52: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/52.jpg)
Now with database
AWSS3
![Page 53: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/53.jpg)
Database scalability
AWSS3
DB is (almost) never on latency critical path
No need for low latency DB
![Page 54: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/54.jpg)
Database scalability
AWSS3
Throughput required is low
We can approximate S3 capacity as infinite
![Page 55: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/55.jpg)
Database scalability
AWSS3
Ubiquitous and uniform from application servers point of view.
![Page 56: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/56.jpg)
Remember?
AWSS3
![Page 57: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/57.jpg)
How it actually works
Data from the S3 is uniformly
available to any ec2 instance
![Page 58: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/58.jpg)
And as you zoom in...
![Page 59: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/59.jpg)
And zoom in...
![Page 60: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/60.jpg)
And zoom in you see...
EC2 instance
![Page 61: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/61.jpg)
Always the samekind of structure
A fractal approach to throughput
![Page 62: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/62.jpg)
Fractal
“A cauliflower shows how an object can be made of many parts, each of which is like a whole, but smaller.”
Benoît B. Mandelbrot
http://www.flickr.com/photos/paulobrabo/3588387063
![Page 63: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/63.jpg)
Homework
The exact same solution might not work for you...
but look for that unit of throughput
![Page 64: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/64.jpg)
You needXXXX
Smallish instancesto serve 0.25
billions uncacheable reqs/day
Answer time
![Page 65: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/65.jpg)
You need0XXX
Smallish instancesto serve 0.25
billions uncacheable reqs/day
Answer time
![Page 66: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/66.jpg)
You need00XX
Smallish instancesto serve 0.25
billions uncacheable reqs/day
Answer time
![Page 67: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/67.jpg)
You need0012
Smallish instancesto serve 0.25
billions uncacheable reqs/day
Answer time
![Page 68: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/68.jpg)
What’s1200 ?
![Page 69: Erlang as a cloud citizen, a fractal approach to throughput](https://reader034.vdocument.in/reader034/viewer/2022051513/5479e42a5806b58f048b4740/html5/thumbnails/69.jpg)
Thanks
Paolo Negri @hungryblank
http://www.wooga.com/jobs