erlang factory layered architecture - final
DESCRIPTION
TRANSCRIPT
![Page 2: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/2.jpg)
2
![Page 3: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/3.jpg)
A legacy of growth
![Page 4: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/4.jpg)
4
Social and casual gaming platform
50+ portals190+ countries
![Page 5: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/5.jpg)
Casual games
Social games
Multi player games
Native games
![Page 6: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/6.jpg)
0"
50"
100"
150"
200"
250"1/1/07"
5/1/07"
9/1/07"
1/1/08"
5/1/08"
9/1/08"
1/1/09"
5/1/09"
9/1/09"
1/1/10"
5/1/10"
9/1/10"
1/1/11"
5/1/11"
9/1/11"
1/1/12"
5/1/12"
0"
10"
20"
30"
40"
50"
60"
70"
80"
90"
1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
Traffic growth over 1600%
Tech dept. growth over 2000%
200+ million unique usersper month
![Page 7: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/7.jpg)
7
XMLInterfacing
Business
logic
SQL
Auth
oriz
atio
n +
Auth
entic
atio
n
Caching
Json Interfacing + plain GET
Businesslogic
Auth
oriz
atio
n
SQL
SwiftAPI
CachingCaching
MemcacheCluster
Mysql DB
Intefacing
Businesslogic
Caching
CachingBusiness
logic
Intefacing
Businesslogic
Schema
Caching
CachingBusiness
logic
Businesslogic
Mysql DBMysql DB
Mysql DB
Mysql Write DB
MysqlReader
MysqlReaders
SQL
SQL
SQL
SchemaSchemaSchema
SQL
SQL
![Page 8: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/8.jpg)
• Inconsistent design & interfaces• Unclear responsibili4es• Technical debt • Independent teams developing
the same features.• 7 layers of caching hell• Scaling
Problems
![Page 9: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/9.jpg)
• Inconsistent design & interfaces• Unclear responsibili4es• Technical debt • Independent teams developing
the same features.• 7 layers of caching hell• Scaling
Problems
not a problem
:)
![Page 10: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/10.jpg)
9
![Page 11: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/11.jpg)
A new approach
![Page 12: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/12.jpg)
11
• Flexible
• Predictable
• Scalable
New architecture expectations
Buzzword
CompliantTM
![Page 13: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/13.jpg)
Choices made
• Simple• Clean separa4on• No horizontal dependencies• Fault tolerant• Distributed
12
![Page 14: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/14.jpg)
13
• Caching• Persistence• Sharding
Layers Responsibilities
Interface
Service
Storage
Client • Integra4on• State
• Authen4ca4on• Aggrega4on• Valida4on
• Authoriza4on• Composi4on• Transforma4on
![Page 15: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/15.jpg)
13
• Caching• Persistence• Sharding
Layers Responsibilities
Interface
Service
Storage
ClientS
tateless• Integra4on• State
• Authen4ca4on• Aggrega4on• Valida4on
• Authoriza4on• Composi4on• Transforma4on
![Page 16: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/16.jpg)
14
Application• Small & Simple• Limited scope• Quick to deploy
Platform• Abstracts complexity• Low change rate• Long lifetime
Building blocks
![Page 17: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/17.jpg)
Walkthrough
![Page 18: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/18.jpg)
16
3rd PartyBackend
https post/user/get/
UserInterface
UserService Awards
Service
get(me)
1
1 FriendsService
1
count
UserService
2
get([friends])
{“auth”: {“token”: “UwAA_AWeff...”}}
Interfaces
list
ApplicationInterface
HighscoreService
1 list
2
/application/highscores/
![Page 19: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/19.jpg)
17
Widgets
![Page 20: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/20.jpg)
17
logo widget profile widget
search widget
navigation widget
recommendation widget
recent_games widget
new_games widget most_played widget
social_games widget
Widgets
![Page 21: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/21.jpg)
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+
![Page 22: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/22.jpg)
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+navigation
header
search
logo
![Page 23: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/23.jpg)
home_page
http://www.agame.com/
...
...preprocess
wid
get p
latfo
rm
body
footer
new games
...
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+navigation
header
search
logo
![Page 24: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/24.jpg)
Servicesuser
interface{“context”:{ “guid”: 62623423, “siteid” : 88, “locale” : “en_US”},”guid”: 324112535}
user service
profilestorage
check fetch filter modify
{ “gid”: “324112535”, “username” : “-fooba-”, “firstName” : “john” “avatarId” : 99142}
{ “gid”: “324112535”, “username” : “-fooba-”, “avatarId” : “http://agame.com/avatars/99142.png”}
get
![Page 25: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/25.jpg)
Storage platform
• API instead of SQL• Buckets of data = Key-‐Value with Schema• Abstracts storage
20
Shard 1 Shard 2 Shard ..
Storage Node 1
Bucket Bucket Bucket
Cache Storage Node 2
Bucket Bucket Bucket
Cache
![Page 26: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/26.jpg)
Job handlers
21
AccountInterface
UserService
Profilebucket
interface-account.register{“guid”:7734}
AuditHandler
WelcomeHandler
sendemail
Audit Logservice-user.register
{“result”:”success”,”guid”:”73324”}
rabbitmq
![Page 27: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/27.jpg)
Fitting it together
![Page 28: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/28.jpg)
23
Widget PlaGorm
Profilewidget
Gamewidget
Authin@erface
Gameservice
Awardinterface
Userinterface
Userservice
Awardservice
Storage PlaGorm
Gamesbucket
UserAwardbucket
Profilesbucket
Awardservice
Userservice
inte
rface
serv
ice
stor
age
Load balancer
GameAwardbucket
Profilesbucket
Protobuf
Protobuf
JSON
![Page 29: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/29.jpg)
24
• Strictly define EVERYTHING • input/output format, urls, • rpc func4ons, types, valida4on
• Strict rules on changes• Everything op4onal• Unless you’re really, really sure (and have proof!)
• Generate code and documenta4on• Interface derived from specifica4on• Documenta4on derived from specifica4on
How we do it
![Page 30: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/30.jpg)
25
• XML: Defini4on
• XSLT: Transforma4on
• Piqi: • Protobuf & Json encoding/decoding
• HTTP RPC(h`p://piqi.org/)
![Page 31: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/31.jpg)
26
• Small applica4ons running on their own nodes
• Scale up where necessary
• Many of them on the same machine
• Small cluster already contains hundreds of nodes
What we ended up with…
![Page 32: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/32.jpg)
27
• Connec4ng all nodes is a bad idea:• Not needed• Number of open connec4ons
• Solu4on: use hidden nodes• Each node specifies the nodes it needs.• How does it scale?
Keeping it connected
![Page 33: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/33.jpg)
28
Number of servers (~30 nodes / server)
Num
ber o
f TC
P co
nnec
tions
Hidden nodes vs. visible nodes
240$ 320$ 400$ 480$ 560$ 640$ 720$1770$7140$
16110$
28680$
44850$
64620$
87990$
114960$
145530$
0"
20000"
40000"
60000"
80000"
100000"
120000"
140000"
160000"
2" 4" 6" 8" 10" 12" 14" 16" 18"
Hidden" Visible"
![Page 34: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/34.jpg)
29
• Each applica4on has it’s own router
• Scans & Monitors
• Balances requests across nodes
• Parallelizes requests as much as possible
• Applica4on can only talk to what it has defined
Router
![Page 35: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/35.jpg)
30
• Use proper transports (Erlang RPC vs. HTTP)
• Yes there is some transport overhead
• Typically < 1ms per layer (+network latency)
• But each layer has predictable response 4mes
• As long as enough downward resources are available
But what about response times?
![Page 36: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/36.jpg)
31
The old ...
![Page 37: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/37.jpg)
32
.. and the new.
![Page 38: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/38.jpg)
Closing Words
![Page 39: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/39.jpg)
34
• Interface layer has been defined and largely implemented
• ImplemenMng the service & storage layer step by step• Meanwhile we have adapters in place
• Several of the new API’s are in use in producMon
• First portals on the new widget plaGorm will be deployed in the coming months
Where are we now..
![Page 40: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/40.jpg)
35
• Deploying to mul4ple datacenters this year
• Bring elas4city to our deployments
• Decommission our “old architecture”
• Erlang game plahorm
... And where we’re going
![Page 41: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/41.jpg)
36
• An architectural vision is crucial• Integra4ng into a mess is a mess• Layers with clear responsibili4es• Strict communica4on• Package small• Change is the only constant• Op4mize for predictability first
Lessons Learned
![Page 42: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/42.jpg)
Questions
37
![Page 43: Erlang factory layered architecture - final](https://reader033.vdocument.in/reader033/viewer/2022051209/54881aecb479590a0d8b5605/html5/thumbnails/43.jpg)
38
Thank you!