Download - Adam Cataldo Tuesday, January 24, 2012
![Page 1: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/1.jpg)
EngineeringEngineeringv
Adam CataldoTuesday, January 24, 2012
Quick DeployA distributed systems approach to developer productivity
![Page 2: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/2.jpg)
Engineering 2
Outline
LinkedIn architecture Quick Deploy Cool technical details
![Page 3: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/3.jpg)
Engineering 3
LinkedIn Largest professional networking site
135M+ members 14 languages 200+ countries and
territories Estimated 4B+ people
searches per year
![Page 4: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/4.jpg)
Engineering 4
LinkedIn stack at 20,000 feet
Online system for serving up data to clients– Profile pages, company pages, LinkedIn groups, etc.– Key metric: Latency
Offline system for creating insight from data– People you may know, contact degree of separation,
search indexing, etc.– Key metric: Throughput
![Page 5: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/5.jpg)
Engineering 5
Offline data processingHadoopVoldemort
Map
Map
Map
Map
Reduce
Reduce
Reduce
![Page 6: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/6.jpg)
Engineering 6
Online web applicationClient Presentation Business Logic Data Service
Datastore
![Page 7: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/7.jpg)
Engineering 7
Growing Pains for the Online SystemAs time has gone by:
– We’ve added new services to support new features
– We’ve added new services to make our site faster and more available
– Our engineering productivity dropped
![Page 8: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/8.jpg)
Engineering 8
Quick Deploy
Developers only locally deploy the services they actually modify
Every service your service depends on is available in some data center
This makes development move much faster!
AngelikaClayton
![Page 9: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/9.jpg)
Engineering 9
Early Integration DatacenterDev Box
QD Proxy
Browser
Local Mid Tier
Shared Frontend
Shared Data Service
DB
![Page 10: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/10.jpg)
Engineering 10
In DetailBrowser Proxy
EIFrontend
Dev-localMid Tier
EIData Service
QD CookieQD Cookie
QD IC
QD IC
QD IC
QD IC
![Page 11: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/11.jpg)
Engineering 11
Proxy Behavior
Proxy receives request
Has QD cookie?
Route to developer’s local service
Route to service in EI
Read identifier
from cookie
Read identifier from IC
Is service local?
no
yes
yes
no
![Page 12: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/12.jpg)
Engineering 12
QD Cookie and Invocation Context
A developer creates an identifier key, and registers his local services with the proxy
He identifies himself by adding his key to a cookie from his browser
When a front end gets a request with a QD cookie, it puts the key in the invocation context
Services pass the invocation context with every outbound request
![Page 13: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/13.jpg)
Engineering 13
Shared data
OracleMySQL
VoldemortMemcached
Change Alfred’slanguage to
Japaneseきんぎょ
Huh?
![Page 14: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/14.jpg)
Engineering 14
ActiveMQEarly Integration DatacenterDev Box
Producer
Consumer
ConsumerProducesmessage Consumes
message
QueueNever seesmessage
![Page 15: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/15.jpg)
Engineering 15
TimersEarly Integration DatacenterDev Box
Service withTimer
Service
RPC Target
RPC TargetSchedulestime-triggered
RPC call Gets RPC call aftertimer triggers
Never gets RPC call
![Page 16: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/16.jpg)
Engineering 16
Proxy in detail
Admin ConsolePython/Flask Web App
ProxyApache Traffic Server
> 10,000 QPS!
Routing ProfilesApache Zookeeper
ProxyApache Traffic Server
ProxyApache Traffic Server
![Page 17: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/17.jpg)
Engineering 17
Why Apache Zookeeper
Distributed file system for small files Basically, a key-value data store, where the keys
are file names Clients can listen for file change and directory
change events
Great for storing data Also, great publish/subscribe system
![Page 18: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/18.jpg)
Engineering 18
Dynamic DiscoveryEarly Integration DatacenterDev Box
QD ProxyBrowser
Local Mid Tier
Shared Frontend
Shared Data Service DB
![Page 19: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/19.jpg)
Engineering 19
Conclusion
The LinkedIn architecture scales well in production
Quick deploy makes the architecture scale for developers
20% productivity gain across the engineering organization
![Page 20: Adam Cataldo Tuesday, January 24, 2012](https://reader036.vdocument.in/reader036/viewer/2022062501/5681685e550346895ddea082/html5/thumbnails/20.jpg)
Engineering 20
Thanks
RobLoh
SweeLim
SteveIhde
JimDumont