faбrik - timesopen: sockets and streams - sept. 2012
TRANSCRIPT
NYT faбrik
TimesOpen – 12 September 2012
2
Who
Architect Infrastructure Group Previous
– US Army– Harvard– Tech companies– United Nations
3
What we’ll cover
A story faбrik overview Code and demo
4
Takeaways
Application Developer– Demand ‘events’ (no polling, no thanks, been there, done that)– Demand infrastructure that scales and you don’t have to worry about
Infrastructure Engineer– Decouple, flatten, simplify– Outsource complexity
5
The Story: Harvard U, 1986+
Bad: – No Internet (pre-web)– data locked in mainframe– large central clerical staff– monolithic central systems
Good:– Vision of an Information Utility– desire to innovate– lots of desktop computers (30,000)– email everywhere although over diverse networks and technologies
6
The Story: Harvard U, 1986+
Solution: – Relational database (decouple data from application)– Email backbone (decouple producers from consumers)– Event-driven desktop applications (flatten)– Identical code on mainframe (simplify)
Result:– Data warehouse unlocked (before the term was coined)– Central clerical staff functions upgraded/dispersed– Old central systems replaceable and, ultimately, replaced– Happy users!
7
faбrik – Why?
?
8
NYT Mission
Enhance society by creating, collecting and distributing high quality news, information and entertainment
- Distributing: publish / subscribe- Collecting: gather / analyze- High Quality: fast, reliable, accurate
9
faбrik
Asynchronous Messaging Framework
For client devices as well as our apps
Enabled by:– Websockets– Robust message handling software– Amazon Web Services
Focusing on simple, common services
10
faбrik – solve the problem!
App
11
faбrik – basic
App
App
App
Message Broker
12
faбrik – basic
App
App
App
Message Broker
Amazon Web Services
• EC2• S3• Identity &
Access Mgt• DynamoDB• Route 53…
13
faбrik – basic++
OtherApp
AppBuddy
Service Buddy
Message Broker
Message Broker
Service Buddy
“Retail”
“Wholesale”
14
faбrik: Current Implementation
Open source– Erlang/OTP– RabbitMQ– Nodejs – Sockjs (websockets +)– Python (gevent, select/epoll, tornado, futures, …)– ZeroMQ (new)
Automated build/deployment
15
faбrik – active/active cluster
Service Buddy
‘a’
Service Buddy
‘b’
Zone ‘a’ Zone ‘b’
Region Wherever
16
faбrik – active/active cluster
Service Buddy
‘a’
Service Buddy
‘b’
Zone ‘a’ Zone ‘b’
Region Wherever
Got your back!
Got yours too!
17
faбrik
18
faбrik
19
faбrik
20
faбrik – Layers
21
faбrik – Layers
22
faбrik – Layers
23
faбrik – Layers
24
25
26
Pub / Sub
27
Pub / Sub
28
Pub / Sub
29
Pub / Sub
30
Pub / Sub
31
Gather / Analyze
32
Gather / Analyze
33
Gather / Analyze
34
Gather / Analyze
35
Gather / Analyze
36
Gather / Analyze
37
Gather / Analyze
38
faбrik
Demo
Code
App