![Page 1: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/1.jpg)
Die LMAX-Architecture with Disruptors: 6M Transactions per Second
Stephan Schmidt, Vice CTO, brands4friends
![Page 3: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/3.jpg)
3
![Page 4: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/4.jpg)
![Page 5: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/5.jpg)
brands4friends No.1 Shopping Club in Germany > 360k daily visitors > 4.5M Users eBay company
20.04.12 5 WJAX 2011
![Page 6: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/6.jpg)
6
![Page 7: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/7.jpg)
7
![Page 8: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/8.jpg)
Development at brands4friends Team Java and web developers, data warehouse developers Process Scrum since 2009 Kanban for DWH since 2012
![Page 9: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/9.jpg)
LMAX - The London Multi-Asset Exchange
20.04.12 Fußzeilentext 9
"We aim to build the highest performance financial exchange in the world"
![Page 10: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/10.jpg)
High Performance Transaction Processing
20.04.12 10 Fußzeilentext
![Page 11: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/11.jpg)
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
![Page 12: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/12.jpg)
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
Queue
Queue
Queue
Queue
Queue
Queue
![Page 13: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/13.jpg)
Ghz CPU
Cores
![Page 14: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/14.jpg)
20.04.12 Fußzeilentext 14
Actors? SEDA?
![Page 15: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/15.jpg)
Stuff that did not work for various reasons
20.04.12 Fußzeilentext 15
1. RDBMS
2. Actors
3. SEDA
4. J2EE …
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
Queue
Queue
Queue
Queue
Queue
Queue
![Page 16: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/16.jpg)
LMAX Architecture
20.04.12 16 Fußzeilentext
![Page 17: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/17.jpg)
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
Queue
Queue
Queue
Queue
Queue
Queue
![Page 18: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/18.jpg)
Size
Node Node Node Node
Linked List Queue
Add Remove
Array Queue
Cache Line Cache Line
AddRemove
![Page 19: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/19.jpg)
Queue as a data structure Problems with Queues
19
1. Reading (Take) and Writing (Add) are both write access => Write Contention
2. Write Contention solves with Locks 1. Other solutions include Deques
3. Locks lead to context switches to the kernel 1. Context switches lead to CPU cache misses etc.
2. Kernel might use opportunity to do other stuff as well
![Page 20: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/20.jpg)
Locks Costs according to LMAX Paper
20
Method Time in ms Single Thread 300 Single Thread mit Lock 10.000 Zwei Threads mit Lock 224.000 Single Thread mit CAS 5.700 Zwei Threads mit CAS 30.000 Single Thread/ Volatile Write
4.700
“Compare And Swap” Atomic Reference etc. in Java => No Context Switch Memory Read/Write Barrier
![Page 21: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/21.jpg)
LMAX Data Structure – Ring Buffer
21
Ring Buffer
Publisher Event Processor
![Page 22: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/22.jpg)
Pre-Allocation of Buckets
22
Ring Buffer
31
24
1918
Publisher
30 29 28272625
23
22
21
20
17161514131211
109
87
6
5
432
10
Event Processor
2^5• No (less) GC problems • Objects are near each other in memory
=> cache friendly
![Page 23: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/23.jpg)
Coordination
23
Ring Buffer
31
24
1918
Publisher
30 29 28272625
23
22
21
20
17161514131211
109
87
6
5
432
10
Event Processor
2^5
Claim Strategy
1.Claim 2.Write 3.Make Public by advancing sequence
Wait Strategy
![Page 24: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/24.jpg)
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
Queue
Queue
Queue
Queue
Queue
Queue
Latency
![Page 25: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/25.jpg)
Receive Message
Journal
Replicate
Unmarshall
Business Logic
![Page 26: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/26.jpg)
Service / Transaction Processor
Receive Unmarshal ReplicateJournal Business Logic Marshall Send
Datenstruktur
Datenstruktur
![Page 27: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/27.jpg)
Ouput DisruptorOuput DisruptorInput Disruptor Ouput Disruptor
Business Logic Handler
LMAX Architektur
![Page 28: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/28.jpg)
28
Input Disruptor
Receiver
Journaler
Replicator
Un-Marshaller
Business Logic Handler
Output Disruptor
Publisher
Marshaller
HA Node
File System
Jede Stage kann mehrere Threads haben
![Page 29: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/29.jpg)
29
31
24
1918
Receiver
Journaler
Replicator
Business Logic Handler
Receiver writes on 31. Journaler and Replicator read on 24 and can move up the sequence to 30.
Business Logic Handler needs to stay behind all others.
Un-Marshaller can move beyond Journaler and Replicator up to 30.
Un-Marshaller
![Page 30: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/30.jpg)
Java API
20.04.12 30 Fußzeilentext
![Page 31: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/31.jpg)
![Page 32: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/32.jpg)
![Page 33: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/33.jpg)
P1
C1
C2
C3
C4
![Page 34: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/34.jpg)
C1P1
C2
C3
C4
![Page 35: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/35.jpg)
P1
C1 C2
C3 C4
![Page 36: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/36.jpg)
C1
C2
C3
C4P1
![Page 37: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/37.jpg)
C1
P1
P2
![Page 38: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/38.jpg)
20.04.12 Fußzeilentext 38
Demo
![Page 39: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/39.jpg)
LMAX Low Level Ideas
20.04.12 Fußzeilentext 39
1. Simple Code
2. Everything in memory
3. Single threaded per CPU for business logic
4. Business logic has no I/O, I/O is done somewhere else
5. Scheduler “knows” dependencies of handlers
![Page 40: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/40.jpg)
6M TPS? How did LMAX do it?
40
10K+ TPS
If you don't do anything stupid
3 billions of instructions on modern CPU
100K+ TPS
Clean organized code
Standard libraries
1000K+ TPS
Custom, cache friendly collections
Performance Testing
Controlled GC
Very well modeled domain
x 10
x 10
![Page 41: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/41.jpg)
We’re looking for very good developers
![Page 42: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/42.jpg)
Thanks! @codemonkeyism [email protected]
![Page 43: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/43.jpg)
43
Images CC from Flickr: nimboo, imjustcreative, gremionis, justonlysteve, John_Scone, Matthias Wicke, irisgodd3ss, TunnelBug, alandd, seasonal wanderer, raulbarraltamayo, Gilmoth, Dunechaser, graftedno1
![Page 44: Die LMAX-Architecture with Disruptors: 6M Transactions per ...donatas/Vadovavimas/Temos... · Costs according to LMAX Paper 20 Method Time in ms Single Thread 300 Single Thread mit](https://reader033.vdocument.in/reader033/viewer/2022042215/5ebb8757e79e7b5c8a758533/html5/thumbnails/44.jpg)
Sources
20.04.12 Fußzeilentext 44
“Disruptor: High performance alternative to bounded queues for exchanging data between concurrent threads”, Martin Thompson, Dave Farley, Michael Barker, Patricia Gee, Andrew Stewart, 2011
"The LMAX Architecture”, Martin Fowler, 2011
http://martinfowler.com/articles/lmax.html
“How to do 100K+ TPS at less than 1ms latency”, Martin Thompson, Michael Barker, 2010