atmosphere conference 2015: pubsub++ - few tips that make your life with kafka easier
TRANSCRIPT
Krzysztof DębskiAtmosphere 2015
PubSub++or how to make life with Kafka easier
@DebskiChris Kraków, 19.05.2015, Atmosphere
Why do we need a PubSub?
@DebskiChris Kraków, 19.05.2015, Atmosphere
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Distributed environment
@DebskiChris Kraków, 19.05.2015, Atmosphere
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Domain
Domain
Domain
Distributed environment
@DebskiChris Kraków, 19.05.2015, Atmosphere
Orchestration vs. Choreography
@DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka
@DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka architecture
Service
Producer
Service
Consumer
KafkaBroker
Zookeeper< Kafka 0.9
@DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka data1
0 9 78 6 5 34 2 1
8 5 4 2 1
10 9 7 6 3
5 4 2
8 1
Data Topic Partition
@DebskiChris Kraków, 19.05.2015, Atmosphere
Consumer groups
Consumer 1
Broker
P0
Broker
P1
Broker
P2
Broker
P3
Consumer 2
Consumer 3
Consumer 4
Consumer 5
Consumer 6
Consumer group 1 Consumer group 2
@DebskiChris Kraków, 19.05.2015, Atmosphere
Publish & subscribe
Producer_1…Producer_n
Consumer_group_1Consumer_group_2
Remove old messages
Remove old events
Publish event
Read eventRead event
@DebskiChris Kraków, 19.05.2015, Atmosphere
Leader & Replicas
Service
Producer
Service
Consumer
Broker
Zookeeper
Broker
Broker
P1 P0
P2 P1
P0 P2
@DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leadersBroker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2, 3
@DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leadersBroker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2, 3
@DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leadersBroker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 1Replicas: 3, 1 ISR: 1Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2
@DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leadersBroker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 1Replicas: 3, 1 ISR: 1, 3Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2, 3
@DebskiChris Kraków, 19.05.2015, Atmosphere
Acknowledges
0 - don’t wait for response from broker
1 - only leader needs to respond
-1 - all replicas in sync
Spee
d
Safety
@DebskiChris Kraków, 19.05.2015, Atmosphere
Events security
Basically, there is no authentication
You can create any topic
You can publish everywhere
You can subscribe to any topic
@DebskiChris Kraków, 19.05.2015, Atmosphere
@DebskiChris Kraków, 19.05.2015, Atmosphere
Synchronous world
RESTService
AService
B
@DebskiChris Kraków, 19.05.2015, Atmosphere
Asynchronous made simple
Service A
Service B
REST REST/JMS
@DebskiChris Kraków, 19.05.2015, Atmosphere
Hermes
Hermes Frontend
Hermes Frontend
Hermes Frontend
Hermes Consumer
Hermes Consumer
REST
REST, JMS
@DebskiChris Kraków, 19.05.2015, Atmosphere
Event identification
Hermes Frontend
KafkaBroker
POST{“event”: ”test”}
{ "id": "58d7ff07-dd0e-4103-9b1f-55706f3049e6", "timestamp”: 1430443071995, “data”: {“event”: ”test”}}
HTTP 201 CreatedMessage-id: 58d7ff07-dd0e-4103-9b1f-55706f3049e6
@DebskiChris Kraków, 19.05.2015, Atmosphere
100 ms timeout(on the client side)
@DebskiChris Kraków, 19.05.2015, Atmosphere
Network
Nagle’s algorithm 42 ms -> 6 ms
High load on client serverswrong timeout calculation
@DebskiChris Kraków, 19.05.2015, Atmosphere
65 ms timeout(on the server side)
@DebskiChris Kraków, 19.05.2015, Atmosphere
Normal operation
Hermes Frontend
Producer
Hermes Consume
rConsumer
KafkaBroker
Zookeeper
POST
HTTP201Created
@DebskiChris Kraków, 19.05.2015, Atmosphere
Slow kafka response
Hermes Frontend
Producer
Hermes Consume
rConsumer
KafkaBroker
Zookeeper
POST
HTTP202Accepted
@DebskiChris Kraków, 19.05.2015, Atmosphere
Improved offset management
HermesProducer
Hermes consumer
Remove old messages
Publish event
Commited
Local unsent queue
Read event
@DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Hermes consumer
Remove old messages
Local unsent queue
New event
Serviceinstanc
e
@DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Hermes consumer
Remove old messages
Local unsent queue
New event
Serviceinstanc
e
HTTP 503 Unavailable
@DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Remove old messages
Local unsent queue
New event
Serviceinstanc
e
HTTP 503 Unavailable
Check TTL & Add to queue
Hermes consumer
@DebskiChris Kraków, 19.05.2015, Atmosphere
Resend speed
100% adapt 1/s 1/min
@DebskiChris Kraków, 19.05.2015, Atmosphere
Lost events
Hermes Frontend
Producer
Hermes Consume
rConsumer
KafkaBroker
Zookeeper
Tracker
Publicationdata Deliv
ery
attempts
@DebskiChris Kraków, 19.05.2015, Atmosphere
Groups and Topics
pl.allegro.hermes.demo.basic Group Topic
@DebskiChris Kraków, 19.05.2015, Atmosphere
Security
Authentication and authorization interfaces
You can create any topic in your group
You can publish everywhere
Group owner defines subscriptions
@DebskiChris Kraków, 19.05.2015, Atmosphere
Multi datacenter
Hermes Frontend
Hermes Manager
Hermes Frontend
Hermes Consumer
Hermes Consumer D
C 2D
C 1
@DebskiChris Kraków, 19.05.2015, Atmosphere
What’s in it for me?
@DebskiChris Kraków, 19.05.2015, Atmosphere
What we plan to do?
@DebskiChris Kraków, 19.05.2015, Atmosphere
Lower overhead
Avro
RFC 7540
@DebskiChris Kraków, 19.05.2015, Atmosphere
Avro
RFC 7540 => HTTP/2
Lower overhead
@DebskiChris Kraków, 19.05.2015, Atmosphere
Improve security
Service authentication
@DebskiChris Kraków, 19.05.2015, Atmosphere
What’ve we learned?
@DebskiChris Kraków, 19.05.2015, Atmosphere
Tradeoffs
You cannot have exactly one delivery
http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/
@DebskiChris Kraków, 19.05.2015, Atmosphere
Clients
Not all clients are the same.
@DebskiChris Kraków, 19.05.2015, Atmosphere
Early adopters are your friends
You know exactly what features are needed.
@debskichris
allegrotech.io