tuning speculative retries to fight latency (michael figuiere, minh do, netflix) | cassandra summit...
TRANSCRIPT
Tuning Speculative Retriesto Fight Latency
Minh Do - Senior Distributed System Engineer Michaël Figuière - Senior Distributed System Engineer
Agenda
What are Speculative Retries?
C* Setting & Internal Code
C* Testing and Outcome
Datastax Java Driver
Speculative Retries
• “The Tail at Scale” - Google paper• Hedged Requests vs. Speculative Retries
Speculative Retries
• Tuning available both in C* and Java Driver
• It is all about the additional read retries to improve the latency tail
C* Setting & Internals
• Read path only • Set at table level:
– None, Always, Custom, Percentile• Using different executors for different behaviors• Impacted by Read Repair (for simplicity, ignore in
this talk)
No Retry Policy
• Use NeverSpeculatingReadExecutor• Coordinator computes a list of alive replicas:
– #blockFor• Sends only one full data request to one replica• Sends additional digest request(s) to others
depending on Consistency Level (CL)
No Retry Policy
Example using CL_LOCAL_QUORUM (replication factor 3)
Always Retry Policy
• Use AlwaysSpeculatingReadExecutor• List of replicas: (#blockFor + 1) or all nodes• Coordinator sends 2 full data requests to 2
replicas• Sends additional digest requests to others
depending on Consistency Level
Always Retry Policy
Example using CL_ONE
Custom/Percentile Retry Policy
• Use SpeculatingReadExecutor• List of replicas: #blockFor + 1• Send 1 full data request and 1 or more digest request to
replicas depending Consistency Level• Use the last replicas on the list for the retries• Coordinator waits for a duration to retry on another replica
– Custom: duration in millisecs– Percentile: the percentile on the sampling latency
distribution
Custom/Percentile Retry Policy
Example using CL_LOCAL_QUORUM (replication factor 3)
C* test - setup
• Cassandra: 12 nodes (8 cores/60Gb RAM), approx 120Gb data per node
• Client: 6 (8 cores/30 Gb RAM)• Use Linux Traffic Control (tc) to simulate network glitch on
one C* node using delayed packet transmission technique
C* test - result
• Clients send 50K reads/sec with 3K writes/sec, • Use TC to force a slow node • Enable 95th Speculative Retry setting• Throughput degraded to 30K reads/sec on Coord.• Avg. latency gets doubled 0.5ms to 1ms• 95th/99th latencies have many spikes to 2x-10x
C* test - result
• Client 18K reads/sec with 1K writes/sec• Use TC to force a slow node • Keep 95th Speculative Retry setting on• Throughput increased to 20+K reads/sec• Avg. latency gets doubled 0.5ms to 1ms• 95th/99th latencies are lower, no spikes or more
stable
C* test - conclusion
• More stable and lower 95th/99th latencies: – Build cluster with extra capacities and turn on
Speculative Retry• C* cluster is already near its max capacity
– Disable Speculative Retry as performance getting worse
Client Speculative Retries
• SpeculativeExecutionPolicy
• Speculative retries for both reads and writes
• But only for idempotent requests
Client Speculative Retries
Example using CL_LOCAL_QUORUM (replication factor 3)
NoSpeculativeExecutionPolicy
• Just the normal Driver behavior
• Retries are attempted only after a fixed timeout as part of Failover
ConstantSpeculativeExecutionPolicy
Trigger a Speculative Retry if the Coordinator hasn’t answer within a given delay.
PercentileSpeculativeExecutionPolicy
Trigger speculative retries if the chosen coordinator doesn’t answer within a given
percentile of its typical response time
Questions
?