a look at tcp behavior & bufferbloat - wild...

Produced by and Copyright, 2016Philip Hippensteel

A Look at TCP Behavior & Bufferbloat:

Understanding, Detecting and Mitigating Performance Issues

NY and New EnglandTelephone Associations: 2/1/17

Dr. Phil [email protected](717) 448-0552

2Produced by and Copyright, 2016Philip Hippensteel

Why Look at this Topic?

Bufferbloat creates inefficiency in worker performance if they use the network or the Internet. It is not a security issue (although exploits

may be possible). IT generally is not unaware of the issue. Also, your ISP will likely be clueless.


How Did This problem Develop?

Due to poor understanding of network operations, particularly the TCP protocol, we: Developed the notion that dropping packets was

always bad. Didn’t really study and understand TCP, even though it

carried 90% of the traffic across our networks. Increased link speed dramatically on both the LAN and

the WAN. Increased speed differences at network connections.


What is Bufferbloat?

The effect that oversized buffers have on network latency. It causes Slow responses to web requests. Decreases in throughput of file transfers when the

return path is congested. Decreases in quality of adaptive bit rate video

(Netflix, Hulu, etc.) Can be easily confused with the affects of

malware.


The Background

Understanding bufferbloat means understanding TCP operation The role of DNS How buffers work and are managed

Understanding it’s impact means understanding DNS HTTP operation


Normal TCP Operation(Send Policy)

Station A Station Bsyn

syn/ackack

datadata

data ack

datadata

data

data

Cwnd=2

Cwnd=4

Cwnd= 8

VIP: The TCP sequence numbers increase by one per byte transferred. i.e. The seq # measures the number of TCP bytes

delivered.

data ack

Etc.

data


TCP Exchange

* View generated with Viavi Observer Gigastor.


The Receive Policy

When packets arrive in order: Ack a packet if either: 200 ms. has transpired since last ack OR Ack second packet received but unacked.

When packet arrives out-of-order Ack previously received in-order packet


Error Free Ideal TCP Performance

TCP send action has three phases. * Assuming no jitter, uniform delay, no packet loss, etc. Steady-state is a goal that can only be reached under ideal

conditions. Slow-start assumes cwnd ≤ ssthresh (later).

time

Segments sent

slowstart

CongestionAvoidance

Steady State*


Wireshark Shows Slow Start

Note delta time


Ideal TCP Performance w/ Packet loss

time

Segments sent

slowstart

Congestion Avoidance Steady

State

Dropped packet

slowstart


More on Send/Receive Policies

Sender Policy: cwnd: the number of segments that may

currently be sent before receiving an ack. rwnd: the number of incoming receiver buffer

bytes available number of segments that can be sent:

Min(cwnd, rwnd)


Cwnd and Latency

Cwnd depends on RTT. Therefore increased latency slows the

grow of cwnd (the ability to send).

cwnd

48

163264

1 RTT 2 RTT 3 RTT 4 RTT 5 RTT

time


Error Handling (more)

When a packet arrives out-of-order, the receiver:(1) immediately ack the previous packet (sender gets dup ack)

(2)Next packet out-of-order, receiver sends same ack. (sender has second dup ack)

(3) Next packet out-of-order, receiver sends same ack (sender receives third dup ack)

(4) Sender retransmits packet requested and proceeds with packet loss policy.

(5) Typically sender cuts cwnd to half and restarts slow-start.

slowstart

CongestionAvoidanc

e

slowstart


Wireshark Shows Dup-acks


TCP Stack Variations

Tahoe TCP Reno TCP New Reno Compound – current MS implementation Cubic – current Linux distros


Real Example of Uncongested TCP:Netflix

This is a connection to Netflix. DSL: 20M down, 2 M up.


Real Example of Uncongested TCP:Ubuntu Download

Connection to a Ubuntu repository server.

DSL: 20M down, 2 M up.


Real Example of TCP:Netflix(SSL) & Web Browsing(HTTP)

DSL: 20M down, 2 M up.


Real Example of TCP:Ubuntu Download & Dropbox Upload

• DSL: 20M down, 2 M up.

• So, at t=55, why doesn’t TCP use the available total bandwidth of 22Mb/sec?

** I believe this slide should be memorized!!!


PART 2: HTTP: HYPERTEXT TRANSFER PROTOCOL


HTTP Has Been Around

HTTP versions 0.9,1.0,1.1 Essentially a client request-server fulfill

protocolClient ServerStart TCP Session

Start TCP sessionGet Request(s)

Get Response (200 OK)Response Transmitted

TCP Session CloseTCP Session Cloe


HTTP 1.1 Typical Operation

Client Browser Web ServerGet index file

Index file transmittedGet page components

Other components sent

* * A recent web development text quoted a study in 2013 that showed that the average web request involved 90 retrievals of text, images, scripts, icons, etc. Many of these will involve a DNS query/response.


PART 3: BUFFERS AND BUFFERBLOAT


Where Are Buffers

In the stack between layers. In the NIC drivers. In the edge routers (especially wireless

routers). In boundary devices between service

providers.


Purpose of Buffers

Smooth flow of packets. Allow QOS policies to be implemented. Make packet drops more unlikely. Allow for security policy implementations.


Flows Though Different BW Links

Above: height is BW, width is time used. So, area is data carried. Ack rate is dependent on slowest link.

receiver

High BWLow BW


Buffer Action

1) Group of segments arrive

2) Segments buffered3) Segments begin to exit

buffer at slower speed4) Another group of

segments arrive5) Buffer full6) Last to arrive are

dropped (tail-drop)discards

Flow direction


Buffers and File Transfer


Ubuntu Download Starts


A Buffer Overfills


When is segment Retransmitted?

Note: 413 dup acks received before retranmission!


BUFFERS, DNS, HTTP AND SLOW START


A Practical View of DNS Use

request number dns queries average response time (ms)

get vanguard 15 24 get ebay 26 28 get google 18 25 get irs.gov 6 35 get highmark 2 25 get highmark mission values 3 36 get washington.edu 22 80


Buffers and DNS

Test with wired Ethernet/DSL


Buffers and DNS with WiFi/DSL


DNS and Upload on Wi-fi

No Upload

SimultaneousUpload


Summarizing Bufferbloat Affects

Slows slow start Decreases file transfer throughput by

delaying acks Increases DNS response times, increasing

response times for loading web pages


Part 3: Mitigation


Queue Management:Commonly Deployed Drop Policies

RED: random early drop WRED: weighted RED CoDel: drop based on queue occupancy fq-codel: codel with fairness to flows.

upstream

downstream

discard


Good Queues, Bad Queues

Bad Queue

Good Queue

packets

time


CoDel Operation

The critical metric is minimum queue sojourn time. A second parameter, target, is also used. Drop Policy: if a packet sojourns have a

minimum that exceed target for more than one interval time, then randomly drop a packet. In practice, interval = 100 ms and target =

5-10% of RTT.


CoDel Illustrated

Per packet sojourn time

time

target

interval

interval


fq-codel Operation

1: Incoming Flow 2: Random assignment to up to 1024 sub queues. 3: CoDel applied per queue. 4. One MTU per queue scheduled out, round robin.

●●●

2

1

3


Summary:Big Buffer Problems

Slow ack on large file transfers. Slow DNS response resulting in decreased

web performance. Collaboration Voice/RTP video: without VLANs or QOS, it can

cause packet loss (due to jitter buffer overflow) Video conferencing: cause SVC based VC to

degrade Adaptive video: cause decrease in resolution

or playback pause.


TESTING FOR BUFFERBLOAT


ICSI Netalyzr


www.dslreports/speedtest/


Manual Test

Start a continuous ping (ping www.google.com –t) Run test such as www.speedtest.net Observe change in ping times


Another Manual Test(continuous ping with upload)

Upload starts

Upload stops


Some Final Thoughts

Does QOS mitigate bufferbloat? No. It can determine which flows are affected

and by how much they are affected. If fq-codel is a mitigation, where should it

be implemented? At the edge router or in front of it.

Can traffic shaping mitigate bufferbloat? Yes, but at the expense of some throughput.


AQM with Wireless

Source: Controlling Queue Delay, ACM Queue, Kathy Nichols and Van Jacobson, 2012.


Questions?

a look at tcp behavior & bufferbloat - wild...

Documents