Deploying Safe User-Level Network Services withicTCP

Haryadi S. Gunawi Andrea C. Arpaci-Dusseau

Remzi H. Arpaci-Dusseau

The ADvanced Systems Laboratory (ADSL) Univ. of Wisconsin - Madison

Motivation Vast number of proposed modifications to TCP/IP

Some adopted, others not deployed TCP Vegas [SIGCOMM `94] TCP Nice [OSDI `02] Congestion Manager (CM) [SIGCOMM `99]

Heart of deployment problem: OS kernel OS tend to be substantial and complex Vendors dislike changing them when benefit is not imminent A range of OS-es must implement the modifications

Transition of good research ideas into wide-spread use is slow

Emerging different network environments Wireless (lossy network), Load-balancing (packet reordering) Different extensions for different network environment TCP only support some

Current TCP implementation

TCP

TCP Applications

Reno

f ( )

f ( )

in-kernel extensions

Inside to outside

TCP

TCP Applications

f ( )

Information Control

Why in-kernel extensions?

Information and control only available within the kernel

Ex: TCP Vegas• Information: per-packet

RTT and TCP States• Control: Congestion

Window Questions?

Can extensions be built in application layer?

What information and control need to be exposed?

Proposed Framework icTCP (“information and control” TCP)

Address the problem of deployment Slightly modified in-kernel TCP stack Exports key pieces of information Provides safe control to user-level libraries Given information and control, extensions can be built in

application layer Evaluation

Converting TCP to icTCP requires a small amount of additional code

The resulting flow is TCP friendly Minimal overhead Implement 6 user-level extensions Little complexity in implementing extensions at user-

level

Benefits

lib-f ( )

icTCP

Apps

Deployable

New services packaged as libraries and easily downloaded.

lib-f icTCP

App1 App2

lib-g

Flexible

Developers can tailor libraries to the their needs

lib-g ( )

icTCP

Apps

lib-f ( )

Composable

Services are used together for more powerful functionality

icTCP facilitates the development of user-level extensions

Outline Motivation and Overview icTCP Design and Implementation

Information Control

5 Axes of Evaluation Conclusion

icTCP Design Different TCP

connections, different libraries

User-Libraries on top of icTCP

icTCP exposes Information and Control

Information Which information should

be exposed? Too little limited extensions Too much expanded

interface

2 types of Information: Variables part of TCP

specification (RFC 793)• cwnd, ssthresh

Message list and Ack list• More detailed information• History of recent packets and

acknowledgements• Enabled by user-level services

to save memory

TCP Clients

User Libraries

How and when to obtain information? Polling

BSD socket interfaces Unnecessary run-time

overhead• Kernel-user space copy

When to obtain accurately?

TCP variables are updated:

• upon receipt of an ACK• end of a round

Interrupt mechanism icTCP notifies application

when these events happen

CPU Overhead

0

2

4

6

8

10

Reno Poll /Send()

Poll /Ack

Poll /Round

CPU

Utilz

atio

n (%

)

Control Allow internal TCP variables

to be externally set in a safe manner

What variables and what values?

TCP-Friendly: don’t harm other competing standard flows

Philosophy: icTCP must be conservative

Control is allowed if not aggressive

Control 10 (virtual) variables that can be safely set

User Libraries

TCP Clients

Implementation of Safe-Control Idea: Virtual variables

congestion window (cwnd) virtual congestion window (vcwnd) Manipulate policies through virtual variables TCP Reno keeps running using the original variables

Safe Ranges Enforce friendliness 0 ≤ vcwnd ≤ cwnd Without safe ranges, resulting flows not TCP-friendly

Choosing 10 variables, defining safe-ranges Check safe-setting theoretically and empirically Theoretically: Reno equation

• congestion window: control how many packets in the network• vcwnd > cwnd : more packets in the network, thus more aggressive

Empirical Prove• Set virtual congestion window outside the safe ranges

Outline Motivation and Overview icTCP Design and Implementation Framework Evaluation Conclusion

5 Axes of Evaluation

12

4 3

5

How easy to convert TCP to icTCP?

Is icTCP friendly?

What are the overheads? Does it scale?

What types of extension can be built and deployed with icTCP?

How difficult to develop TCP extensions in this way?

Converting TCP to icTCP icTCP in Linux 2.4.18 Add 316 lines of code

Non-intrusive modifications

TCP toicTCP1

Network Safety: Unconstrained icTCP

2Friendly?

TunconsReno

Reno

RenoTReno

TReno

Runcons :Tuncons

TCP-Friendly: ~ 1

UnconstrainedicTCP

(allow variable setting to any value)

TCP-Unfriendly

Network Safety: Constrained icTCP

TconsReno

TReno

Rcons :Tcons

Reno

RenoTReno

Safety Safe Ranges are required!

TCP-Friendly

2Friendly?

UnconstrainedicTCP

ConstrainedicTCP

.

(allow variable setting within the safe ranges)

TCP-Unfriendly

Scalability and Overhead What is the overhead? Does it scale?

Rate of getting info and setting variables Different extensions, get/set at different rate

Two factors: Per-ack or per-round interrupts Need the message list and ack list

3 synthetic libraries: per-ack interrupt per-round interrupt per-round interrupt + gets message list

3 Scale?Overhead?

OverheadScale?Overhead?3

per-ack (4 conns)

(96 conns)

per-round 12%

per-ack 30%

S

R1

R3

R2

R4

12 MB/s per-round + msgList (8 conns)

Noticeable overhead, but not prohibitive

TCP Extensions Range of TCP extensions can be built on top of icTCP Implement 3 sets case studies (6 extensions):

Congestion window:• TCP Vegas [Brakmo, SIGCOMM ’94]• TCP Nice [Venkataramani, OSDI ’02]• Congestion Manager (CM) [Balakhrisnan, SIGCOMM ’99]

Duplicate threshold• Reordering-Robust Ext [Zhang, ICNP ‘03] • Efficient Fast Retransmit [Tamura, LCN ‘98]

Retransmission Timeout• Eifel RTO [CCR ‘00]

4Range ofExtensions?

icTCP

lib-Vegas

User-level TCP Vegas Implementation

TCP Clients

TCP states

msg list

Vegas

f vcwnd

vcwnd

4Range ofExtensions?

TCP Vegas [Brakmo et.al.,SIGCOMM ’94]

per-round interrupt

Using information and control is simple Complexity is within the algo itself

Does it work?

User-Level Lib. TCP VegasIn-kernel TCP Vegas

Reno

Same Behavior

TCP Reno: Send more packets until drop.

TCP Vegas: Maintain the “right” amount of extra data in the network

e.g. keep only 3 packets in the bottleneck queue

4Range ofExtensions?

icTCP Strengths Implement less aggressive TCP

variants at user-level No need to push changes into the

kernel Ideally suited for tuning parameters

whose optimal values depend upon the environment

Lossy Network retransmit faster Use Efficient Fast Retransmit

Packet reordering postpone retransmission Reordering Robust (RR) Extensions

Different environment, opposing solutions

TCP favors one solution Correcting errors in parameter values

Eifel RTO [CCR ’00]: RTO prediction flaw when RTT drops

In newer Linux version, this prediction flaw is corrected with adding 2 new lines of code

Must wait vendors to correct this flaw

4Range ofExtensions?

TCP + f lib-f icTCP

App2

RR

PacketReordering

dt > 3dt < 3

App1

Lossy Network

EFR

Other approaches Can’t implement all extensions

Only allow 10 existing variables to be set Conservative: Safety

STP (Self-Spreading Transport Protocols) [SOSP ’03]

Remotely upgrade other’s protocol stack More TCP extensions can be deployed Use separate module to enforce

friendliness

Why we don’t use enforcer Drawback: terminate non-conforming

flows icTCP modulates aggressive flows

4Range ofExtensions?

Set existing

variables with limits

lib-f icTCP

Apps

Set existingvars to any

values

lib-f icTCP

Apps

Enforcer

vs.

Ease of Development How difficult to develop TCP extensions in

this way? Complexity at user-level vs. in-kernel

5DifficultDevelop?

Libray vs In-Kernel Implementation

0

200

400

600

TCP Vegas TCP Nice CM RR

Extensions

Line

s of

Cod

e

User-Level Library In-Kernel

1200(*)

438

Composability Composing multiple libraries

Vegas: good for small bottleneck queues RR: good for reordering Environment where both situations exist?

• Vegas+RR: Better throughput

TCP Clients

icTCP

Vegas RR RR

Vegas

5DifficultDevelop?

Conclusion icTCP

Philosophy: Information and Control Non-intrusive, simple, and TCP-friendly Systems with icTCP reap benefits of

user-level TCP extensions

Change TCP once now, no need to change it later

Questions?

The ADvanced Systems Laboratory

www.cs.wisc.edu/adsl

Related Work Mogul et.al. [HotNets-2003]

Arbitrary state setting Web100 and Net100

Export a variety of per-connection statistics Does not ensure network safety

STP [SOSP-2003] Remotely upgrade other’s protocol stack Invasive changes to the kernel Additional machinery to ensure friendliness

InfoTCP/infokernel [SOSP-2003] icTCP exposes more information icTCP allows services to directly set cwnd inside of TCP icTCP allows more variables to be controlled