tcp performance over gigabit networks arne Øslebø nordunet2002

TCP Performance over Gigabit Networks

Arne Øslebø

NORDUnet2002

Modern Network Interface Cards

● Programmable network processors● Hardware checksuming

● Offloads main CPU● Interrupt coalescing

● Less interrupts● On board memory● Zero-copy

● Mainly operating system issue

Test tools● netperf

– used for measuring throughput between two systems.– http://www.netperf.org/

● vmstat / Microsoft Management Console– gives information about CPU load and interrupts/s.

● netstat– gives information about packet loss, retransmitions etc.

● One testrun consists of:– running netperf using various TCP window sizes.– for each TCP window size, netperf is run three or more

times. Results presented are average values.– Before running netperf, information is collected from

netstat– While running netperf, information is collected from

vmstat.

Laboratory testing● Hardware:

● Motherboard: Supermicro 370DE6● Memory: 1GB SDRAM● CPU: 1GHz Pentium III● NIC: 3COM 3C985B-SX

● Operating systems:● Linux: 2.4.9● NetBSD: current● Windows 2000: service pack 2

● RTT=0.227ms

Point to point - throughput

4KB 8KB 16KB 32KB 64KB 128KB

256KB

512KB

1024KB

2048KB

0

100

200

300

400

500

600

700

800

900

1000

Throughput

Linux(1500)

Windows(1500)

NetBSD(1500)

Linux(9000)

Windows(9000)

NetBSD(9000)

TCP window

Mbit/s

Point to point - CPU load


256KB

512KB

1024KB

2048KB

0

10

20

30

40

50

60

70

80

90

100

CPU load

Linux(1500)-t

Linux(1500)-r

NetBSD(1500)-tNetBSD(1500)-r

Linux(9000)-t

Linux(9000)-rNetBSD(9000)-t

NetBSD(9000)-r

TCP window

%

Point to point - interrupts/s


256KB

512KB

1024KB

2048KB

0250050007500

10000125001500017500200002250025000275003000032500350003750040000

Interrupts

Linux(1500)-t

Linux(1500)-r

NetBSD(1500)-t

NetBSD(1500)-r

Linux(9000)-t

Linux(9000)-r

NetBSD(9000)-t

NetBSD(9000)-r

TCP window

Int/s

Analyzing results

● Problems with using tcpdump● Negative effect on throughput● Drops many packets● Interrupt Coalescing

● Gives inaccurate time stamps● Low performance using Windows

2000 probably related to driver.

MTU Linux NetBSD Windows 1500 792 640 3759000 988 988 692

Real world experiments

MTU size: n=normal(1500) j=jumboframe(4470)

Naming: nn

country: c=Czech republicf=Finlandn=Norways=Switzerland

These tests have been possiblethanks to the good help from:●Jari Miettinen, CSC●Tuomas Toropainen, CSC●Simon Leinen, SWITCH●Sven Ubik, CESNET

Czech Finland Norway Switzerland NICCzech 43.2 51.1 31.1Allied Telesyn AT2970SX/SCFinland 43.2 20.5 57.43COM 3C996Norway 51.1 20.5 65.53COM 3C985BSwitzerland 31.1 57.4 65.5 SysKonnect SK-9843

RTT:

Finland - Norway

64KB

128KB

256KB

512KB

1MB 2MB 3MB 4MB 5MB 6MB 7MB 8MB 10MB

0

100

200

300

400

500

600

700

800

900

1000

Throughput

nn->fj

fj->nn

nj->fj

fj->nj

TCP window

Mbit/s

Finland - Switzerland

64KB

128KB

256KB

512KB


0255075

100125150175200225250275300325350375400

Throughput

sn->fj

fj->snsj->fjfj->sj

TCP window

Mbit/s

Norway - Switzerland

64KB

128KB

256KB

512KB


0

25

50

75

100

125

150

175

200

225

Throughput

nn->sn

sn->nn

nj->sj

sj->nj

TCP window

Mbit/s

Variations over a day

0 1 2 3 4 5 6 7 8 9 10

11

12

13

14

15

16

17

18

19

20

21

22

23

0

100

200

300

400

500

600

700

800

900

1000

Throughput

nj-fj

nj-sj

Time of day

Mbit/s

Czech-Finland-Norway

64KB

128KB

256KB

512KB


0

50

100

150

200

250

300

350

400

450

500

550

600

Throughput

nn-cncn-nn

fj-cn

cn-fj

TCP window

Mbit/s

Analyzing results

● Good performance using jumboframes.

● Problems with large TCP window sizes● Asymetric results

– Probably driver and hardware related● Variance in results to Switzerland

caused by SACKs and retransmits

Czech Finland Norway SwitzerlandCzech 482 601 -Finland 219 130 48Norway 181 230 79Switzerland - 107 123

Fro

m

To

MTU=1500Czech Finland Norway Switzerland

Czech - - -Finland - 568 364Norway - 953 219Switerland - 396 215

MTU=4470

To

Fro

m

Conclusions and further work● Conclusions:

– Diffucult to achieve high performance– Still possible if you use:

● A good PC● A good network card● A good driver for the card

– Large TCP windows leads to performance degradation.– Default maximum TCP window size too small for high

speed networks.● Further work:

– Will investigate asymetric results in more details.● Use DAG cards for packet capture to get accurate TCP traces.

● Full report will be posted on http://www.uninett.no/prosjekt/tcp-revisited/

tcp performance over gigabit networks arne Øslebø nordunet2002

Documents