the network transport layer and the application or tcp/ip and vlbi data

5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack ObservatoryR. Hughes-Jones Manchester

1

The Network Transport layer and the Application

or

TCP/IP and VLBI Data

Richard Hughes-Jones & Stephen Kershaw The University of Manchester

www.hep.man.ac.uk/~rich/ then “Talks” and look for Haystack


2

Outline

Throughput Tests on Mark5s TCP Memory-2-memory tests CPU Load tests

Data delay on a TCP link – How suitable is TCP? 4th Year MPhys Project

Stephen Kershaw & James Keenan The effect of distance

Throughput on the 630Mbit JB-JIVE UKLight Link TCP Performance


3

VLBI Application Protocol

Want to examine the data wave front

VLBI signal wave front


4

TCP: Data moving - Filling the Pipe TCP has to hold a copy of data in flight in case of re-transmission Optimal (TCP buffer) window size depends on:

Bandwidth end to end, i.e. min(BWlinks) i.e. the bottleneck bandwidth Round Trip Time (RTT)

The number of bytes in flight to fill the entire path: Bandwidth*Delay Product BDP = RTT*BW Can increase bandwidth by orders of magnitude

RTT

Time

Sender Receiver

ACKSegment time on wire = bits in segment/BW


5

VLBI Application Protocol

Want to examine how TCP moves Constant Bit Rate Data tcpdelay a test program:

instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Record TCP Stack activity with web100

Data1

●●●

Timestamp1

Time

TCP & Network Receiver

Timestamp2

Sender

Data2Timestamp4

Timestamp5

Data4

Timestamp3

Data3

Packet loss

VLBI data is produced at Constant Bit Rate


6

Check the Message Send Times

10,000 Messages Message size: 1448 Bytes Wait time: 0 TCP buffer 64k Route:

Man-UKL-JIVE-prod-Man RTT ~26 ms

Slope 0.44 ms/message From TCP buffer size &

RTT Expect: ~42 messages/RTT~0.6ms/message

Sen

d tim

e se

c

1 sec

Message number

Manchester 4th Year MPhys ProjectStephen Kershaw & James Keenan


7

Send Time Detail

100 ms

Message 102Message 76

About 25 us One rtt

Sen

d tim

e se

c

26 messages

Message number

TCP Send Buffer limited After SlowStart the

TCP Buffer is full

packets sent out in bursts each RTT

Program blocked on sendto()


8

1-Way Delay

1 w

ay d

elay

100

ms

Message number

100 ms

10,000 Messages Message size: 1448 Bytes Wait time: 0 TCP buffer 64k Route:

Man-UKL-JIVE-prod-Man RTT ~26 ms


9

= 1.5 x RTT

= 1 x RTT 26 ms

Message number

≠ 0.5 x RTT

1 w

ay d

elay

10

ms

10 ms

Why not just 1 RTT? After SlowStart TCP Buffer Full Messages at front of TCP Send Buffer have to wait for next ACKs burst – 1 RTT later Messages further back in the TCP Send Buffer wait for 2 RTT

1-Way Delay Detail


10

Problem #1 Packet Loss

Is it important ?


11

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 105

0

2

4

6

8

10

12

14x 10

6

Message numberTi

me

/ us


12

0.9 0.95 1 1.05 1.1 1.15 1.2

x 105

2.6

2.8

3

3.2

3.4

3.6

x 106

Message number

Tim

e / u

s


13

Packet loss

Is TCP is suitable for use in transferring real-time eVLBI data, on a lossy network?

t


14

Packet loss

0 1 2 3 4 5 6

x 105

2.92

2.94

2.96

2.98

3

3.02

3.04

3.06

3.08

3.1

3.12x 10

8


15

Packet loss

0 1 2 3 4 5 6 7 8 9 10

x 106

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5x 10

6

Message number

Tim

e / u

s

Difference in receive time, dropped packets minus normal


16

Problem #1 Packet Loss

Is it important ?


17

5 ms

Message number

Route:B2B on LAN gig8-gig1

Ping 188 μs

10,000 Messages Message size: 1448 Bytes Wait times: 0 μs

Drop 1 in 1000

Manc-JIVE tests showtimes increasing with a “saw-tooth” around 10 s

1-Way Delay with packet drop

800 us

28 ms1

way

del

ay 1

0 m

s


18

Standard TCP not optimum for high throughput long distance links Packet loss is a killer for TCP

Check on campus links & equipment, and access links to backbones Users need to collaborate with the Campus Network Teams Dante Pert

New stacks are stable and give better response & performance Still need to set the TCP buffer sizes ! Check other kernel settings e.g. window-scale maximum Watch for “TCP Stack implementation Enhancements”

TCP tries to be fair Large MTU has an advantage Short distances, small RTT, have an advantage

TCP does not share bandwidth well with other streams The End Hosts themselves

Plenty of CPU power is required for the TCP/IP stack as well and the application Packets can be lost in the IP stack due to lack of processing power Interaction between HW, protocol processing, and disk sub-system complex

Application architecture & implementation are also important The TCP protocol dynamics strongly influence the behaviour of the Application.

Users are now able to perform sustained 1 Gbit/s transfers

Summary & Conclusions


19

More Information Some URLs 1 UKLight web site: http://www.uklight.ac.uk MB-NG project web site: http://www.mb-ng.net/ DataTAG project web site: http://www.datatag.org/ UDPmon / TCPmon kit + writeup:

http://www.hep.man.ac.uk/~rich/net Motherboard and NIC Tests:

http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/

TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html

TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004

PFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ Dante PERT http://www.geant2.net/server/show/nav.00d00h002


20

Lectures, tutorials etc. on TCP/IP: www.nv.cc.va.us/home/joney/tcp_ip.htm www.cs.pdx.edu/~jrb/tcpip.lectures.html www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm www.cis.ohio-state.edu/htbin/rfc/rfc1180.html www.jbmelectronics.com/tcp.htm

Encylopaedia http://www.freesoft.org/CIE/index.htm

TCP/IP Resources www.private.org.il/tcpip_rl.html

Understanding IP addresses http://www.3com.com/solutions/en_US/ncs/501302.html

Configuring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt

Assigned protocols, ports etc (RFC 1010) http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols

More Information Some URLs 2

http://www.nv.cc.va.us/home/joney/tcp_ip.htm

http://www.cs.pdx.edu/~jrb/tcpip.lectures.html

http://www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS


21

Any Questions?


22

Backup Slides


23

TCP – providing reliability Positive acknowledgement (ACK) of each received segment

Sender keeps record of each segment sent Sender awaits an ACK – “I am ready to receive byte 2048 and beyond” Sender starts timer when it sends segment – so can re-transmit

Segment n

ACK of Segment nRTT

Time

Sender Receiver

Sequence 1024Length 1024

Ack 2048

Segment n+1

ACK of Segment n +1RTT

Sequence 2048Length 1024

Ack 3072

Inefficient – sender has to wait


24

Message 102Message 76

100 ms

Sen

d tim

e se

c

26 messages

Comparison of Send Time & 1-way delay

Message number


25

1 way delay

μs

Packet number

1 way delay – 10000 packets

Packet 1214

1575 packets

~ 5.5 x RTT


26

10,000 Messages Message size: 724 Bytes Wait times: 20, 25, 30, 35,

40, 45 μs TCP buffer 64k

1 w

ay d

elay

100

ms

Message number

1-Way Delay 724 byte msg


27

Packet number

1-Way Delay 724 bytes Detail

10,000 Messages Message size: 724 Bytes Wait times: 20, 25, 30, 35,

40, 45 μs TCP buffer 64k

Regular cycle of ~125 packets

1 w

ay d

elay

100

ms


28

Throughput Measurements

UDP Throughput Send a controlled stream of UDP frames spaced at regular intervals

n bytes

Number of packets

Wait timetime

Zero stats OK done

●●●

Get remote statistics Send statistics:No. receivedNo. lost + loss patternNo. out-of-orderCPU load & no. int1-way delay

Send data frames at regular intervals

●●●

Time to send Time to receive

Inter-packet time(Histogram)

Signal end of testOK done

Time

Sender Receiver


29

tcpdump / tcptrace tcpdump: dump all TCP header information for a specified

source/destination ftp://ftp.ee.lbl.gov/

tcptrace: format tcpdump output for analysis using xplot http://www.tcptrace.org/ NLANR TCP Testrig : Nice wrapper for tcpdump and tcptrace tools

http://www.ncne.nlanr.net/TCP/testrig/

Sample use: tcpdump -s 100 -w /tmp/tcpdump.out host hostname tcptrace -Sl /tmp/tcpdump.out xplot /tmp/a2b_tsg.xpl


30

tcptrace and xplot X axis is time Y axis is sequence number the slope of this curve gives the throughput over time. xplot tool make it easy to zoom in


31

Zoomed In View Green Line: ACK values received from the receiver Yellow Line tracks the receive window advertised from the receiver Green Ticks track the duplicate ACKs received. Yellow Ticks track the window advertisements that were the same as the

last advertisement. White Arrows represent segments sent. Red Arrows (R) represent retransmitted segments


32

TCP Slow Start

the network transport layer and the application or tcp/ip and vlbi data

Documents

tcp buffer size rtt

tcp link

tcp buffer sizes

packet lossis tcp

instrumented tcp program

buffer wait

buffer limitedafter

prodman rtt