ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester1

VLBI & Protocols

vlbi_udp Multiple Flow Tests

Richard Hughes-Jones The University of Manchester

www.hep.man.ac.uk/~rich/ then “Talks”

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester2

Setting up the Tests

Thank you to the people at all the sites for your help and making facilities available

Including: Onsala JIVE Torun and Pioner Poznan GARR Bologna Jodrell and Manchester Network Group Dante

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester3

Dedicated

Gbit link

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM linkMedicina

Italy

Chalmers University

of Technolog

y, Gothenbur

g

TorunPoland

Gbit link

Metsähovi

vlbi_udp: Multi-site Streams

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester4

iGrid2002 monolithic code Convert to use pthreads

control Data input Data output

Code branch for Simon’s file transfer tests

Work on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on

semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do

Multi-flow Network performance – being set up Nov/Dec06 3 Sites to JIVE: manc UKLight; Manc production; Bologna GEANT PoP Measure: throughput, packet loss, re-ordering, 1-way delay

To do Add code for: MarkV card and PCEVN interface

vlbi_udp: UDP on the WAN

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester5

vlbi_udp: Some of the Problems

JIVE made Huygens, mark524 (.54) and mark620 (.59) available Within minutes of Arpad leaving the Alteon NIC of mark524 lost the

data network! OK used mark623 (.62) – faster CPU

Firewalls needed to allow vlbi_udp ports Aarrgg (!!!) Huygens is SUZE Linux

Routing – well this ALWAYS needs to be fixed !!! AMD Opteron did not like sched_getaffinity() sched_setaffinity()

Comment out this bit udpmon flows Onsala to JIVE look good udpmon flows JIVE mark623 to Onsala & Manc UKL don’t work

Firewall down stops after 77 udpmon loops Firewall up udpmon cant communicate with Onsala

CPU load issues on the markV systems Don’t seem to be able to keep up with receiveing UDP flow AND

emptying the ring buffer Torun PC / Link lost during the tests

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester6

vlbi_udp Onsala mark623

It worked ! No not quite !!! Asked for a flow of 600Mbit/s (-w20 us)

got ~300 Mbit/s (-w58 us) 1.8% packet loss 1.1% re-ordered (mainly by 1 or 2 packets)

Onsala gig7 (Academic internet to Ams then UKLight)Onsala gig8 (all Academic internet)

~4.5% lost and ~3.5% re-ordered

vlbi_udp_Onsala-gig8_w20_4Dec06

1

10

100

1000

10000

100000

1000000

10000000

0 20 40 60 80 100 120 140 160

Time between frames us

N(t

)

vlbi_udp_gig7-Huy_w20_2Dec06

1

10

100

1000

10000

100000

1000000

10000000

0 20 40 60 80 100 120 140 160

Time between frames us

N(t

)

2.2GHz AMD Athlon Udpmon OK but single thread Vlbi_udp uses recvmsg() &

iovectors

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester7

Multiple vlbi_udp Flows Gig7 Huygens UKLight 15 us spacing 800 Mbit/s Gig8 mark623 Academic Internet 20 us spacing 600 Mbit/s Bologna mark620 Academic Internet 30 us spacing 400 Mbit/s

SURFnet Access link

SJ5 Access link

GARR Access link

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester8

Multiple vlbi_udp Flows Gig7 Huygens

UKLight 15 us spacing 816 Mbit/s del <1Mbit/s

step 1 Mbit/s Zero packet loss Zero re-ordering

Gig8 mark623 Academic Internet 20 us spacing 612 Mbit/s 0.6 falling to 0.05% packet loss 0.02 % re-ordering

Bologna mark620 Academic Internet 30 us spacing 396 Mbit/s 0.02 % packet loss 0 % re-ordering

vlbi_udp_3flows_6Dec06

0100200

300400500600700

800900

1000

0 2000 4000 6000 8000 10000 12000 14000

Time s

Wir

e R

ate

Mb

it/s

00.10.2

0.30.40.50.60.7

0.80.91

Pa

ck

et

los

s %

vlbi_udp_3flows_6Dec06

0100

200300400500

600700800

9001000

0 2000 4000 6000 8000 10000 12000 14000

Time s

Wir

e R

ate

Mb

it/s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Pa

ck

et

los

s %

vlbi_udp_3flows_6Dec06

0100

200300

400500600

700800

9001000

0 2000 4000 6000 8000 10000 12000 14000

Time s

Wir

e R

ate

Mb

it/s

0

0.02

0.04

0.06

0.08

0.1

0.12

Pa

ck

et

los

s %

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester9

Any Questions?

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester10

Backup Slides