TCP-XM“Multicast Transport for Grid Computing”

Karl Jeaclekarl.jeacle@cl.cam.ac.uk

Multi-Service Networks 2003

Rationale Would like to achieve high-speed bulk data

delivery to multiple sites Multicasting would make sense Existing multicast research has focused on

sending to a large number of receivers But Grid is an applied example where sending to

a moderate number of receivers would be extremely beneficial

Multi-Service Networks 2003

Reliability Reliability of data transfer is essential But multicast packets are typically UDP Could adopt or invent a new reliable multicast

protocol that allows reliable multicast (e.g. PGM) But what about taking an existing reliable unicast

protocol (e.g. TCP) and turning it into a reliable multicast protocol?

Multi-Service Networks 2003

Multicast TCP? TCP provides a single reliable byte stream

between two end-points Modifying this to support multiple end-points may

seem a little unnatural But there is some precedent…

SCE – Talpade & AmmarTCP-XMO – Liang & CheritonM/TCP – Visoottiviseth et al

Multi-Service Networks 2003

Obvious problems ACK implosion

Sender swamped by ACKs from receivers!Yes, so 40 byte ACKs on 1500 byte data packets

means symmetric data traffic with ~40 receivers. But this beats 40x unicast!

End-point asymmetryUnicast TCP: pipe has two end-pointsMulticast TCP: pipe has n end-points

Sender needs application framing to demultiplex received data

Multi-Service Networks 2003

Implementation issues Protocol change == kernel change

Widespread test deployment difficult Two prong approach

Modify TCP protocol…But incorporate into a userspace library

Library will implement modified TCP over UDPPerformance is sacrificedBut deployment is no longer an issueLibrary can be used within Globus

Multi-Service Networks 2003

TCP/IP/UDP/IP

SendingApplication

TCP

IP

UDP

IP

ReceivingApplication

TCP

IP

UDP

IP

native implementation

prototype deployment

Multi-Service Networks 2003

TCP engine Where does initial TCP come from? Could use BSD or Linux

Extracting from kernel could be problematic More compact alternative

lwIP = Lightweight IPSmall but fully RFC-compliant TCP/IP stack

lwIP + multicast extensions = TCP-XM

Multi-Service Networks 2003

TCP

SYN

SYNACK

ACK

FIN

FIN

ACK

ACK

ACK

DATASender Receiver

Multi-Service Networks 2003

TCP-XM

Sender

Receiver 1

Receiver 2

Receiver 3

Multi-Service Networks 2003

API Sender API changes

New connection typeConnect to port on array of destinationsSingle write sends data to all hosts

TCP-XM in use:conn = netconn_new(NETCONN_TCPXM);netconn_connectxm(conn, remotedest, numdests, group, port);netconn_write(conn, data, len, …);

lwIP uses “netconn” APIBut a socket API is also available

Multi-Service Networks 2003

PCB changes Every TCP connection has an associated

Protocol Control Block (PCB) TCP-XM adds:

struct tcp_pcb {…struct ip_addr group_ip; // group addrenum tx_mode txmode; //

uni/multicastu8t nrtxm; // # retransstruct tcp_pcb *nextm; // next “m” pcb

}

Multi-Service Networks 2003

Linking PCBs

*next points to the next TCP session *nextm points to the next TCP session that’s part of a

particular TCP-XM connection Minimal timer and state machine changes

next

nextm

M1

next

nextm

U1

next

nextm

M2

next

nextm

M3

next

nextm

U2

Multi-Service Networks 2003

TCP Group Option

New group option sent in all TCP-XM SYN packets Presence implies multicast capability Non TCP-XM hosts will ignore (no option in SYNACK) Sender will automatically revert to unicast

1 byte 1 byte 4 bytes

kind=50 len=6 Multicast Group Address

Multi-Service Networks 2003

TCP-XM connection Connection

User connects to multiple unicast destinationsMultiple TCP PCBs created Independent 3-way handshakes take placeSSM or random ASM group address allocated

(if not specified in advance by user/application)Group address sent as TCP optionAbility to multicast depends on TCP option

Multi-Service Networks 2003

TCP-XM transmission Data transfer

Data replicated/enqueued on all send queuesPCB variables dictate transmission modeData packets are multicastRetransmissions are unicastAuto switch to unicast after failed multicast

CloseConnections closed as per unicast TCP

Multi-Service Networks 2003

TCP-XM reception Receiver

No API changeNormal TCP listenAuto IGMP join on TCP-XM connectAccepts data on both unicast/multicast ports tcp_input() modified to check for valid data in packets

addressed to multicast groups

Multi-Service Networks 2003

Future To do:

Protocol workParallel unicast / multicast transmissionFallback / fall forwardMulticast look aheadMultiple groupsGlobus integration

ExperimentsLocal networkUK eScience centres Intel PlanetLab

Multi-Service Networks 2003

All done!

Thanks for listening!

Questions?