1

Detailed plan - UVA

• Dynamic circuit setup/release• Equipment to purchase• What we plan to implement• Wide-area circuits

• High-throughput transport protocol item• Support for delay-controlled transport

protocol• Support for router disconnect

Malathi Veeraraghavanmv@cs.virginia.edu

04/22/23 2

Already purchased• Two 64-bit 100Mhz bus PCs with GbE• One Cisco 15454 unit

– 10/100 Mbps Ethernet card – 12 ports– 1 OC3 card – 4 ports– STS-1 level crossconnect card– Control card with release 4.0 software

• Augment 15454 with– OC48 card– GbE card

04/22/23 3

PC

NIC I

PC

NIC I

Eth. Sw.

XCEthernet OC3

NIC II NIC II

A simple Item I demo

• FTP sessions starts on TCP connection through NIC1• Signaling module uses TL1 to set up the circuit• Large file transfer through NIC II using ST transport protocol• Routing decision based on file size

– if large enough, attempt circuit setup

Ctrl

SigST ST

Signaling Signaling

FTP

TCP TCP

FTPRouting decision

Routing decision

04/22/23 4

Software architecture• End host software modules

– Develop each module separately and perform “unit” tests

– System integration & testing

SignalingDelay-controlledtransportprotocol

Database: name, gateway,

authenticationRoutingdecision

File transferapp. Remote

visualization app.MiddlewareHigh-throughput

transportprotocol

04/22/23 5

For a wide-area test• To test this software on a wide-area

basis, we will need additional software• Multiple 15454s

– need routing database– need inventory database

• Obtain software from Canarie or Starlight teams and demo centralized solution

04/22/23 6

Value limited• In implementing centralized

solution– Others are already testing this

solution– Not scalable

• Utilization and hence cost implications• Conclusion: Explore distributed

solution

04/22/23 7

Equipment to purchase• One more PC with GbE card, 64-bit PCI 133Mhz

bus and high-end disks• Two Cisco 15454s with GbE and OC48 cards• Three SONET crossconnects with UNI-N/NNI

software– Unfortunate that 15454 only implements UNI-C, and

not UNI-N or NNI– This means to set up a crossconnection through the

15454, TL1 is the only choice.– Hence to explore distributed solution, we need

crossconnects with UNI-N/NNI

04/22/23 8

Signaling software – option 1

• Provision GbE/EoS circuits through 15454s using TL1 commands a priori

• Signaling software at end host generates UNI-C RSVP-TE Path messages

• Leverages routing and inventory data located at XCs

PC PC

15454 15454

SONETXC with

UNI-N/NNI

SONETXC with

UNI-N/NNI

Ethernet switch

XC

15454

PC

04/22/23 9

Signaling software – option 2

• Mixed information on Cisco’s plans to implement UNI-N/NNI – best date heard: first half of 2005

• Late for our Yr. 1 demo

PC PC

15454 15454

SONETXC with

UNI-N/NNI

SONETXC with

UNI-N/NNI

Ethernet switch

Trigger UNI-C message

Database:map dest. IP addr.to 15454 address

TL1 TL1

04/22/23 10

For SONET XC purchase• To get competitive bids from

equipment vendors, team up with:– Guy Almes, Internet 2– Bill St. Arnaud, CA*net 4– Tom West, NLR– Bill Wing, ESnet

04/22/23 11

Wide-area circuits• Choices widening

– Internet2 placing a 15454 in MANLAN facility– 10Gbps Lambda between NY and Amsterdam

planned to be moved from Abilene router to MANLAN TDM node

– CUNY connected by NYSERnet dark fiber to MANLAN site and CA*4net site

– Allows us to place one PC with our software at CUNY and ship another software module with/without PC to a prof. in Chicago or Amsterdam to test our software long-distance

04/22/23 12

Wide-area circuits• Two options:

– Ship equipment (PCs, 15454s, XCs) to organizations already connected by high-speed circuits and test our software

– Pay for high-speed circuits to interconnect our four institutions

• Two tests:– Application, middleware, transport– Dynamic circuit setup/release

04/22/23 13

Documentation plans• Overall architecture – Feb. 28, 03• Detailed specifications

– Signaling module – March 31, 03– Routing decision module – April 30,

03– Database – March 31, 03

14

Detailed plan - UVA

• Dynamic circuit setup/release• Equipment to purchase• What we plan to implement• Wide-area circuits

• High-throughput transport protocol item• Support for delay-controlled transport

protocol• Support for router disconnect

Malathi Veeraraghavanmv@cs.virginia.edu

04/22/23 15

Lessons learned from demos

• Losses not only due to link errors• Losses occur when receive buffer

overflows– With rate control based flow control, set

circuit and sending rate to the receive rate

– Nevertheless, if other tasks are scheduled on the receive PC, the receive rate is not a guaranteed steady rate

04/22/23 16

Follow-up steps: Step 1• Investigate receive buffer overflow

losses– if window based flow control is used,

why is the sender not just shutdown when the receive buffer is full?

– if timed correctly, should have idle circuit rather than losses

– use tcpdump to analyze

04/22/23 17

Step 2• Work with application-level

transport implementation to achieve the desired set of features for memory-to-memory transfers– e.g., retransmissions on circuit unless

needed at the end, etc.

04/22/23 18

Step 3• Disk-to-disk:

– Stay with application-level transport impl.– But use zero-copy patch of Linux

• it removes need for copy from kernel memory to application memory

• sendfile system call• is there a corresponding recvfile call?

– Achieve max. rate possible with IDE disks or high-end disks

• 135MB/sec with dual channels > 1Gbps

04/22/23 19

Step 4• Test kernel-level transport

implementations– Net100 implementation of TCP can be

tuned to run with no congestion control– How about NAKs and rate-based flow

control?– Retransmissions at the end on TCP/IP

path

04/22/23 20

O/S bypass implementation

• This requires implementation of ST on processor/FPGA on Ethernet card

• If ST is run on top of Ethernet, then it is not an OS bypass implementation– Ethernet driver moves payload from NIC to

kernel memory• This is a step beyond kernel-level impl!• Leave task for a DOE proposal with

LANL for 10GbE

04/22/23 21

Equipment for transport protocol implementations

• Already purchased: Two 64-bit 100Mhz bus PCs with GbE

• Augment with – High-end disks

• Purchase one more PC with high-end disks

22

Detailed plan - UVA

• Dynamic circuit setup/release• Equipment to purchase• What we plan to implement• Wide-area circuits

• High-throughput transport protocol item• Support for delay-controlled transport

protocol• Support for router disconnect

Malathi Veeraraghavanmv@cs.virginia.edu

04/22/23 23

Plans for measurements• Purchase a high-end workstation with

large disk space to ship to NCSU for data collection – Remote visualization application

• To determine if a low-speed (10 or 100Mbps) bidirectional circuit can be established and left open even during think times

• Establish high-speed unidirectional circuit for large downloads on-demand

24

Detailed plan - UVA

• Dynamic circuit setup/release• Equipment to purchase• What we plan to implement• Wide-area circuits

• High-throughput transport protocol item• Support for delay-controlled transport

protocol• Support for router disconnect

Malathi Veeraraghavanmv@cs.virginia.edu

04/22/23 25

Good news• Just heard that we won an

Internet2/Cisco grant of two Cisco 12000 series routers– Loan for one year– Renewable– Will be shipped soon after

Thanskiving

04/22/23 26

Plans for personnel• Senior graduate student: Xuan Zheng• Offered RA to a new PhD student

starting from the Spring semester• Will select two more MS/Phd students

from CS students admitted in Fall 2003

• Advertise post-doc position• Created web site