internet in a box building distributed system testbed in ramp zhangxi tanxiaofan jiang computer...

Internet in a Box Building Distributed System Testbed in RAMP

Zhangxi Tan Xiaofan JiangComputer Science Department

University of California, BerkeleyElectrical Engineering DepartmentUniversity of California, Berkeley

2CS268 Spring 2006

Introduction Testing and debugging large scale distributed

systems is difficult Problem with existing approaches

Scalability – O(100) nodes Reproducibility - PlanetLab Observability - Don’t know what’s going on inside Cost, Space and Power

Build distributed system testbed with over 1,000 nodes using multi modular FPGA system Building block, HW/SW and networking infrastructure

this semester

3CS268 Spring 2006

Methodology Intermediate platform –

XUP Fit 6/8 RISC processors per

chip Running uClinux with

TCP/IP network Software routers on chip Sample distributed

applications (Chord) Final platform - Research

Accelerator for MultiProcessing (RAMP) Scale up to 16 RISC (?)

processors per chip x 4 = 64 cores

4CS268 Spring 2006

Previous Status Single processor

running uClinux 2.4 on XUP board Running @ 100 Mhz,

256M DDR Zebra open source

router (not fully tested) Virtual NIC and point-to-

point communication between processors

Support polling and interrupt

Firmware has been developed

5CS268 Spring 2006

Current Status Overview Multiple processors on one chip

6 cores on XUP running uClinux Multiprocessor debugging and bootstrapping XUP Clusters!

Network Support Ethernet Linux driver and TCP/IP network (Socket interface) Network Interface Card (NIC)

TCP/IP interconnect network Transparent internal inter-core network and external inter-XUP

network Demo application porting and performance evaluation

TTCP Chord / i3 (after porting) Web server w/CGI support

6CS268 Spring 2006

System Architecture

0GLB

1GLB

2GLB

3GLB

4GLB

5GLB

6GLB

7GLB

8GLB

9GLB

10GLB11GLB12GLB13A 14A 15A 16A 17A 18A 19A

20A

21A

22A

23B

24B

25B

26B

27B

28B

29B

30B 31B 32B 33B 34B 35B 36C 37C 38D 39E

opb2plb_bridgeopb2plb_bridge_0

SOPBMPLB


SOPBMPLB


SOPBMPLB


SOPBMPLB


SOPBMPLB


SOPBMPLB

ilmb_1dlmb_1

bram_blocklmb_bram_1

PORTAPORTB

lmb_bram_if_cntlrdlmb_cntlr_1

SLMBBRAMlmb_bram_if_cntlrilmb_cntlr_1

SLMBBRAM

C

opb_uartliteRS232_Uart_1

SOPB60

MyNICmicroblaze_1_NIC1023

SOPB0010203040

opb_intcopb_intc_1

SOPB0

opb_timeropb_timer_1

SOPB70

SLAVES OF mb_opb_1

microblazemicroblaze_1

ILMBDLMB

DOPBIOPB

SFSL0

MFSL0

SFSL1MFSL1

SFSL2

MFSL2

SFSL3

MFSL3SFSL4

MFSL4

0

PROCESSOR

ilmb_0dlmb_0


PORTAPORTB



SLMBBRAM

A

plb_ddrDDR_256MB_32MX64_rank1_row13_col10_cl2_5

SPLB

SLAVES OF plb


SOPB11


SOPB01

opb_intcopb_intc_0

SOPB1


SOPB21

SLAVES OF mb_opb_0


DLMBILMB

DOPBIOPB

MFSL0SFSL0

1

PROCESSOR

ilmb_3dlmb_3


PORTAPORTB



SLMBBRAM


SOPB12


SOPB02

opb_intcopb_intc_3

SOPB2


SOPB22

SLAVES OF mb_opb_3


ILMBDLMB

DOPBIOPB

SFSL0MFSL0

2

PROCESSOR

ilmb_2dlmb_2


PORTAPORTB



SLMBBRAM


SOPB13


SOPB03

opb_intcopb_intc_2

SOPB3


SOPB23

SLAVES OF mb_opb_2


DLMBILMB

DOPBIOPB

MFSL0SFSL0

3

PROCESSOR

ilmb_5dlmb_5


PORTAPORTB



SLMBBRAM


SOPB14


SOPB04

opb_intcopb_intc_5

SOPB4


SOPB24

SLAVES OF mb_opb_5


ILMBDLMB

DOPBIOPB

SFSL0MFSL0

4

PROCESSOR

ilmb_4dlmb_4


PORTAPORTB



SLMBBRAM


SOPB15


SOPB05

opb_intcopb_intc_4

SOPB5


SOPB25

SLAVES OF mb_opb_4


DLMBILMB

DOPBIOPB

MFSL0SFSL0

5

PROCESSOR

MASTER_SLAVEopb_ethernetEthernet_MACB

MSOPB

IP

util_vector_logicsysclk_inv

opb_mdmdebug_module

util_vector_logicddr_clk90_inv

dcm_moduledcm_1E

dcm_moduledcm_0D

util_vector_logicclk90_inv

0NIC_01

0NIC_21

0NIC_41

0NIC_51

0NIC_12

NIC_13

1NIC_14

2NIC_15

NIC_010NIC_13 0NIC_12 NIC_21 2NIC_15 NIC_511NIC_14 NIC_41NIC_10 NIC_10NIC_31 NIC_31

mb_opb_0

mb_opb_1

plb

mb_opb_2

mb_opb_3

mb_opb_4

mb_opb_5

KEYSYMBOLS

bus interface

shared bus

Bus connections

master or initiator

slave or target

master slave

monitor

External Ports

input

output

inout

COLORSBus Standards

DCR

FCB

FSL

LMB

OPB

PLB

SOCM

XIL (prefix) P2P

GEN. P2P, USER, etc

SPECSEDK VERSION 8.1.01

ARCH virtex2p

PART xc2vp30ff896-7

GENERATED Sat Apr 29 05:55:11 2006

7CS268 Spring 2006

A XUP cluster

ROUTER

uB

uB

uB

uB

uB uB

uB

uB

uB

uB

uB uB

uB

uB

uB

uB

uB uB

OTHER DEVICES

OTHER DEVICES

TO EXTERNAL NETWORK

8CS268 Spring 2006

i3 / Chord / DHT

9CS268 Spring 2006

Available Applications / Utilities

chmod df du find grep ifconfig iptables-restore lsmod netstat pwd route telnetd traceroute vi zcat

agetty cmp dhclient echo flatfsd gunzip ifdown iptables-save mkdir nslookup rdate rsyslogd test true vmstat

arp cp dhcpd egrep free gzip ifup kill modprobe null reboot sed tftp ttcp wc

arping crond dhcrelay env ftp hd inetd killall more passwd reset sh thttpd umount wget

basename crontab dmesg expand ftpd head init ln mount ping rm tail time uname which

busybox date dpkg false ftpget hostname insmod login msh portmap rmdir tcpdump top uptime whoami

cat dd dpkg-deb fgrep ftpput i3serverd iptables ls mv ps rmmod telnet touch version xargs Python

10CS268 Spring 2006

Challenges Porting apps is a painstaking job

Configuration/Makefile problem Cross-compiling (Dynamic linking library, CPU type and etc.) Fork (using pthread library might be an alternative)

More cores? 4-core version is quite stable now 6-core version (near 100% BRAM and Slices

utilization) Hitting the power-wall without heat sink 100 Mhz is difficult Building and debugging is much more difficult Place and route time is unbearable! (floor plan tools required)

11CS268 Spring 2006

Future Work

8 cores?Migrating design to RAMP (some

dependencies)And more…

internet in a box building distributed system testbed in ramp zhangxi tanxiaofan jiang computer...

Documents

cs268 spring

berkeley slide

xup cluster slide

semester slide

system architecture

i3 chord dht slide

chip running uclinux

xup board