fpx network platform 1 john lockwood, [email protected] assistant professor washington...

FPX Network Platform 1

John Lockwood,[email protected]

Assistant Professor

Washington UniversityDepartment of Computer Science

Applied Research Lab1 Brookings Drive

Saint Louis, MO 63130

http://www.arl.wustl.edu/arl/projects/fpx/workshop_0801/agenda.html

Supported by: NSF ANI-0096052 and Xilinx Corp.

Field-programmable Port Extender (FPX)August 2001 Workshop


Workshop Objectives

• Learn to accelerate network processing with reprogrammable hardware

• Understand System-On-Chip design

• Perform Hardware/Software Co-design

• Obtain hands-on experience with the Field Programmable Port Extender (FPX)


Outline

• Motivation – Hardware enables packet processing at link speed– FPGAs provide flexibility for dynamically reconfiguration– Networks can be extended with FPGAs to provide

enhanced functionality and performance.

• Research Results– Field programmable Port Extender (FPX) allows modules

to be dynamically installed in a network.– FPX serves as an open platform for rapid prototype of

firewall and router plug-in modules– Courses and Workshops held at Washington University to

develop new System-On-Chip networking modules


Workshop Activities

• Integrate FPX Infrastructure Components– SDRAM Memory Controller– Internet Protocol Wrappers

• Operate FPX Software Tools– NCHARGE Control Software– PARBIT FPGA Tools

• Implement Networking Modules on the FPX– Cell processing module in VHDL– Packet Processing Module– KCPSM Active Networking Module


FPX Workshop Agenda: Times and Location

– Wednesday, Aug 15, 2001 • 8am: Breakfast

– 5th floor Jolley Atrium

• 9am-Noon: Session I – Sever 201 Lab

• Lunch – WashU Campus

• 1pm-5pm: Session II– Sever 201 Lab

– Thursday, Aug 16, 2001

• 8am: Breakfast – 5th floor Jolley Atrium

• 9am-Noon: Session III – Sever 201 Lab

• Lunch – 5th floor Jolley Atrium

• 1pm-5pm: Session IV – Sever 201 Lab

On-line Agenda:http://www.arl.wustl.edu/arl/projects/fpx/workshop_0801/agenda.html


Building Networks with Reprogrammable hardware


Technology Options for Evolvable Internet Hardware

...

...

...

... ...

......

Reprogrammable Hardware

Network Processors

Fle

xibi

lity

Performance

FullyReprogrammable

HighPerformance

Microprocessor

ASIC


Reprogrammable Device Configuration

GRMLocal R outing

CLB PIP

• Routing Module : • Interconnection of Blocks

4 LUT

G4

G3

G2

G1

G

4 LUT

F4

F3

F2

F1

F

3 LUT

H

S

R

D Q

S

R

D Q

H1

Din Clk

YQ

Y

XQ

X

M

M

M

M

CLB • CLB : • Primitive element of FPGA

...

...

...

... ...

......

3rd Generation LUT-based FPGA

Pad Routing CLB M atrix I/O

M acroBlock(uP,M em)

• FPGA : • Matrix of CLBs and

Routing Modules


The FPX Network Platform


Configuration of Network Packet Processor

• Packet processing hardware performs:– Packet classification– Packet forwarding– Address Translation– Data modification– Packet buffering– Active Networking (Application-level data processing)

FPX

ExtenderPort

programmableField- Card

OC3/OC12/OC48

LineCardOC3/OC12/OC48

Line

NetworkPackets

NetworkPackets


Field Programmable Port Extender


Configuration of Internet Router

• Additionally, Router interface performs:– Internet route lookup– Traffic policing and shaping

IPP

IPP OPP

OPP

Switch

Fabric

Gigabit

IPP

IPP

OPP

OPP

FPX

ExtenderPort

programmableField-Card

OC3/OC12/OC48

Line

NetworkPackets

FPX

ExtenderPort

programmableField-Card

OC3/OC12/OC48

Line

NetworkPackets


Port Processing at edge of Gigabit Switch

• Original Network Switch– Line card connects to

Gigabit switch backplane

Fiber Optics


FPX

Line Card

SDRAMs

NID RAD SRAMs

VRM

Gigabit switch backplane SDRAMsVRM

FiberOpticsLine Card

• FPX-Enhanced Router– Line card connects to



Complete Router Platform


Combination Router Hardware and Software

• Implement link speed opertions on hardware• Implement higher-level functions in software• Migrate functionality on the critical path

ExtenderPort

programmableField-

FPXExtender

Field-

Port

IPP

programmable

OPP

IPP

IPP OPP

OPP

IPP

FabricSwitchGigabit

WUGS

OPPFPXCardOC3/

OC12/OC48

Line

Card

SPCSmartPort

Card

SPCSmartPort

CardOC3/

OC12/OC48

Line


Port Configuration

SRAM

To/

Fro

m o

ther

net

wor

k po

rts

InterfaceProgramming

SelectMap

Interfacetemp

AsynchronousInterface

FPX

Switch

Three Port

APICM

odul

e

Pentium

Line

Car

d

SPC

Hardware-basedPacket Processing

Virtual Circuit

Mod

ule

Software-basedPacket Processing

NID

RAD

WUGS

Switching

Lookup Table

ECEC

ccp

VC VC

VC

Error Check

VC

Circuit

Control CellProcessor

Switch

Four Port

Synch, Buffered

Data

SRAM

SRAMData

BridgeSouth

RAD

SDRAM

SRAM

Program

Bridge

North

DataData

SDRAM

Data

ModuleIntel Embedded

PCI Bus

PCI Interface


The FPX Architecture


Architecture of the FPX • RAD

– Large Xilinx FPGA– Attaches to SRAM and

SDRAM– Reprogrammable over

network– Provides two user-defined

Module Interfaces

• NID – Provides Utopia

Interfaces between switch & line card

– Forwards cells to RAD– Programs RAD

SRAM

EC

Mo

du

le

EC

NID

Switch LineCard

Mo

du

le

RAD

VC VC

VCVC

RAD

Program

SDRAM SDRAM

Data Data

SRAM

Data

SRAM

Data


Field Programmable Port Extender

– NID : Network Interface Device– RAD : Reprogrammable Application Device


FPX SRAM

– Provide low latency for fast table-lookups– Zero Bus Turnaround (ZBT) allows back-to-back

read / write operations every 10ns– Dual, Independent Memories – 36-bit wide bus


FPX SDRAM

– Dual, independent SDRAM memories– 64-bit wide, 100 MHz– 64MByte / Module : 128 Mbyte total [expandable]– Burst-based transactions [1-8 word transfers]– Latency of 14 cycles to Read/Write 8-word burst


Hardware Device


FPX Module Interface


FPX Interfaces Provides

• Well defined Interface– Utopia-like 32-bit fast data interface– Flow control allows back-pressure

• Flow Routing– Arbitrary permutations of packet flows through ports

• Dynamically Reprogrammable– Other modules continue to operate even while new

module is being reprogrammed

• Memory Access– Shared access to SRAM and SDRAM– Request/Grant protocol


Reprogrammable Application Device (RAD)

• Spatial Re-use of FPGA Resources– Modules implemented using FPGA logic– Module logic can be individually reprogrammed

• Shared Access to off-chip resources– Memory Interfaces to SRAM and SDRAM– Common Datapath to send and receive data

Mo

du

le

Mo

du

le

Network Interfaces to NID

RAD

SRAMData

SRAM

SDRAM SDRAM

Data Data

Data


Network Module Hardware Interface

SRAM_D_OUT[35:0]SRAM_ADDR[17:0]SRAM_WR_RD

SRAM_REQSRAM_GR

SRAM_D_IN[35:0] SRAMInterface

FPXNetworkModule

READY_L

CLKRESET_L

ENABLE_L

ModuleInterface

SDRAM_RQ

SDRAM_BL[4:0]SDRAM_ADDR[26:0]SDRAM_WR_RD

SDRAM_GRSDRAM_DATA[63:0] SDRAM

Interface

SDRAM_EN

SDRAM_OP_FIN

TCA_MOD_OUT

D_MOD_IN[31:0]SOC_MOD_IN

D_MOD_OUT[31:0]SOC_MOD_OUT

TCA_MOD_IN

DataInterface

fpx_module.vhd


RAD Module Interface

• Cell I/O and Flow Control– 32-bit wide UTOPIA-style interface w/ unique timing

• Off-chip Memory Access– Arbitrated access to SRAM and SDRAM via standard

interface• Control (clock, reset, and reconfiguration control)


Infrastructure Services


Routing Traffic Flows Between Modules

• Functions– Check packets for errors– Process commands

• Control, status, & reprogramming

– Implement per-flow forwarding

NID

LineCardSwitch

EC ECEC

VC VC

VC VC

VC

ccp

ccp

• Traffic flows routed among– Switch– Line Card– RAD.Switch– RAD.Linecard


Example NID Routing Configurations

• Hardware Module Bypass (Default)

0VC

EC

LineCard

VC

ccpEC

VC VC

Switch

NID

RAD

No Modules Installed

1

2 3

Default Flow Action(Bypass)

0VC

EC

LineCardSwitch

VC

ccpEC

VC VC

Mod

ule One

ModuleInstalled

2 3

1

(Per-flow Output Queueing)Egress (SW) Processing

NID

RAD

0VC

EC

VC

ccpEC

VC

LineCardSwitch

VC

Mod

uleOne

InstalledModule

1

2 3

(IP Routing)Ingress (LC) Processing

NID

RAD

00

VC

ccp

VC

EC

VC VC

LineCardSwitch

Mod

ule

Mod

ule

EC

1

2 3

Full RAD Processing(Packet Routing and Reassembly)

NID

RAD

0

Mod

ule

Mod

ule

VCVC

ECccp

VC

EC

VC

LineCardSwitch

1

2 3

Full Loopback Testing(System Test)

NID

RAD

0

Switch

Mod

ule

Mod

ule

VCVC

ECccp

VC

EC

VC

LineCard

2 3

1

(Chained Modules)Dual Egress Processing

NID

RAD

• Egress Processing

• Ingress Processing

• Ingress+Egress

• Loopback

• Chained Egress Processing


NID Flow Entry Example

• Problem:– Route flow on VCI 45

from Switch to Mod_sw– Route Flow on VCI 45

from Mod_sw to LC– Route Flow on VCI 45

from Linecard to Switch

• Solution– VCI [45] = 1,3,X,0

0VC

EC

NID

VC

ccpEC

VC

LineCardSwitch

VC

RAD

Mod

uleOne

InstalledModule

1

2 3

(IP Routing)Ingress (LC) Processing

__ __ __ __VCI 45

d1 d2 d3 d4


FPX Control : NCHARGE


Fpx_control{0-7}.{0/1}

Pictorial view of fpx_control interfaced with hardware

Control Cell Format for 32/36 bit RAD SRAM Memory Operations

OpCode = 0x15 SRAM Memory Operation ResponseOpCode = 0x14 SRAM Memory OperationVPI = 0x000, VCI = 0x0023 (35) RAD Control Cell

N/AX C(7:5)

N/AModuleIDOpCode

WORD 1

FRDV

1

F - 32 or 36 bit: 1 = 36 bit, 0 = 32 bit

FRDV

R - Read or Write: 1 = Read, 0 = WriteD - Device: 1 = Device 1, 0 = Device 0V - Valid Command: 1 = Valid command, 0 = Invalid, EOC

32 Bit format'N' Address

36 Bit format1 Address

36 Bit format2 Address

WORD 0 (31:0)

WORD 1 (31:0)

WORD 0 (31:0)

V FRD

HEC

ADDR(18:0)

ADDR(18:0)

ADDR(18:0)

WORD N

CRCSequence #

CM DATA

WORD 0

N/AW0(35:32)

W1(35:32)W0(35:32)

PTI

0

PAD

VCIGFC / VPI

OpCode

HEC

HDR

PL2

PL3

PL4

PL5

PL6

PL7

PL8

PL9

PL10

PL11

PL1

12345678910111213141516171819202122232425262728293031 0

ModuleID = 0x00 RAD Control Cell Processor

SwitchController


Controlling the FPX

• Methods of Communication- NCHARGE- Telnet- Web Interface / CGI- Basic_send- User Applications

• Software Plug-ins- Concepts- Functionality

• Emulation– Nid_listener– Rad_listener

BasicSend

CGI

Fip MemoryManager

AccessWEBBasic Telnet

Send

NID NID

RAD

0.0

Gigabit Switch

OC-3 Link

NCHARGE 7.1

SoftwareController

Fip

RemoteApplications

VCI 76 (NID), VCI 100 (RAD)

VCI 115 (NID), VCI 123 (RAD)

(up to 32 VCIs)

Read

Washington University

RAD

NCHARGE


Web Access to NCHARGE

- Radio Button Interface- Allows user to submit commands using CGI scripts- Provides for Switch Reset- http://fpx.arl.wustl.edu

Web Access Provides:


FPX Control and Reconfiguration


Switch Element

IPPIPPIPPIPPIPPIPPIPP

OPPOPPOPPOPPOPPOPPOPPOPPIPP

LCLCLCLCLCLCLCLC

LCLCLCLCLCLCLCLC

WUGS

Example: Write Configuration Memory

R A DReprogrammable

ApplicationDevice

Virtex1000E fg680

S D R A M

(ba cks id e)

8Mb

it Z

BT

SR

AM

8Mb

it Z

BT

SR

AM

S D R A M

(ba cks id e)

O SC6 2.5 M H z

O SC1 00 M H z

DeviceInterfaceNetworkN ID

R ADP rogram

FIFO E PR O MN ID

WU

GS

Sw

itch

Bac

kpla

ne C

onn

ect

or

OC

3 /

OC

12

/ OC

48 L

inec

ard

Con

nect

or

(backside)

V R M

V R M

1.8V

2.5V(backside)

P C B Trace D e ns ity

FPX

VPI

PTI = 0 0

VCI = 0x34

HEC

Reserved

7 06 5 4 2 13

CRC

Sequence #

CMData

OPCODE

Control CellSwitchController

• Switch Controller generates command to write 32-byte data element to RAD Configuration Memory

ccp

• Control Cell sent containing Memory Address and Data

• Switch Element routes control cell to FPX

• NID Element on FPX writes Data from Payload into Rad Configuration Memory

RAD Configuration Memory


Reprogramming Logic

R A DReprogrammable

ApplicationDevice

Virtex1000E fg680

S D R A M

(ba cks id e)

8Mb

it Z

BT

SR

AM

8Mb

it Z

BT

SR

AM

S D R A M

(ba cks id e)

O SC6 2.5 M H z

O SC1 00 M H z

DeviceInterfaceNetworkN ID

R ADP rogram

FIFO E PR O MN ID

WU

GS

Sw

itch

Bac

kpla

ne C

onn

ect

or

OC

3 /

OC

12

/ OC

48 L

inec

ard

Con

nect

or

(backside)

V R M

V R M

1.8V

2.5V(backside)

P C B Trace D e ns ity

• Switch Controller writes RAD configuration memory to NID

• NID reads RAD config memory to program RAD

• NID programs at boot from EPROM

– Bitfile for RAD arrives transmitted over network via control cells

– Performs complete or partial reprogramming of RAD

• Switch Controller issues {Full/Partial} reconfigure command

Switch Element

IPPIPPIPPIPPIPPIPPIPP

OPPOPPOPPOPPOPPOPPOPPOPPIPP

LCLCLCLCLCLCLCLC

LCLCLCLCLCLCLCLC


System-On-Chip Design using Dynamic Hardware Plugins (DHP)


Module Loading / Unloading

FP

X M

od

ule

FP

X M

od

ule

FP

X M

od

ule

SD

RA

MS

RA

M

FPGA’s Long Lines

Intrachip Module Switching

...

Data

Combining Modules within the Chip

• Modules fit together at static I/O interfaces

• Partial reprogramming of FPGA used to install/remove modules

• Modules added and removed while other modules process packts

• Statically-configured ‘Long Lines’ provide chip-wide routing


Implementing modules in Virtex1000E•Virtex 1000E logic resources

–Globally accessible IOBs–64 x 96 CLB array

• 4 flops/LUTs per CLB–96 Block SelectRAMs

• 4096 bits per block• 6 columns of 16 blocks• 6 columns of dedicated interconnect

Ingress Path Egress Path

Dou

ble

DH

P M

odu

le

Dou

ble DH

P M

odule

DH

P M

odu

le

DH

P M

odu

le DH

P M

odule

DH

P M

odule

•DHP Modules–64 x 12 CLB array(768 CLBs, 3072 flops)

•Double DHP Modules–64 x 24 CLB array(1536 CLBs, 6144 flops)

•16 BRAMs (8KB) per Module•3 DHP Modules per path•1 SRAM interface per path•1 SDRAM interface per path

IOB RingVersaRing

CLB columns BRAMsBRAM Interconnect


Dynamic Hardware Plugins [On RAD FPGA]

• Module Synthesis Constraints– Infrastructure

with target regions reserved for DHP modules insertions

– PARBIT parameters

• Target Locations (Row, Col)

Right IO

BsL

eft I

OB

s

Top IOBs

RA

M

RA

M

RA

M

RA

M

RA

M

RA

M

TARGET 1 TARGET 2 TARGET 3 TARGET 4T1 (7,8) T2 (7,20) T3 (7,68) T4 (7,80)

Bottom IOBs

INFRASTRUCUTURE


Example Application : String Processing


“Hello, World” Module Function

P46

.. Payload ..

P45 P47P44 P47

.. Payload ..

Compare

P45 P44 P46

'L'

VCI=5

[48 bytes in12 words]

Header[5 bytes in

2 words]

Payload

'H' 'E' 'L' 'L'

'O' ' ' 'W' 'O'

PAD

'R' 'L' 'D' '.'

'H' 'E' 'L'

'O' P5

VPI VCI=5

PAD

VPI=X

P8 P9 P10 P11

P6 P7

... Copy

32 bits

Copy

Write

Match+Write

Match

Match

...

JWL:ARL 07/00


Evolvable Internet Hardware

Place and Route FPGA

Gates(Xilinx)

Module to RAD(Ncharge)

Download

SynthesizeTo Logic

(Synplicity)

Compile Design(Vcom)

Observe NetworkBehaviour

Through Module(Ncharge)

Route Traffic

TweakDesign


Modular Interface to SDRAM


SDRAM Controller Overview

Module 0

Module 1

Module 2

SDRAM

SDRAM Controller


Example : Leaky Bucket

- Buffers the incoming cells in a FIFO - Generates tokens at a regular interval

- Gives out a cell from the FIFO when the number of tokens > 0 - Destroys a token when a cell is given out

bursty data leaking data


Internet Protocol Wrappers


Payload Processing Environment

– Higher-Level data processing on the FPX– Wrapper Framework

NetApp

Wrapper

Wrapper


Frame / IP Packet / UDP / Application Layers

UDP Processor

IP Processor

Cell Processor

Frame Processor

Data Output

Data Input

Application-levelHardware Module

Interfaces to Off-Chip Memories


Soft-core Active Network Processor : The KCPSM Network Module


The FPX KCPSM Module

KCPSM

DATA

MEMORY

PROGRAM

MEMORY

ADDR INST

INTERFACED_MOD_IN

CONTROL

ADDR DATA

PROTOCOL WRAPPERS

INSTADDRPORT_IDBUSI/O

BUS

UD

P P

AC

KE

TS

AT

M C

EL

LS

D_OUT_MOD

CONTROLSIGNALS

UD

P P

AC

KE

TS

AT

M C

EL

LS

ENABLE_L READY_L

SOC_MOD_IN

TCA_MOD_IN

SOC_OUT_MOD

TCA_OUT_MOD

D_OUT_MODD_MOD_IN

CLKRESET_L

SIGNALS


Simulating a KCPSM Program

• The input data cell

• The output data cell

The 1st – 2nd word is the ATM Cell Header


The 3rd – 7th word is the Internet Protocol Header


The 8th – 9th word is the UDP Header


The 10th word specifies it is a data packet


The 11th – 13th word is the data string ‘Hello World’

The 11th – 13th word is the data string ‘Hello World’









The 11th – 13th word is the data string ‘Uryyb Jbeyq’

The 11th – 13th word is the data string ‘Uryyb Jbeyq’


Applications for the FPX: Fast IP Lookup (FIPL)


Fast IP Lookup Algorithm

– Function:• Search for best matching prefix using Trie algorithm

– Contributors• Will Eatherton, Zubin Dittia, Jon Turner, David Taylor, David Wilke,

Prefix Next Hop*

01*47

10* 2110* 90001* 11011* 000110* 501011* 3

0 1

0

0 0

0

0

0

1 1

1

1 1

1

1

1

1


Hardware Implementation in the FPX

SRAM1

SRAM2

IP LookupEngine

counter

On-Chip Cell Store

SRAM1 Interface

Control CellProcessor

PacketReassembler

RAD FPGA

NID FPGA

ExtractIP Headers

Remap VCIsfor IP packets

LC SW

RequestGrant0 1

0

0 0

0

0

0

1 1

1

1 1

1

1

1

1


Fast IP Lookup (FIPL) ApplicationRoute add 141.142.5.0/24 8Route delete 141.142.0.0/16

Controlcells

FIPLFast IP Lookup Lookup

(X.Y.Z.W)Nexthop

FPGA

RAM External SRAM

FIPLMemory Manager

Software

Commands

HardwareLookup


Conclusion

• Field programmable Port Extender (FPX) Platform

– Open Platform for hardware development• Modular Interfaces

– Allows integration components to build System-on-Chip (SoC)

• Library of Internet packet processing functions– Simplifies design of new functionality

• Interoperable w/existing software systems– Unifies Active Networking Hardware with Software

– FPX Homepage• http://www.arl.wustl.edu/arl/projects/fpx/


Acknowledgements

• Several Individuals have contributed to this work:

– Washington University• Jon Turner• Sarang Dharmapurikar • Todd Sproull• David Taylor• Florian Braun• Henry Fu• Dave Lim• Edson Horta

– Xilinx• Dave Parlour

fpx network platform 1 john lockwood, [email protected] assistant professor washington...

Documents