© 2002

Implementing Switch Fabrics in FPGAsImplementing Switch Fabrics in FPGAs

AgendaAgendaStratix GX IntroductionSystem OverviewCentralized Switch FabricsDistributed Switch FabricsSummary

Stratix GX IntroductionStratix GX Introduction

Transceiver Channel

Logic ArrayBlocks (LABs)

M512 Blocks

M4K Blocks

M-RAM Block

DSP Blocks

Transceiver Blocks

I/O Elements (IOEs)

Phase-Locked Loops (PLLs)

Source-Synchronous I/O

Channels With Dynamic Phase

Alignment

IOEs Support PCI, Memory Interfaces

IOEs Support PCI, Memory Interfaces

Typical Networking EquipmentTypical Networking Equipment

Router,L2 Switch,L3 Switch

Line Cards

Host Card

Switch Fabric Card

Typical Data FlowTypical Data Flow

PHYPHY Framer/Framer/MACMAC

Traffic Traffic ManagerManager

Queue/Queue/BufferBufferMgmtMgmt

PHYPHY

PHYPHY

PHYPHY

Framer/Framer/MACMAC

Framer/Framer/MACMAC

Framer/Framer/MACMAC

Packet Packet Processing Processing

& Forwarding& Forwarding

Control Control ProcessorProcessor

Line Card

Tran

scei

ver

Tran

scei

ver

Switch Fabric Card

SwitchSwitchFabricFabric

SchedulerScheduler

Control FunctionData Path Function

Tran

scei

ver

Tran

scei

ver

Backplane4 x 3.125 Gbps

Centralized Switch Fabrics− Each Line Card Connects to

Switch Fabric Card− Switch Fabric Card Routes

Data− Redundant Switch Fabric

Cards Needed

Switc

h Fa

bric

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G Backplane

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G BackplaneDistributed Switch Fabrics− Each Line Card Connects

to Every Other Line Card− Switch Module on Each

Line Card Routes Data− No Need for Redundancy

Backplane Architectures Backplane Architectures

© 2002

Centralized Switch FabricsCentralized Switch Fabrics

Switch Fabric Card

Crossbar,Crossbar,SchedulerScheduler

Tran

scei

ver

Tran

scei

ver

Backplane

Port 1

Port 2

Port 3

Port 4

Port 5

Port 1 Port 2 Port 3 Port 4 Port 5

Inputs

Outputs

Effective in Balanced Traffic−Traffic Evenly Spread among

All Line CardsIneffective When Individual Ports Have High Traffic

−Backlog of Packets

Centralized: Crossbar FabricsCentralized: Crossbar Fabrics

Crossbar FunctionCrossbar Function

Crossbar

Port 1

Port 2

Port n

Scheduler/Arbitrator

Line CardLine Card

Line CardLine Card

Line CardLine Card

Crossbar Function – Reverse RouteCrossbar Function – Reverse Route

Line CardLine Card

Line CardLine Card

Line CardLine Card

Port 1

Port 2

Port n

Scheduler/Arbitrator

Port n Calls for

Data from Port 1

Crossbar Function – Reverse RouteCrossbar Function – Reverse Route

Ingress

Egress

Line CardLine Card

Line CardLine Card

Line CardLine Card

Port 1

Port 2

Port n

Scheduler/Arbitrator

MUX Structure

Repeated for All Ports

Queue

Crossbar

Head-of-Line Blocking ProblemHead-of-Line Blocking ProblemScheduler/Arbitrator

Line CardLine Card

Line CardLine Card

Line CardLine Card Port n

Port 1

Port 2Queue22--nn

11--22 11--nn

Head-of-Line Blocking:

Packet ‘1-2’ Waits for Packet‘1-n’ to Be Transmitted

Queue* AA--BB

A: Input PortB: Output Port

*Queuing Can AlsoOccur on Line Cards

Crossbar

Scheduler/Arbitrator

Line CardLine Card

Line CardLine Card

Line CardLine Card

Port 1

Port 2Queue22--nn

Head-of-Line BlockingEliminated with

Separate Output Queuesfor Every Output Port

Virtual Output Queuing (VOQ)Virtual Output Queuing (VOQ)

Port nQueue*

*Queuing Can AlsoOccur on Line Cards

AA--BBA: Input PortB: Output Port

11--2211--nn

11--2211--nn

Switch Fabric Operates on Time SlotsSwitch Fabric Must Be “Configured” During Each Time Slot

Methods of Scheduling Packets/Cells for Switch Fabrics− SLIP, Islip, Parallel Iterative Matching (PIM) , Least Recently Used

(LRU)

Switch Fabric Scheduling MethodsSwitch Fabric Scheduling Methods

3

n

IngressPort 3

IngressPort n

EgressPort 1

33--11

Scheduler

Stratix GX Devices for CrossbarsStratix GX Devices for CrossbarsSupports Many Links in One Device− Up to 20 Integrated 3.125-

Gbps TransceiversOutput Port Queuing− Up To 183 M4K RAM

Blocks(183 x 4KBits)

More Queuing− Up to 4 M-RAM Blocks (4 x

512 Kbits)Scheduling & Flow Control Functions− Over 41K Logic Elements

(LEs)

3.125 GbpsTransceiver Channel

Logic ArrayBlocks (LABs)

M512 Blocks

M-RAM Block

Source-Synchronous I/O Channels

with DPA

M4K Blocks

Centralized Switch Fabric Centralized Switch Fabric

Switc

h Fa

bric

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G

Lin

e C

ard

10G Backplane

Backplane Transceiver

Queue/Buffer Management

Traffic Management

Backplane Transceiver

Switch Fabric

Stratix GX Crossbar ImplementationStratix GX Crossbar Implementation Switch Fabric Card

20 x 3.125 Gbps Backplane

Programmable Scheduler− Design to Your Traffic

PatternIntegrated Transceiver, Crossbar & SchedulerBuffering Capabilities

to Line Cards

1834,096 BitsM4K512 KBits

512 Bits

Size

4M-RAM

384M512

Number of Blocks in

EP1SGX40

RAM Block

Tran

scei

ver

Tran

scei

ver FIFOFIFO

FIFOFIFO

FIFOFIFO

FIFOFIFO

Scheduler

Crossbar Sample ImplementationCrossbar Sample Implementation8x8 Crossbar FabricFixed Cell Size16-Bit Data PathVOQ Depth: 8 CellsOutput Queue Depth: 32 CellsRound Robin Scheduler

160 MHzfMAX

8Transceiver Channels

10KLogic Elements96M4K RAM Blocks

Design Results

Switch Fabric Card

8 x 3.125 GbpsBackplane

to Line Cards

Tran

scei

ver

Tran

scei

ver FIFOFIFO

FIFOFIFO

FIFOFIFO

FIFOFIFO

Scheduler

Stratix GX Devices for Line Cards Stratix GX Devices for Line Cards

Traffic Traffic Management,Management,

Buffer Buffer ManagementManagement

Tran

scei

ver

Tran

scei

ver

External Memory

10 Gbps Interface(Proprietary, SPI-4.2,

XSBI, etc.)

40 Gbps Memory Interface

(DDR, QDR/QDRII)

from NetworkProcessing

10 Gbps Backplane(Proprietary, XAUI, SONET/SDH, etc.)

to Switch Fabric

Card

Programmable Traffic ManagerIntegrated Transceiver & Traffic Manager /Buffer ManagerSuperior Buffer Management Capabilities

Common Crossbar Design NeedsCommon Crossbar Design Needs

Design Updates/Corrections in Field

Standard Part

Stratix™ GX

Time-to-Market

Real-Time Design & Debug

Customizable Scheduling, Buffering, & Fabric

Needs ASIC

© 2002

Distributed Switch FabricsDistributed Switch Fabrics

Line CardPacket

Processing & Traffic

Management

Backplane

Line CardPacket

Processing & Traffic

Management

Line CardPacket

Processing & Traffic

Management

Switch ModuleSwitch ModuleSwitch Module

Every Line Card Connects to Every Other Line Card via High-Speed Serial Links

Distributed Switch FabricDistributed Switch Fabric

PHYPHY Framer/Framer/MACMAC

Traffic Traffic ManagerManager

Queue/Queue/BufferBufferMgmtMgmt

PHYPHY

PHYPHY

PHYPHY

Framer/Framer/MACMAC

Framer/Framer/MACMAC

Framer/Framer/MACMAC

Packet Packet Processing Processing

& Forwarding& Forwarding

Control Control ProcessorProcessor

Line Card 1

Tran

scei

vers

Tran

scei

vers

Control FunctionData Path Function

Switch ModuleSwitch Module

to Line Card 2

to Line Card 3

to Line Card N

Typical Data FlowTypical Data Flow

Xcvr

RX NRX N--1 1 AggregatorAggregator

Xcvr Xcvr

Ingress Traffic

TX TX FIFOFIFO

RX RX FIFOFIFO

Egress Traffic

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

Stratix GX TransceiverStratix GX Logic

TX 1TX 1--N N SwitchSwitch

Distributed Switch ModuleDistributed Switch Module

To Other Line Cards

Xcvr

Stratix GX TransceiverStratix GX Logic

Xcvr Xcvr

Ingress Traffic

RX RX FIFOFIFO

Egress Traffic

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

Head-of-Line Blocking

Eliminated

TX TX FIFOFIFO

TX 1TX 1--N N SwitchSwitch

Distributed Switch ModuleDistributed Switch Module

To Other Line Cards

RX NRX N--1 1 AggregatorAggregator

Xcvr Xcvr Xcvr

Ingress Traffic Egress Traffic

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

TX & RX TX & RX FIFO,FIFO,Flow Flow

ControlControl

Flow Controllers

on Each Line Card

Work Together to Utilize

Fabric Efficiently

TX TX FIFOFIFO

TX 1TX 1--N N SwitchSwitch

RX RX FIFOFIFO

Distributed Switch ModuleDistributed Switch Module

To Other Line Cards

RX NRX N--1 1 AggregatorAggregator

Stratix GX TransceiverStratix GX Logic

Xcvr

RX NRX N--1 1 AggregatorAggregator

Xcvr Xcvr

Ingress Traffic Egress Traffic

FIFOFIFO

TX TX FIFOFIFO

TX 1TX 1--N N SwitchSwitch

RX RX FIFOFIFO

FIFOFIFO FIFOFIFO

External Memory

40 Gbps Memory Interface

(DDR, QDR/QDRII)

Buffer Buffer ManagementManagement

Traffic Traffic ManagementManagement

To Other Line Cards

Stratix GX TransceiverStratix GX Logic

Integrated Traffic & Buffer ManagementIntegrated Traffic & Buffer Management

Traffic Traffic ManagerManager

Queue/Queue/BufferBufferMgmtMgmt

10 Gbps Line Card

Tran

scei

vers

Tran

scei

vers

Switch ModuleSwitch Module

External Memory

10-Gbps Interface(Proprietary, SPI-4.2,

XSBI, etc.)

40-Gbps Memory Interface

(DDR, QDR/QDRII)

ProprietaryBackplane(up to 20 x

3.125 Gbps)

from NetworkProcessing to Line Cards

Stratix GX SolutionStratix GX Solution

Distributed Switch Fabric Sample ImplementationDistributed Switch Fabric Sample Implementation

Traffic Traffic ManagerManager

Queue/Queue/BufferBufferMgmtMgmt

10 Gbps Line Card

Tran

scei

vers

Tran

scei

vers

Switch ModuleSwitch Module

External Memory

160 MHzfMAX

16Transceiver Channels

1M-RAM Block

35KLogic Elements160M4K RAM Blocks

Design Results

16 Line Card SystemFixed Cell Size64-Bit Data PathFIFO Depth (Tx & Rx)

Line Side: 128 CellsBackplane Side: 32 Cells

Includes Buffer & Traffic Manager with External Memory Interface

Stratix GX for Distributed Switch FabricsStratix GX for Distributed Switch Fabrics

Can Support Many Links in One Device− Up to 20 Integrated 3.125-

Gbps TransceiversAbundant Queuing Resources− Up to 183 M4K RAM

Blocks(183 x 4KBits)

− Up to 4 M-RAM Blocks(4 x 512 Kbits)

Traffic Management, Buffer Management & Flow Control Functions− Over 41K LEs Available − Enhanced Memory

Interface Support

3.125 GbpsTransceiver Channel

Logic ArrayBlocks (LABs)

M512 Blocks

M-RAM Block

Source-Synchronous I/O Channels

with DPA

M4K Blocks

Common Distributed Switch Fabric Design NeedsCommon Distributed Switch Fabric Design Needs

Stratix GXStandard Part

ASICNeeds

N/ACustomizable Traffic &Buffer Management

N/ADesign Updates/Corrections in Field

N/A

N/A

N/A

Time-to-Market

Real-Time Design & Debug

Customizable Scheduling, Buffering, & Switching

Switch Fabrics:Stratix GX Devices Deliver!Switch Fabrics:Stratix GX Devices Deliver!

Backplane Interface− Up to 20 Integrated 3.125-Gbps Transceivers

High-Speed Chip-to-Chip Interfaces− 1-Gbps Source-Synchronous Channels with Dynamic Phase Alignment

(DPA)Programmable Digital Functions− Stratix Device-Based Programmable Logic Architecture

External Memory Buffering− DDR, FCRAM, SDR, ZBT, QDR/QDRII

Internal Buffering− Over 400-KBytes of TriMatrix Memory

Proprietary & Atypical Implementations− Designed to Provide Superior Flexibility