© 2002
Implementing Switch Fabrics in FPGAsImplementing Switch Fabrics in FPGAs
AgendaAgendaStratix GX IntroductionSystem OverviewCentralized Switch FabricsDistributed Switch FabricsSummary
Stratix GX IntroductionStratix GX Introduction
Transceiver Channel
Logic ArrayBlocks (LABs)
M512 Blocks
M4K Blocks
M-RAM Block
DSP Blocks
Transceiver Blocks
I/O Elements (IOEs)
Phase-Locked Loops (PLLs)
Source-Synchronous I/O
Channels With Dynamic Phase
Alignment
IOEs Support PCI, Memory Interfaces
IOEs Support PCI, Memory Interfaces
Typical Networking EquipmentTypical Networking Equipment
Router,L2 Switch,L3 Switch
Line Cards
Host Card
Switch Fabric Card
Typical Data FlowTypical Data Flow
PHYPHY Framer/Framer/MACMAC
Traffic Traffic ManagerManager
Queue/Queue/BufferBufferMgmtMgmt
PHYPHY
PHYPHY
PHYPHY
Framer/Framer/MACMAC
Framer/Framer/MACMAC
Framer/Framer/MACMAC
Packet Packet Processing Processing
& Forwarding& Forwarding
Control Control ProcessorProcessor
Line Card
Tran
scei
ver
Tran
scei
ver
Switch Fabric Card
SwitchSwitchFabricFabric
SchedulerScheduler
Control FunctionData Path Function
Tran
scei
ver
Tran
scei
ver
Backplane4 x 3.125 Gbps
Centralized Switch Fabrics− Each Line Card Connects to
Switch Fabric Card− Switch Fabric Card Routes
Data− Redundant Switch Fabric
Cards Needed
Switc
h Fa
bric
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G Backplane
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G BackplaneDistributed Switch Fabrics− Each Line Card Connects
to Every Other Line Card− Switch Module on Each
Line Card Routes Data− No Need for Redundancy
Backplane Architectures Backplane Architectures
© 2002
Centralized Switch FabricsCentralized Switch Fabrics
Switch Fabric Card
Crossbar,Crossbar,SchedulerScheduler
Tran
scei
ver
Tran
scei
ver
Backplane
Port 1
Port 2
Port 3
Port 4
Port 5
Port 1 Port 2 Port 3 Port 4 Port 5
Inputs
Outputs
Effective in Balanced Traffic−Traffic Evenly Spread among
All Line CardsIneffective When Individual Ports Have High Traffic
−Backlog of Packets
Centralized: Crossbar FabricsCentralized: Crossbar Fabrics
Crossbar FunctionCrossbar Function
Crossbar
Port 1
Port 2
Port n
Scheduler/Arbitrator
Line CardLine Card
Line CardLine Card
Line CardLine Card
Crossbar Function – Reverse RouteCrossbar Function – Reverse Route
Line CardLine Card
Line CardLine Card
Line CardLine Card
Port 1
Port 2
Port n
Scheduler/Arbitrator
Port n Calls for
Data from Port 1
Crossbar Function – Reverse RouteCrossbar Function – Reverse Route
Ingress
Egress
Line CardLine Card
Line CardLine Card
Line CardLine Card
Port 1
Port 2
Port n
Scheduler/Arbitrator
MUX Structure
Repeated for All Ports
Queue
Crossbar
Head-of-Line Blocking ProblemHead-of-Line Blocking ProblemScheduler/Arbitrator
Line CardLine Card
Line CardLine Card
Line CardLine Card Port n
Port 1
Port 2Queue22--nn
11--22 11--nn
Head-of-Line Blocking:
Packet ‘1-2’ Waits for Packet‘1-n’ to Be Transmitted
Queue* AA--BB
A: Input PortB: Output Port
*Queuing Can AlsoOccur on Line Cards
Crossbar
Scheduler/Arbitrator
Line CardLine Card
Line CardLine Card
Line CardLine Card
Port 1
Port 2Queue22--nn
Head-of-Line BlockingEliminated with
Separate Output Queuesfor Every Output Port
Virtual Output Queuing (VOQ)Virtual Output Queuing (VOQ)
Port nQueue*
*Queuing Can AlsoOccur on Line Cards
AA--BBA: Input PortB: Output Port
11--2211--nn
11--2211--nn
Switch Fabric Operates on Time SlotsSwitch Fabric Must Be “Configured” During Each Time Slot
Methods of Scheduling Packets/Cells for Switch Fabrics− SLIP, Islip, Parallel Iterative Matching (PIM) , Least Recently Used
(LRU)
Switch Fabric Scheduling MethodsSwitch Fabric Scheduling Methods
3
n
IngressPort 3
IngressPort n
EgressPort 1
33--11
Scheduler
Stratix GX Devices for CrossbarsStratix GX Devices for CrossbarsSupports Many Links in One Device− Up to 20 Integrated 3.125-
Gbps TransceiversOutput Port Queuing− Up To 183 M4K RAM
Blocks(183 x 4KBits)
More Queuing− Up to 4 M-RAM Blocks (4 x
512 Kbits)Scheduling & Flow Control Functions− Over 41K Logic Elements
(LEs)
3.125 GbpsTransceiver Channel
Logic ArrayBlocks (LABs)
M512 Blocks
M-RAM Block
Source-Synchronous I/O Channels
with DPA
M4K Blocks
Centralized Switch Fabric Centralized Switch Fabric
Switc
h Fa
bric
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G
Lin
e C
ard
10G Backplane
Backplane Transceiver
Queue/Buffer Management
Traffic Management
Backplane Transceiver
Switch Fabric
Stratix GX Crossbar ImplementationStratix GX Crossbar Implementation Switch Fabric Card
20 x 3.125 Gbps Backplane
Programmable Scheduler− Design to Your Traffic
PatternIntegrated Transceiver, Crossbar & SchedulerBuffering Capabilities
to Line Cards
1834,096 BitsM4K512 KBits
512 Bits
Size
4M-RAM
384M512
Number of Blocks in
EP1SGX40
RAM Block
Tran
scei
ver
Tran
scei
ver FIFOFIFO
FIFOFIFO
FIFOFIFO
FIFOFIFO
Scheduler
Crossbar Sample ImplementationCrossbar Sample Implementation8x8 Crossbar FabricFixed Cell Size16-Bit Data PathVOQ Depth: 8 CellsOutput Queue Depth: 32 CellsRound Robin Scheduler
160 MHzfMAX
8Transceiver Channels
10KLogic Elements96M4K RAM Blocks
Design Results
Switch Fabric Card
8 x 3.125 GbpsBackplane
to Line Cards
Tran
scei
ver
Tran
scei
ver FIFOFIFO
FIFOFIFO
FIFOFIFO
FIFOFIFO
Scheduler
Stratix GX Devices for Line Cards Stratix GX Devices for Line Cards
Traffic Traffic Management,Management,
Buffer Buffer ManagementManagement
Tran
scei
ver
Tran
scei
ver
External Memory
10 Gbps Interface(Proprietary, SPI-4.2,
XSBI, etc.)
40 Gbps Memory Interface
(DDR, QDR/QDRII)
from NetworkProcessing
10 Gbps Backplane(Proprietary, XAUI, SONET/SDH, etc.)
to Switch Fabric
Card
Programmable Traffic ManagerIntegrated Transceiver & Traffic Manager /Buffer ManagerSuperior Buffer Management Capabilities
Common Crossbar Design NeedsCommon Crossbar Design Needs
Design Updates/Corrections in Field
Standard Part
Stratix™ GX
Time-to-Market
Real-Time Design & Debug
Customizable Scheduling, Buffering, & Fabric
Needs ASIC
© 2002
Distributed Switch FabricsDistributed Switch Fabrics
Line CardPacket
Processing & Traffic
Management
Backplane
Line CardPacket
Processing & Traffic
Management
Line CardPacket
Processing & Traffic
Management
Switch ModuleSwitch ModuleSwitch Module
Every Line Card Connects to Every Other Line Card via High-Speed Serial Links
Distributed Switch FabricDistributed Switch Fabric
PHYPHY Framer/Framer/MACMAC
Traffic Traffic ManagerManager
Queue/Queue/BufferBufferMgmtMgmt
PHYPHY
PHYPHY
PHYPHY
Framer/Framer/MACMAC
Framer/Framer/MACMAC
Framer/Framer/MACMAC
Packet Packet Processing Processing
& Forwarding& Forwarding
Control Control ProcessorProcessor
Line Card 1
Tran
scei
vers
Tran
scei
vers
Control FunctionData Path Function
Switch ModuleSwitch Module
to Line Card 2
to Line Card 3
to Line Card N
Typical Data FlowTypical Data Flow
Xcvr
RX NRX N--1 1 AggregatorAggregator
Xcvr Xcvr
Ingress Traffic
TX TX FIFOFIFO
RX RX FIFOFIFO
Egress Traffic
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
Stratix GX TransceiverStratix GX Logic
TX 1TX 1--N N SwitchSwitch
Distributed Switch ModuleDistributed Switch Module
To Other Line Cards
Xcvr
Stratix GX TransceiverStratix GX Logic
Xcvr Xcvr
Ingress Traffic
RX RX FIFOFIFO
Egress Traffic
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
Head-of-Line Blocking
Eliminated
TX TX FIFOFIFO
TX 1TX 1--N N SwitchSwitch
Distributed Switch ModuleDistributed Switch Module
To Other Line Cards
RX NRX N--1 1 AggregatorAggregator
Xcvr Xcvr Xcvr
Ingress Traffic Egress Traffic
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
TX & RX TX & RX FIFO,FIFO,Flow Flow
ControlControl
Flow Controllers
on Each Line Card
Work Together to Utilize
Fabric Efficiently
TX TX FIFOFIFO
TX 1TX 1--N N SwitchSwitch
RX RX FIFOFIFO
Distributed Switch ModuleDistributed Switch Module
To Other Line Cards
RX NRX N--1 1 AggregatorAggregator
Stratix GX TransceiverStratix GX Logic
Xcvr
RX NRX N--1 1 AggregatorAggregator
Xcvr Xcvr
Ingress Traffic Egress Traffic
FIFOFIFO
TX TX FIFOFIFO
TX 1TX 1--N N SwitchSwitch
RX RX FIFOFIFO
FIFOFIFO FIFOFIFO
External Memory
40 Gbps Memory Interface
(DDR, QDR/QDRII)
Buffer Buffer ManagementManagement
Traffic Traffic ManagementManagement
To Other Line Cards
Stratix GX TransceiverStratix GX Logic
Integrated Traffic & Buffer ManagementIntegrated Traffic & Buffer Management
Traffic Traffic ManagerManager
Queue/Queue/BufferBufferMgmtMgmt
10 Gbps Line Card
Tran
scei
vers
Tran
scei
vers
Switch ModuleSwitch Module
External Memory
10-Gbps Interface(Proprietary, SPI-4.2,
XSBI, etc.)
40-Gbps Memory Interface
(DDR, QDR/QDRII)
ProprietaryBackplane(up to 20 x
3.125 Gbps)
from NetworkProcessing to Line Cards
Stratix GX SolutionStratix GX Solution
Distributed Switch Fabric Sample ImplementationDistributed Switch Fabric Sample Implementation
Traffic Traffic ManagerManager
Queue/Queue/BufferBufferMgmtMgmt
10 Gbps Line Card
Tran
scei
vers
Tran
scei
vers
Switch ModuleSwitch Module
External Memory
160 MHzfMAX
16Transceiver Channels
1M-RAM Block
35KLogic Elements160M4K RAM Blocks
Design Results
16 Line Card SystemFixed Cell Size64-Bit Data PathFIFO Depth (Tx & Rx)
Line Side: 128 CellsBackplane Side: 32 Cells
Includes Buffer & Traffic Manager with External Memory Interface
Stratix GX for Distributed Switch FabricsStratix GX for Distributed Switch Fabrics
Can Support Many Links in One Device− Up to 20 Integrated 3.125-
Gbps TransceiversAbundant Queuing Resources− Up to 183 M4K RAM
Blocks(183 x 4KBits)
− Up to 4 M-RAM Blocks(4 x 512 Kbits)
Traffic Management, Buffer Management & Flow Control Functions− Over 41K LEs Available − Enhanced Memory
Interface Support
3.125 GbpsTransceiver Channel
Logic ArrayBlocks (LABs)
M512 Blocks
M-RAM Block
Source-Synchronous I/O Channels
with DPA
M4K Blocks
Common Distributed Switch Fabric Design NeedsCommon Distributed Switch Fabric Design Needs
Stratix GXStandard Part
ASICNeeds
N/ACustomizable Traffic &Buffer Management
N/ADesign Updates/Corrections in Field
N/A
N/A
N/A
Time-to-Market
Real-Time Design & Debug
Customizable Scheduling, Buffering, & Switching
Switch Fabrics:Stratix GX Devices Deliver!Switch Fabrics:Stratix GX Devices Deliver!
Backplane Interface− Up to 20 Integrated 3.125-Gbps Transceivers
High-Speed Chip-to-Chip Interfaces− 1-Gbps Source-Synchronous Channels with Dynamic Phase Alignment
(DPA)Programmable Digital Functions− Stratix Device-Based Programmable Logic Architecture
External Memory Buffering− DDR, FCRAM, SDR, ZBT, QDR/QDRII
Internal Buffering− Over 400-KBytes of TriMatrix Memory
Proprietary & Atypical Implementations− Designed to Provide Superior Flexibility