048866: packet switch architectures
DESCRIPTION
048866: Packet Switch Architectures. Introduction. Dr. Isaac Keslassy Electrical Engineering, Technion [email protected] http://comnet.technion.ac.il/~isaac/. Course Content. Most of the results in the course are less than 10 years old! Material is new and still changing… - PowerPoint PPT PresentationTRANSCRIPT
048866: Packet Switch Architectures
Dr. Isaac KeslassyElectrical Engineering, Technion
http://comnet.technion.ac.il/~isaac/
Introduction
Spring 2006 048866 – Packet Switch Architectures 2
Course Content
Most of the results in the course are less than 10 years old! Material is new and still changing… No reference book available yet…
Uses slides by Profs. Nick McKeown and Balaji Prabahakar (Stanford)
Spring 2006 048866 – Packet Switch Architectures 3
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 4
What is Routing?
R3
A
B
C
R1
R2
R4 D
E
FR5
R5F
R3E
R3D
Next HopDestination
D
Spring 2006 048866 – Packet Switch Architectures 5
What is Routing?
R3
A
B
C
R1
R2
R4 D
E
FR5
R5F
R3E
R3D
Next HopDestination
D
16 3241
Data
Options (if any)
Destination Address
Source Address
Header ChecksumProtocolTTL
Fragment OffsetFlagsFragment ID
Total Packet LengthT.ServiceHLenVer
20
byte
s
Spring 2006 048866 – Packet Switch Architectures 6
What is Routing?
A
B
C
R1
R2
R3
R4 D
E
FR5
Spring 2006 048866 – Packet Switch Architectures 7
Points of Presence (POPs)
A
B
C
POP1
POP3POP2
POP4 D
E
F
POP5
POP6 POP7POP8
Spring 2006 048866 – Packet Switch Architectures 8
Where High Performance Routers are Used
R10 R11
R4
R13
R9
R5
R2R1 R6
R3 R7
R12
R16R15
R14
R8
(10 Gb/s)
(10 Gb/s)(10 Gb/s)
(10 Gb/s)
Spring 2006 048866 – Packet Switch Architectures 9
What a Router Looks Like
Cisco GSR 12416 Juniper M160
6ft
19”
2ft
Capacity: 160Gb/sPower: 4.2kWFull rack
3ft
2.5ft
19”
Capacity: 80Gb/sPower: 2.6kWHalf-a-rack
Spring 2006 048866 – Packet Switch Architectures 10
Core Router Market
Estimates cover different parameters Dell’Oro (Feb. 17, 2005):
$1.2 billion in 2004 (up 66%) Includes high-end (10Gbps) router market
Dell’Oro (Feb. 21, 2006): $4.4 billion in 2005 Includes “worldwide service provider router market”
Infonetics (Feb. 21, 2006): $5.2 billion in 2005 (up 31%) Includes core/edge routers
Core router market Relatively small (in comparison to enterprise market), but brings
expertise to company Vendors: Cisco, Juniper, Avici, Nortel, Lucent, Alcatel, Huawei, etc.
Spring 2006 048866 – Packet Switch Architectures 11
Router Market
Source: Infonetics (Nov. 2004)
Spring 2006 048866 – Packet Switch Architectures 12
Basic Architectural Componentsof an IP Router
Control Plane
Datapathper-packet processing
SwitchingForwarding
Table
RoutingTable
Routing Protocols
Spring 2006 048866 – Packet Switch Architectures 13
Per-packet processing in an IP Router
1. Accept packet arriving on an incoming link.
2. Lookup: lookup packet destination address in the forwarding table, to identify outgoing port(s).
3. Header Processing: Manipulate packet header: e.g., decrement TTL, update header checksum.
4. Switching: Send packet to the outgoing port(s).5. Buffering: Buffer packet in the queue.
6. Transmit packet onto outgoing link.
Spring 2006 048866 – Packet Switch Architectures 14
Generic Router Architecture
LookupIP Address
UpdateHeader
Header ProcessingData Hdr Data Hdr
~1M prefixesOff-chip DRAM
AddressTable
AddressTable
IP Address Next Hop
QueuePacket
BufferMemoryBuffer
Memory~1M packetsOff-chip DRAM
Spring 2006 048866 – Packet Switch Architectures 15
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
Data Hdr
Data Hdr
Data Hdr
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
Spring 2006 048866 – Packet Switch Architectures 16
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 17
Why Do We Need Faster Routers?
1. To prevent routers from becoming the bottleneck in the Internet.
2. To increase POP capacity, and to reduce cost, size and power.
Spring 2006 048866 – Packet Switch Architectures 18
0,1
1
10
100
1000
10000
1985 1990 1995 2000
Spe
c95I
nt C
PU
res
ults
Why We Need Faster Routers 1: To prevent routers from being the bottleneck
Packet Processing Power
2x / 18 months
Source: SPEC95Int & Coffman and Odlyzko.
Single Fiber Capacity(commercial)
≥ 2x / year
Spring 2006 048866 – Packet Switch Architectures 19
POP with smaller routers
Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs
POP with large routers
Ports: Price >$50k, Power > 400W. It is common for 50-60% of ports to be for interconnection.
Spring 2006 048866 – Packet Switch Architectures 20
Why are Fast Routers Difficult to Make?
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with
Moore’s Law.
Spring 2006 048866 – Packet Switch Architectures 21
Why are Fast Routers Difficult to Make?Speed of Commercial DRAM
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with
Moore’s Law.
0.001
0.01
0.1
1
10
100
1000
1980 1983 1986 1989 1992 1995 1998 2001
Acc
ess
Tim
e (n
s)
Moore’s Law2x / 18 months
1.1x / 18 months
Spring 2006 048866 – Packet Switch Architectures 22
Why are Fast Routers Difficult to Make?
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with
Moore’s Law.
2. Moore’s Law is too slow: Routers need to improve faster than Moore’s
Law.
Spring 2006 048866 – Packet Switch Architectures 23
Router Performance Exceeds Moore’s Law
Growth in capacity of commercial routers: Capacity 1992 ~ 2Gb/s Capacity 1995 ~ 10Gb/s Capacity 1998 ~ 40Gb/s Capacity 2001 ~ 160Gb/s Capacity 2003 ~ 640Gb/s
Average growth rate: 2.2x / 18 months.
Spring 2006 048866 – Packet Switch Architectures 24
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 25
RouteTableCPU Buffer
Memory
LineInterface
MAC
LineInterface
MAC
LineInterface
MAC
Typically <0.5Gb/s aggregate capacity
Shared Backplane
Line Interface
CPU
Memory
First-Generation Routers
Spring 2006 048866 – Packet Switch Architectures 26
RouteTableCPU
LineCard
BufferMemory
LineCard
MAC
BufferMemory
LineCard
MAC
BufferMemory
FwdingCache
FwdingCache
FwdingCache
MAC
BufferMemory
Typically <5Gb/s aggregate capacity
Second-Generation Routers
Spring 2006 048866 – Packet Switch Architectures 27
LineCard
MAC
LocalBuffer
Memory
CPUCard
LineCard
MAC
LocalBuffer
Memory
Switched Backplane
Line Interface
CPUMem
ory FwdingTable
RoutingTable
FwdingTable
Typically <50Gb/s aggregate capacity
Third-Generation Routers
Spring 2006 048866 – Packet Switch Architectures 28
Switch Core Linecards
Optical links
100sof metres
0.3 - 10Tb/s routers
Fourth-Generation RoutersMulti-Racks, Optical Links
Spring 2006 048866 – Packet Switch Architectures 29
Optical Switch Core Linecards
Optical links
100sof metres
10-100Tb/s routers, in project
(Future) Fifth-Generation RoutersOptical Switch Core
Spring 2006 048866 – Packet Switch Architectures 30
Optical Switch Core Optical Linecards
Optical links
100sof metres
100-1000Tb/s routers, in the far future
(Future) Sixth-Generation RoutersAll-Optical Routers
Spring 2006 048866 – Packet Switch Architectures 31
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 32
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
LookupIP Address
AddressTable
AddressTable
LookupIP Address
AddressTable
AddressTable
LookupIP Address
AddressTable
AddressTable
Spring 2006 048866 – Packet Switch Architectures 33
IP Address Lookup
Why it’s thought to be hard: It’s not an exact match: it’s a longest
prefix match. The table is large: about 150,000 entries
today, and growing. The lookup must be fast: about 30ns for a
10Gb/s line.
Spring 2006 048866 – Packet Switch Architectures 34
IP Lookups find Longest Prefixes
128.9.16.0/21 128.9.172.0/21
128.9.176.0/24
0 232-1
128.9.0.0/16142.12.0.0/1965.0.0.0/8
128.9.16.14
Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.
Spring 2006 048866 – Packet Switch Architectures 37
IP Address Lookup
Why it’s thought to be hard: It’s not an exact match: it’s a longest
prefix match. The table is large: about 150,000 entries
today, and growing. The lookup must be fast: about 30ns for a
10Gb/s line.
Spring 2006 048866 – Packet Switch Architectures 38
Address Tables are Large
Source: http://www.cidr-report.org/
Spring 2006 048866 – Packet Switch Architectures 39
IP Address Lookup
Why it’s thought to be hard: It’s not an exact match: it’s a longest
prefix match. The table is large: about 150,000 entries
today, and growing. The lookup must be fast: about 30ns for a
10Gb/s line.
Spring 2006 048866 – Packet Switch Architectures 40
Lookups Must be Fast
12540Gb/s2003
31.2510Gb/s2001
7.812.5Gb/s1999
1.94622Mb/s1997
40B packets (Mpkt/s)
LineYear
500160Gb/s2005?
Spring 2006 048866 – Packet Switch Architectures 41
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 42
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
Spring 2006 048866 – Packet Switch Architectures 43
Fast Packet Buffers
Example: 40Gb/s packet bufferSize = RTT*BW = 10Gb; 40 byte packets
Write Rate, R
1 packetevery 8 ns
Read Rate, R
1 packetevery 8 ns
BufferManager
BufferMemory
Use SRAM?+ fast enough random access time, but
- too low density to store 10Gb of data.
Use SRAM?+ fast enough random access time, but
- too low density to store 10Gb of data.
Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).
Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).
Spring 2006 048866 – Packet Switch Architectures 44
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Spring 2006 048866 – Packet Switch Architectures 45
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
1
2
N
1
2
N
N times line rate
N times line rate
Spring 2006 048866 – Packet Switch Architectures 46
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
1
2
N
1
2
N
Data Hdr
Data Hdr
Data Hdr
Scheduler
Spring 2006 048866 – Packet Switch Architectures 47
Current Internet Router TechnologySummary
There are three potential bottlenecks: Address lookup, Packet buffering, and Switching.
Most difficult – and focus of this course!