ethernet over sonet dr. john s. graham university of london computer centre
TRANSCRIPT
Ethernet over SONET
Dr. John S. Graham
University of London Computer Centre
Ethernet vs SONET
Feature Ethernet SONET
Guaranteed Delivery
No (Packet Switched)
Yes (Circuit Switched)
Guaranteed BW No Yes
Performance Monitoring
CRC Count BER
Protection None (STP?) APS
Fault Location None AIS
Access Costs Low High
SONET/SDH Model
RRMUX DXC MUX
Path Layer Connection
Line LayerLine Layer
Section Layer Section Layer Section LayerSection Layer
SONET Network Elements
SONET STS-1 Frame
Section
Line
PTR
SPE
3
5
86 columns
Path
30 59 9041
Payload Capacity = 756 Bytes = 48.384 Mb/s
Section Overhead Bytes
Data Com
D3
Data Com
D2
Data Com
D1
User
F1
Orderwire
E1
BIP-8
B1
STS-1 ID
C1Framing
A2Framing
A1
Section
Line Overhead Bytes
Line
Orderwire
E2
Line FEBE
Z2
Sync Stat
Z1
Data Com
D12
Data Com
D11
Data Com
D10
Data Com
D9
Data Com
D8
Data Com
D7
Data Com
D6
Data Com
D5
Data Com
D4
APS
K2
APS
K1
BIP-8
B2
Path Overhead Bytes
Path
Multi-frame
H4
User Channel
F2
Path Status
G1
Signal Label
C2
BIP-8
B3
Path Trace
J1
Accurate Bit-Level Timing• Line Coding
– AMI, AMI RZ– HDB3
• Block Coding– 4B/5B (FDDI)– 5B/6B (E3)– 8B/10B (Gigabit Ethernet)
• Scrambling– Frame Synchronous– Self Synchronous
8B/10B Encoding (1/2)• Ensures equal number of 1’s and 0’s for
– Clock recovery– DC Balance
• Detection of single bit transmission errors
• Predefined ‘comma’ (0011111 and 1100000) characters
8B/10B Encoding (2/2)• Data (D) and commands (K) encoded separately• Octet split into 5 MSB and 3 LSB• Code groups generated from lookup tables
Decoded
A B C D E F G H
0 1 0 0 0 0 0 0
2 0
Encoded
A B C D E i F G H j
1 0 1 1 0 1 0 1 0 0
D2.0
Control Words (‘Ordered Sets’)
Symbol Encoding Purpose
F K28.5 D21.5 LINK_NOT_AVAILABLE
C K28.5 D10.5 config_reg LINK_CONFIGURATION
I1 K28.5 D5.6 Idle
I2 K28.5 D16.2 Idle
S K27.7 SoF
T K29.7 EoF
R K23.7 CARRIER_EXTEND
H K30.7 ERROR
Scrambling in SONET/SDH
SDH STM-1 uses x7 + x6 +1 polynomial
ConcatenationSTS-3 155 Mb/s
145.152 Mb/s
STS-3c/VC-4 155 Mb/s149.76 Mb/s
STS-12 622 Mb/s580.608 Mb/s
VC-4-4 622 Mb/s599 Mb/s
STS-12c 622 Mb/s599.04 Mb/s
Provisioning Example
A
D
B
E C
F3 - 5
7
9
10 -12
1
???
1 – 12 (622 Mb/s)
Bandwidth Fragmentation
STS-3c
STS-12
Blocked!
Drop Interface (Node F)
Ring
Virtual Concatenation
STS-3c
STS-12
STS-1-3v
Drop Interface (Node F)
Ring
The Magic of VCAT
ProtocolClient Rate
Contiguous Concatenation
Virtual Concatenation
Fast Ethernet 100 STS-3c 65% STS-1-2v 98%
ESCON 160 STS-12c 26% STS-1-4v 78%
Fibre Channel 850 STS-48c 34% STS-3c-6v 92%
Gigabit Ethernet 1000 STS-48c 40% STS-3c-7v 92%
Fibre Channel 2 1700 STS-48c 66% STS-3c-12v 92%
Differential Delay
TX Ty
Source
STS-1-12v
SinkCore Network(Partial Mesh)
Multiframe IndicatorBit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8
MFI2 MSBs (bits 1 - 4) 0 0 0 0
MFI2 LSBs (bits 5 – 8) 0 0 0 1
CTRL 0 0 1 0
GID 0 0 1 1
0 1 0 0
0 1 0 1
CRC-8 0 1 1 0
CRC-8 0 1 1 1
MST 1 0 0 0
MST 1 0 0 1
1 0 1 0
1 0 1 1
1 1 0 0
1 1 0 1
SQ MSBs (bits 1 – 4) 1 1 1 0
SQ LSBs (bits 5 – 8) 1 1 1 1
VCAT Example
GID-a, SQ#=01
GID-a, SQ#=12
GID-a, SQ#=23
6
5
4
12
11
GID-b, SQ#=310
GID-b, SQ#=29
GID-b, SQ#=18
GID-a, SQ#=37
STS-1-4v STS-1-3v
STS-12
VCAT Puzzle• An STS-1-2v between Chicago and
London
• One STS-1 sent via geostationary satellite
• The other STS-1 sent via transatlantic fibre
• Will it work? ;-)
VCAT: Buffer RequirementsVC Path Type
Transport Signal
Number Paths
Buffer Size
STS-1VC-3
STS-12STM-4
12 18 MB
STS-1VC-3
STS-48STM-16
48 71 MB
STS-3cVC-4
STS-12STM-4
4 18 MB
STS-3cVC-4
STS-48STM-16
12 55 MB
Link Capacity Adjustment Scheme• Modify membership of a VCG
• On-demand and hitless bandwidth changes
• Automatic removal of failed VCG members
• Interworking of LCAS-enabled VCG with non-LCAS VCG
• Old-style ‘bridge and roll’ requires double bandwidth
LCAS Messaging• Member Status (MST)• Re-Sequence Acknowledge (RS-Ack)• Control (CTRL)
– Fixed– Add– Norm– EOS– Idle– DNU
• Group ID• CRC
LCAS Signalling Flows (1/3)
CTRL = ADD
SQ = 255
Source PTE STS-1 #1
CTRL = EOS
SQ = 2
CTRL = NORM
SQ = 1
MST = OK (#3)CTRL = NORM
SQ = 2
CTRL = NORM
SQ = 1
STS-1 #2 STS-1 #3
CTRL = NORM
SQ = 1
CTRL = EOS
SQ = 2
CTRL = EOS
SQ = 3
CTRL = IDLE
SQ = 255
CTRL = NORM
SQ = 1
CTRL = EOS
SQ = 2
RS-ACKCTRL = NORM
SQ = 2
CTRL = NORM
SQ = 1
CTRL = EOS
SQ = 3
MST = FAIL (#3)
LCAS Signalling Flows (2/3)
CTRL = EOS
SQ = 3
Source PTE STS-1 #1
CTRL = NORM
SQ = 2
CTRL = IDLE
SQ = 255
CTRL = EOS
SQ = 3
CTRL = NORM
SQ = 2
CTRL = NORM
SQ = 1
MST = FAIL (#1)CTRL = EOS
SQ = 2
CTRL = NORM
SQ = 1
STS-1 #2 STS-1 #3
RS-ACKCTRL = EOS
SQ = 2
CTRL = NORM
SQ = 1
LCAS Signalling Flows (3/3)
CTRL = IDLE
SQ = 255
Source PTE STS-1 #1
CTRL = NORM
SQ = 2
CTRL = EOS
SQ = 3
CTRL = NORM
SQ = 2
CTRL = NORM
SQ = 1
MST = FAIL (#3)CTRL = EOS
SQ = 2
CTRL = NORM
SQ = 1
STS-1 #2 STS-1 #3
RS-ACKCTRL = EOS
SQ = 2
CTRL = NORM
SQ = 1
CTRL = NORM
SQ = 1
Gigabit Ethernet Implementations
Gigabit Ethernet (802.3z)
FC-0 Interface & Media
FC-1 Encode/Decode
FC-2 Signalling
FC-3 Common Services
FC-4 Upper Layer Mapping
IEEE 802.3 PHY
IEEE 802.3 CSMA/CD
IEEE 802.2 LLC
ANSI X3T11 Fibre Channel IEEE 802.3 Ethernet IEEE 802.3z Gigabit Ethernet
MDL
IEE 802.3 MAC
IEEE 802.2 LLC
PMD
PMA
PCS
1000BASE-LX/SX
Sequence of EventsLINK_NOT_AVAILABLE LINK_NOT_AVAILABLE~ LINK_CONFIGURATION
~ LINK_CONFIGURATION IDLE ~ IDLE SOP PREAMBLE
SFD DA LLC DataSA LENGTH
PADDING FCS
Generic Framing Procedure• ITU-T G.7041 and ANSI T1.105.02• Defines mapping for many types of service
onto SONET/SDH or OTN:– Ethernet, IP/PPP– GbE, Fibre Channel (inc. DVB), FICON etc
• Excellent bandwidth utilization; efficiency tailored to suit different client types
• Simple delineation and robust error control• Extensible
Motivation for GFP (1/2)• ATM
– Cell overhead causes 10% bandwidth inflation– Adaptation functions needlessly complex
• Packet over SONET (POS)– Requires all frames to be converted to PPP
over HDLC– Byte stuffing causes non-deterministic
bandwidth inflation– QoS hard to monitor or guarantee
Motivation for GFP (2/2)• Minimal overhead
• Transport of client PDUs in native format
• Designed for optimized processing
• Easy aggregation of frames from multiple client and multiple protocols into shared bandwidth channels
• Low latency capabilities for SAN
GFP: Types of Frame
GFP
Client FramesControl Frames
Data FramesManagement
FramesIdle Frames
OA&M Frames
GFP: Functional Model
Source = IEEE Communications Magazine, May 2002
GFP: Frame Structure
Source = IEEE Communications Magazine, May 2002
Payload TypesValue of UPI Field Protocol GFP Mode
0x01 Ethernet F
0x02 PPP (IP & MPLS) F
0x03 Fibre Channel T
0x04 FICON T
0x05 ESCON T
0x06 GbE T
0x07 reserved
0x08 MAPOS F
0x09 – 0xFE reserved
0x00 & 0xFF unavailable
GFP: Client-Independent Processes
• Synchronization– Bit Level– Frame Delineation
• Scrambling
• Multiplexing– GFP Frame– Client PDU
GFP: Synchronization
Hunt Presync
SynccH
EC
Mism
atch2
nd c
HE
C M
atch
cHEC Match
No 2nd cHEC Match
cHEC Match
No CHEC Match
GFP: Frame Delineation
PayloadPLI cHEC PLI cHEC Payload
Payload PLI cHEC PayloadPLI cHEC PayloadPLI cHEC
PayloadPLI cHEC
GFP: Client PDU Multiplexing
Source Sink
STS-3c-2vGFP Framer
1
2
3
GFP Framer
1
2
3
1
2
3
GFP-F Encapsulation (1/2)
Preamble
SFD
Destination
Source
Length
Data
FCS
PLI (MSB)
PLI (LSB)
cHEC (MSB
cHEC (LSB)
Payload Header
Payload
FCS
1 8
Bit Transmission Order
Transm
ission Order
7
1
6
6
2
4 4
X = 4 to 64
0 to 65,535 - X
Ethernet MAC Frame
GFP-F Encapsulation (2/2)
Flag (0x7E)
Address
Control
PPP Type
Data
FCS
PLI (MSB)
PLI (LSB)
cHEC (MSB
cHEC (LSB)
Payload Header
Payload
FCS
1 8
Bit Transmission Order
Transm
ission Order
4
X = 4 to 64
0 to 65,535 - X
PPP/HDLC Frame
1
1
2
4
1
GFP-T: 64B/65B Payload
D1 K1 D2 D3 K2D4 D5 D6
000 001 010 011 101 110 111F
65B Sequence 1 001 C1 D2 D3 D4 D5 D61 0 101 C2 D1
100Octet Number
64B Sequence
000 001 010 011 100 101 110 111Octet Number
GFP-T Error Detection• Leading flag bit is errored
• Error affects ‘Last control-code’ indicator
• Control-code location address received in error
• Error causes 4-bit control code to be modified
GFP-T: Superblocks
1,1 1,2 … 1,7 1,8
8,1 8,2 8,7… 8,8
CRC (MSB)
CRC (LSB)
1,1
1,2
…
1,7
1,8
8,1
8,2
…
8,7
8,8
536-bit Byte-Aligned
GFP-T: Bandwidth Requirements
Client Signal
Client Signal
Bandwidth
Minimum Transport Channel
Size
Nominal Transport Channel
Bandwidth
Number Superblocks
per GFP Frame
ESCON 160 Mb/sSTS-1-4v
VC-3-4v193.536 Mb/s 1
Fibre Channel
850 Mb/sSTS-3c-4vVC-4-6v
898.56 Mb/s 13
Gigabit Ethernet
1000 Mb/sSTS-3c-7vVC-4-7v
1.04832 Gb/s 95
Pros and Cons: GFP-F
Higher bandwidth efficiency
Higher Latency
More buffer memory required
Core header fields must be calculated
Pros and Cons: GFP-T
Supports many protocols Low latency Ingress core header fields need not be
calculated
Less bandwidth efficientMore logic required
10 G Ethernet – 802.3ae
10GBASE-R
10GBASE-W
S = 850 nm MM 300 m
L = 1310 nm SM 10 km
E = 1550 nm SM 40 km
Full Duplex 802.3 MAC
64B/66B PCS
Serial PMASerial PMA
WIS
E L SE L S
XGMII
PMDPMD
10 GbE Overview• Usual 802.3 MAC and frame format
– Jumbo frames not included in standard
• Full duplex only– No shared media– No CSMA/CD
• Only optical fibre– No copper interface– LAN-PHY or WAN-PHY at various reaches
The STS-192c WIS Frame
J1
B3
C2
G1
F2
H4
Z3
Z4
Z5
Fixed S
tuff
Synchronous Payload Envelope (SPE) = 16704 Columns
63 columns
9.58464 Gbs-1 Capacity
802.3 FrameIdle Idle
802.3 FrameIdle Idle
…
…
Section & Line Overhead
A1
B1
D1
A1 … A1 A2 … A2
Section
A2
E1
D2
J0
F1
D3
Z0 … Z0
Overhead …
H1
B2
D4
D7
D10
S1
H1
B2
Z1
……
…
H1
B2
Z1
H2
K1
D5
D8
D11
Z2
…
…
H2
Z2
H2
M1
H3
K2
D6
D9
D12
E2
…H3 H3
Line Overhead …
1 2 192 193 194 384 385 386 576
Defined by WAN PHY as Fixed Value
Supported by WAN PHYUnused by WAN PHYOptional for SONET/SDH
Unused by WAN PHYUndefined for SONET/SDH
NetherLight Connectivity
UKLight – NetherLight Today
MetroDirector
UKLight
15454
NetherLight
HDXc
NetherLight
15454
CERN
CoreDirector
UKLight
MetroDirector
UKLight
UKLight – NetherLight Tomorrow?
HDXc
NetherLight
CoreDirector
UKLight
MetroDirector
UKLight
15454
CERN
Requires that MSPPs are GFP-F enabled
15454
NetherLight
Peering Using 10 G WAN-PHY
Force10
StarLight
CoreDirector
UKLight
10GBASE-SW
CoreDirector
UKLightS
TM
-6410GBASE-SW
Switch
UKLight
London
Chicago