TRANSPACKET

Zero packet jitter aggregation and priority mechanisms 09.03.2015

Steinar Bjørnstad Steinar.bjornstad@transpacket.com

TRANSPACKET

•  Low packet-jitter Packet Delay Variation (PDV) applications and requirements

•  Mechanisms for reducing latency and packet-jitter –  FUSION time-window –  FUSION guaranteed service stream aggregation

•  Experimental results using FUSION on GE and 10GE –  Adding lower quality traffic to 10GE guaranteed stream –  GE aggregation into 10GE

2

Outline

TRANSPACKET

•  Year 2000: Optical networks becomes packet switched –  Moving from Circuit to packet switching –  Maintaining properties from circuit switching is desirable

•  Carriers wondering how to maintain circuit properties when moving to IP and Ethernet based networks

3

FUSION history

TRANSPACKET

•  Integrated hybrid optical network (IHON) –  Properties from Circuit and Packet networks

combined –  Still pure packet based, no TDM

•  FUSION networks –  IHON known from academic literature: Published in

major IEEE conferences and journals –  Not standardized –  Commercialized as FUSION networks

(TransPacket)

Integrating packet and circuit

TRANSPACKET

•  Wavelength and OTN services have a high production cost –  Occupies a physical wavelength or TDM channel

resource in the network –  Wavelengths/TDM channels are limited resources –  Wavelengths/TDM channels occupies resources end-to-

end – no intermediate additional aggregation possible •  Ethernet or VPN service preferred as compromise

–  Lower production cost (oversubscription/statistical multiplexing)

–  Does not offer transparency and performance (especially latency) as for wavelengths and OTN channels

5

Carrier pain

TRANSPACKET

ISP

Mobile

Optical networks

Metro Routers/optical switches

Core Optical switches/routers

Access Ethernet switches

TRANSPACKET

Mobile

Wavelength services are costly

Metro Routers/optical switches

Core Optical switches/routers

Access Ethernet switches

ISP

Wavelength !service!

Wavelength !service!

TRANSPACKET

Mobile

Ethernet Virtual wavelength services: Sub-wavelength granularity

Metro Routers/optical sw‹itches

Core Optical switches/routers

Access Ethernet switches

ISP

Fusion  

ISP

Virtual!wavelength !services!

Virtual!wavelength !service!

Fusion  

Fusion  

Fusion  

Fusion  

ISP Fusion  ISP

Virtual!wavelength !services!

TRANSPACKET

•  Ethernet Virtual wavelength services –  Performance close to circuit switching (OTN) –  Ethernet throughput performance –  No packet loss due to contention –  Fixed latency (Packet-jitter in the ns range) –  Low latency, but fibre is 5 microsecond/kilometer

•  Mobile backhaul and fronthaul –  Transparent transport of IEEE1588 packets in mobile

backhaul: Strict demands on PDV –  Radio over Ethernet in mobile fronthaul: Extreme

demands on packet-jitter and strict on latency

9

FUSION applications and requirements

TRANSPACKET

Packet-jitter and latency demands •  Mobile backhaul

–  Using IEEE1588 for synchronization –  Average jitter important to keep low –  Sync. Accuracy in the microsecond range

•  Mobile fronthaul - CPRI over Ethernet –  Peter Ashwood Smith, IEEE 802, July 2014:

• 100us – Maximum one way Delay between Antenna and Compute • 65ns – Maximum variation in Delay (Jitter). • 1-10G – Throughput per antenna (compression possible). • 10-12 – Maximum Bit Error Rate

REQUIRED OBSERVED Average latency = 3us per hop Peak Jitter = +/- 2500ns per hop

TRANSPACKET

Packet-jitter and latency demands •  Mobile backhaul

–  Using IEEE1588 for synchronization –  Average jitter important to keep low –  Sync. Accuracy in the microsecond range

•  Mobile fronthaul - CPRI over Ethernet –  Peter Ashwood Smith, IEEE 802, July 2014:

• 100us – Maximum one way Delay between Antenna and Compute • 65ns – Maximum variation in Delay (Jitter). • 1-10G – Throughput per antenna (compression possible). • 10-12 – Maximum Bit Error Rate

REQUIRED OBSERVED Average latency = 3us per hop Peak Jitter = +/- 2500ns per hop

Peak-Jitter is the main challenge!

TRANSPACKET

Mechanisms for reducing latency and packet jitter (PDV)

Preemption minimizes delay FUSION minimizes packet-jitter

(patent/application: PCT/GB12/50202)

12

TRANSPACKET

Without Hold and Release

•  Preemption isn’t instantaneous. •  Packets with less than min packet size left to

transmit or packets less than 123 octets can’t be preempted.

•  In many use cases, this delay is short enough but not in all cases.

pMAC tx

eMAC tx

MAC Merge tx

IPG

> Min mPacket left

IEEE 802 Berlin March 2015 meeting Monday tutorial IEEE 802.3br

TRANSPACKET

Virtual wavelength from A→D

Underutilized circuit (wavelength): Fusion fills it!

Incoming packets to virtual wavelength,

destined for node D

Time between packets is unused L

–  Pure WDM system (circuit) gives low channel utilization

– 

Fusion switch will insert lower quality (Internet) traffic in voids

Transit traffic passes through on optical layer with minimum or no processing

B

C

C

Packet Switch

D

Input Queue Output Queue

D

C D

D

D

C

A B C D

TRANSPACKET

GST  1  

GST  2  

GST  3  

GST  4  

      1  2  

  3  4  

                 

§  Round-­‐robin  on  the  SM  input  queues  

§  Schedule  SM  packet  that  7its  the  gap    

   

SM  input  queues

§  Reduces  head  of  line  blocking!  

§  One  queue  per  port  :  Avoids  packet  re-­‐ordering  

Time-window: Size of packet gaps are known Lower quality streams

GST: Guaranteed Service Transport stream packets

TRANSPACKET

•  Time-window detection introduce a fixed delay on high-priority traffic, but packet-jitter is zero.

•  The introduced delay = Duration of one lower-priority MTU, e.g. 1518 Bytes

•  Preemption introduce a delay and packet-jitter of 123 Bytes + IFG = 137 Bytes –  Approx. 1/10 of the FUSION time-window approach

Delay and packet-jitter

TRANSPACKET

•  Assuming MTU = 1518 Bytes –  1 Gb/s = 12 microseconds –  10 Gb/s = 1.2 microseconds –  100 Gb/s = 120 nanoseconds

•  Fibre delay is 5 microseconds/km •  Optical networks are typically 10`s of kilometers

and minimum 10 Gb/s •  In optical networks one MTU delay is typically

low compared to transmission delay.

Delay and packet-jitter but how much?

TRANSPACKET

•  FUSION Time-window –  Zero packet-jitter on high priority traffic –  No packet fragments – No “Tax” –  Retransmission mechanisms not required –  Higher throughput on lower-quality traffic than pre-

emption –  Easy to introduce gradually, node-by-node

•  Preemption –  Lower latency than FUSION Time-Window

Pre-emption versus time-window

TRANSPACKET

•  Alternative to 802.1Qbv time gated queues •  Principle:

–  Do a photocopy of each of the incoming lower bitrate streams

–  Place the copy into dedicated (virtual) time-slots, i.e. a container in a higher bitrate stream

–  The copy includes both packets and gaps between packets

–  Every container starts with a synchronization packet

FUSION Guaranteed stream aggregation

TRANSPACKET

1  Container  1  

Container  4  

2  

Container  2  

Container  3  

Container  5  

4  x  maximum  MTU  

Input  Ethernet  Guaranteed  (GST)  streams:  

Input  1  

Input  2  

Input  3  

Input  4  

Input  5  

Fixed  aggregation  delay  α  :    Snapshot  of  all  streams  with  packets  and  inter-­‐packet  lengths  

Guaranteed streams to be aggregated

TRANSPACKET

E.g.  1GE  to  10GE  

1  Container  1  

Container  4  

2  

Container  2  

Container  3  

Container  5  

4  x  maximum  MTU  

Gap  2  

Gap  1  

Gap  3   1  

Container  1  

2   Gap  2  

Gap  1  

Gap  3  

Add  Container  Control  Packet  (CCP)  

for  port  (container)  level  synchronization  

Maintain  length  of  packets  and  gaps  

Packet-­‐gaps  preserved  

Input  Ethernet  Guaranteed  (GST)  stream  

Gaps preserved during aggregation

TRANSPACKET

1  Container  1  

Container  4  

2  

Container  2  

Container  3  

Container  5  

4  x  maximum  MTU  

+

5                    4                    3                  2                      1  

Add  Master  Control  Packet  (MCP)  

for  frame  level  synchronization  

Frame  consists  of  N  (=  5)  containers  

Aggregated guaranteed streams

Grey:  Packet  gaps  available  for  lower  quality  traf7ic  

TRANSPACKET

Aggregated  GST  streams  

                              SM streams (GbE)

GST gap detector

                                                     

SM schedul

er SM

Buffer

     

     

           

5                    4                    3                  2                      1   5                    4                    3                  2                      1  

fixed delay δ

Adding packets into guaranteed streams using time-window approach

Aggregated  GST  streams  with  inserted  Lower  quality  statistically  multiplexed  (SM)  streams      

TRANSPACKET

Performance of FUSION

Experimental results

24

TRANSPACKET

•  Some packet jitter is unavoidable –  Clock domain conversion, typ. < 10 ns at 10 Gb/s –  Header processing (may be fixed) –  Contention and buffering

•  Microseconds/milliseconds jitter •  Jitter depends on traffic-load and scheduling mechanisms

Packet-jitter and latency sources

TRANSPACKET

xe0xe1

ge0

ge1

ge2

ge3

ge4

ge6

ge7

ge8

ge9

N1

Anritsu  Traffic  generator

Gigabit  Ethernet  ports

xe0xe1

ge0

ge1

ge2

ge3

ge4

ge5

ge6

ge7

ge8

ge9

N2

GST

SM

ge5

10  Gb/s  Ethernet  ports

10GE Guaranteed stream (GST) 2 X GE added using time-window (SM)

SM jitter at GST load = 0.5

SM jitter at GST load = 0.2

GST jitter independent of load

TransPacket H1 nodes

MD1230B

TRANSPACKET

Packet-jitter GST 10GE stream for GST load = 0.5

•  Y-axis: Frame count, X-axis: Latency intervals •  GST jitter < 50 ns and independent of SM load

TRANSPACKET

•  Y-axis: Frame count, X-axis: Latency intervals •  SM jitter in the microsecond range

Packet-jitter for lower quality (SM) streams, GST load = 0.5

TRANSPACKET

xe0xe1

ge0

ge1

ge2

ge3

ge4

ge5

ge6

ge7

ge8

ge9

N1

Anritsu  Traffic  generator

Gigabit  Ethernet  ports

xe0xe1

ge0

ge1

ge2

ge3

ge4

ge5

ge6

ge7

ge8

ge9

N2

GST

SM

5 X Guaranteed streams (GST) + 2 X SM aggregation: P-Jitter for SM and GST

GST jitter < 250 ns

SM jitter > 14 Microseconds

TransPacket H1 nodes

MD1230B

TRANSPACKET

•  FUSION (IHON) Time-window minimizes packet-jitter –  Virtual wavelength services –  Mobile back-haul: IEEE1588 Sync accuracy

depends on packet-jitter –  Mobile front-haul (Radio over Ethernet, IEEE

1504.3), jitter < 50 ns achievable over several hops •  FUSION (IHON) guaranteed stream

aggregation –  Minimizes packet-jitter –  Requires buffering at ingress (60 microsecond in

experiment)

Summary and conclusion

TRANSPACKET