d2.4 synchronization service in carrier-grade ethernet...

D2.4

Synchronization service in Carrier-Grade

Ethernet environments

Abstract:

In the context of mobile backhaul network migration from Time Division Multiplexing (TDM)

technologies towards Packet Switched Networks (PSNs) supported by Carrier-Grade Ethernet

as one of the preferred technologies, frequency and time distribution represents a major

challenge. This paper presents the work carried and delivered by Task 2.4.

After briefly reviewing the work accomplished within the mid-term report (M24), this paper

moves to thoroughly discuss the new contributed work focusing on the study of the various

factors that impact time/frequency distribution in PSNs as well as new apparatuses to

guarantee performance compliance with the new requirements imposed by the new cellular

technologies.

Editors:

Alon Geva (RAD Data Communications Ltd)

Authors:

Dinh Thai Bui (ALBLF), Michel Le Pallec (ALBLF), Sebastien Jobert (FT), Alon Geva (RAD)

Date of publication:

October 31th 2010

Synchronization service in CGE environment

TIGER2 – Together IP, GMPLS and Ethernet Reconsidered, Phase 2 /76

Table of Content

1 EXECUTIVE SUMMARY ................................................................................ 9

2 INTRODUCTION .......................................................................................... 11

3 TERMINOLOGIES ....................................................................................... 13

4 MOBILE NETWORK SYNCHRONIZATION REQUIREMENTS .................... 14

5 KEY PROTOCOLS AND THEIR RELATED DEPENDENCIES ....................... 15

5.1 Protocol overview ............................................................................................................................ 15

5.2 Sync-E constraints and performance dependencies ..................................................................... 16

5.3 Timestamp protocols constraints and performance dependencies ............................................. 16

5.3.1 NTP and PTPV2 comparison ...................................................................................................... 16

5.3.2 IEEE 1588V2 constraints and performance dependencies ...................................................... 16

6 DEPLOYMENT SCENARIOS ........................................................................ 19

6.1 Mobile Operator operating all backhaul segments and Sync-E fully deployed ........................... 19

6.2 Mobile Operator operating all backhaul segments and Sync-E partially deployed ................... 20

6.3 Mobile Operator not operating all backhaul segments ................................................................. 21

6.4 Operator deployment focuses ......................................................................................................... 21

7 IMPROVING TIMING-OVER-PACKET (TOP) .............................................. 23

7.1 Summary of work accomplished in M24 ...................................................................................... 23

7.2 Study of Packet Delay Variation (PDV) accumulation over packet networks, for the purpose of

delivering frequency synchronization ....................................................................................................... 25

7.2.1 The need for developing engineering rules for packet based methods ................................... 25

7.2.2 Analysis of PDV generated by telecom networks ..................................................................... 26

7.2.3 Model of a packet slave clock .................................................................................................... 32

7.2.4 PDV accumulation study ........................................................................................................... 33

7.2.5 Conclusion (for clause 7.2) ........................................................................................................ 42

8 TIME AND PHASE DISTRIBUTION OVER PACKET NETWORKS WITH

FULL OR PARTIAL NETWORK SUPPORT .................................................................... 43



8.1 Summary of work accomplished in M24 ...................................................................................... 43

8.2 LTE IPSec challenging environment ............................................................................................. 45

8.2.1 Introduction ............................................................................................................................... 45

8.2.2 LTE IPSec environment description ......................................................................................... 45

8.2.3 Traditional OPS deployment discussion .................................................................................. 46

8.3 New protocol-agnostic mechanism to monitor the DA ............................................................... 47

8.3.1 Principle description with a first proposed implementation .................................................. 47

8.3.2 Alternative implementation of the Delay Asymmetry mechanism ......................................... 50

8.3.3 Conclusion .................................................................................................................................. 50

8.4 New protocol-agnostic mechanism to dampen PDV .................................................................... 51

8.4.1 Packet Delay Variation (PDV) root causes ................................................................................ 51

8.4.2 Case of single synchronization flow .......................................................................................... 52

8.4.3 Case of multiple concurrent synchronization flows ................................................................. 53

8.4.4 Discussions on implementation and conclusion ...................................................................... 55

8.5 Conclusions and outlook (Clauses 8.2 to 8.4) .............................................................................. 55

8.6 Link-to-link TOD distribution using the Sync-E’s ESMC messaging channel ........................... 56

8.6.1 Context description .................................................................................................................... 56

8.6.2 The ESMC ....................................................................................................................................57

8.6.3 Proposals .................................................................................................................................... 59

8.6.4 The Simulation Environment .................................................................................................... 63

8.6.5 Simulation Results ..................................................................................................................... 70

8.6.6 Conclusions ................................................................................................................................ 74

9 CONCLUSION ............................................................................................. 76



Index of Tables

Table 1 – Mobile networks: synchronization requirements at the air interface level ............. 14

Table 2 – ESMC PDU format (Table 11-3/G.8264) ................................................................. 58

Table 3 – QL TLV format (Table 11-4/G.8264) ....................................................................... 59

Table 4 – TOD enhanced ESMC PDU format (proposed changes are highlighted)............... 60

Table 5 – TOD TLV format (proposed changes are highlighted) ............................................ 61

Table 6 – Signal structure of the ESMC signal API ................................................................. 65

Table 7 – Signal structure of the Time/Frequency reference signal API ................................66

Table of Figures

Figure 1 – PTPV2 principles .................................................................................................... 17

Figure 2 – MO owning all backhaul segments with Sync-E fully deployed ............................ 19

Figure 3 – MO owning all backhaul segments with Sync-E partially deployed ..................... 20

Figure 4 – MO not owning all backhaul segments .................................................................. 21

Figure 5 – Distributed ranging mechanism............................................................................. 24

Figure 6 – SSM Tunnel across ToP domain ............................................................................ 25

Figure 7 - Load pattern and resulting delay steps generated during congestion periods (figures extracted from [16]) ........................................................................................... 28

Figure 8 - Delay steps generated with reasonably low levels of load (figure extracted from [20] and [24]) ................................................................................................................... 29

Figure 9 - Comparison of PDV histograms with lowest priority (first plot - peak-to-peak = 302µs), intermediate priority (second plot -peak-to-peak = 161µs), and highest priority (last plot - peak-to-peak = 43µs) for the timing packets in case of congestion (figures extracted from [16]) ......................................................................................................... 30

Figure 10 - PDV measurement when changing the size of the background traffic (figure extracted from [16]) .......................................................................................................... 31

Figure 11 - Comparison of PDV histograms generated by a core equipment (first plot - peak-to-peak = 302µs) and by an access equipment (second plot - peak-to-peak = 1.55ms) when timing packets are not prioritized in case of congestion (figures extracted from [16])................................................................................................................................... 32

Figure 12 - Figure A.1/G.8263 draft - Function model of a PEC-S packet-based clock (source of the data: International Telecommunication Union, ITU-T Working Party 3/15 Temporary Document 318, G.8263 latest draft, 16 May 2010) ...................................... 33

Figure 13 - Illustration of a 60µs floor delay window .............................................................. 34

Figure 14 - Network PDV accumulation model ....................................................................... 35

Figure 15 - PDV of PTP Sync messages and corresponding histograms ................................. 37

Figure 16 - Configuration with 2 nodes used during the experimentation ............................ 38

Figure 17 - Histograms and Cumulative Distribution Functions comparison at 80% of traffic load ................................................................................................................................... 39

Figure 18 - Analysis of successive convolutions ...................................................................... 41

Figure 19 - Automatic computation of the PTPV2 topology thanks to an improved BMC taking into account a local Sync-E support ......................................................................44

Figure 20 – NTP ADD/DROP concept ....................................................................................44

Figure 21 - IPSec tunnel deployment in LTE ...........................................................................46

Figure 22 - IPSec ESP tunnel mode .........................................................................................46

Figure 23 - BCs deployment scenario within a LTE IPSec environment ................................ 47



Figure 24 - First way to implement the new DA monitoring mechanism .............................. 48

Figure 25 - Alternate implementation of the new DA monitoring mechanism...................... 50

Figure 26 – Variation of delay with loading, as referred to G.8061 (Figure I.2) ..................... 51

Figure 27 – “Premium” packet transmission delayed by a “Best Effort” packet ..................... 52

Figure 28 - Beating effect as referred to G.8261 (Figure 20) .................................................. 52

Figure 29 - Timing flows delayed by a multiple of the elementary delay ∆ms ........................ 54

Figure 30 – Possible implementation of the new PDV dampening mechanism ..................... 55

Figure 31 – TOD ESMC messages exchange ........................................................................... 62

Figure 32 – node prime functional block inputs/outputs .......................................................64

Figure 33 – node prime functional block internal structure ................................................... 67

Figure 34 – PRTC prime functional block .............................................................................. 68

Figure 35 – Time Error Analyzer prime functional block ......................................................69

Figure 36 – Time error scope ..................................................................................................69

Figure 37 – Steady-state test setup ......................................................................................... 70

Figure 38 – Steady state test: distribution chain convergence process (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan, node9=magenta, node10=yellow) ............................................................ 71

Figure 39 – Steady state test: distribution chain steady-state performance (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan, node9=magenta, node10=yellow) ............................................................ 72

Figure 40 – Ring restoration test setup .................................................................................. 72

Figure 41 – Ring restoration process (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan, node9=magenta, node10=yellow) ................................................................................................................ 73

Figure 42 – Steady state test: distribution chain steady-state performance (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan, node9=magenta, node10=yellow) ............................................................ 74



Acronym list

1588v2: Std 1588TM-2008

BC: Boundary Clock

BMCA: Best Master Clock Algorithm

CAPEX: CAPital Expenditure

CES: Circuit Emulation Service

CGE: Carrier-Grade Ethernet

DNU : Do Not Use

DSLAM: Digital Subscriber Line Access

Multiplexer

EWMA: Exponentially Weighted Moving Average

E2E: End-to-End

ESMC: Ethernet Synchronization Messaging

Channel

ESSM: Ethernet Synchronization Status Messages

FDD: Frequency Division Duplexing

GbE: Gigabit Ethernet

HSDPA: High Speed Downstream Packet Access

HSUPA: High Speed Upstream Packet Access

IEEE: Institute of Electrical and Electronic

Engineers

IETF: Internet Engineering Task Force

IP: Internet Protocol

ISP: Internet Service Provider

LAN: Local Area Network

MA: Moving Average

MO: Mobile Operator

NE: Network Element

NodeB: UMTS BTS

NTP: Network Time Protocol

OPEX: OPerational Expenditure

OUI: Organization Unique Identifier

OSSP: Organization Specific Slow Protocol

P2P: Peer-to-Peer

PDF: Probability Density Function

PDH: Plesiochronous Digital Hierarchy

PDV: Packet Delay Variation

PDU: Payload Data Unit

PLL: Phase-Locked-Loop

ppb: part-per-billion

ppm: part-per-million

PPS: Pulse per Second

PRC: Primary Reference Clock

PSN: Packet Switched Network

PTP: Precised Time Protocol (other name of IEEE

1588 protocol)

QoS: Quality-of-Service

QL: Quality Level

RoI: Return-on-Investment

SDH: Synchronous Digital Hierarchy

SN: Sequence Number

SSM: Synchronization Status Message

SSF: Server Signal Fail

STM-1: Synchronous Transport Module 1

(155.44Mbit/s)

STP: Spanning Tree Protocol

Sync-E: Synchronous Ethernet

TICTOC : Timing over IP Connection and

Transfer of Clock

TC: Transparent Clock

TDD: Time Division Duplexing

TDM: Time Division Multiplex

TLV: Type Length Value

TE: Traffic Engineering

ToD: Time-of-the-Day

ToP: Timing-over-Packet

TQL: Time Quality Level

TS: Time Stamp

Tx: transmitter

UMTS: Universal Mobile Telecommunications

System

LTE: Long Term Evolution (3GPP 4G)

UMTS: Universal Mobile Telecommunication

System (3GPP 3G)



References

[1] K. O’Donoghue - Synchronization in Packet networks – NTPv4 and IEEE 1588 - International Telecom Sync Forum (ITSF) 2007, November 13th 2007.

[2] IETF RFC5481 - Packet Delay Variation Applicability Statement – March 2009. [3] ITU-T SG15/Q13 WD35 - Effect of routers’ architecture on timing distribution - San Jose, March

2009. [4] 3GPP TS 33.210 version 9.0.0 Release 9 – Network Domain Security (NDS); IP network layer

security. ETSI TS 133 210 V9.0.0. February 2010. [5] 3GPP TS 33.310 version 9.1.0 Release 9 – Network Domain Security (NDS); Authentication

Framework (AF). ETSI TS 133 310 V9.1.0. January 2010. [6] 3GPP TS 33.401 version 9.2.0 Release 9 – 3GPP System Architecture Evolution (SAE); Security

architecture. ETSI TS 133 401 V9.2.0. January 2010. [7] K. Correll, N. Barendt, M. Branicky - Design Considerations for Software Only Implementations

of the IEEE 1588 Precision Time Protocol – Proceedings of 2006 conference on IEEE 1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems.

[8] ITU-T G.8261/Y.1361. Timing and synchronization aspects in packet networks. February 2008. [9] ITU-T SG15/Q13, C685 – Huawei Technologies Co. Ltd, China Academy of Telecommunication

Research, MII - Packet Master Traffic Load Analysis based on ITU-T PTP profile message rates – Lannion, December 2009.

[10] Contribution C449 to SG15 by France Telecom, ‘Proposal of transmission of a phase information using OAM messages’, Geneva, May 2007.

[11] ITU-T G.8264 – Distribution of timing packets through packet networks – October 2008. [12] IEEE Std 1588-2008, IEEE Standard for a Precision Clock Synchronization Protocol for

Networked Measurement and Control Systems, July 2008. [13] ITU-T G.8262 - Timing characteristics of synchronous Ethernet equipment slave clock (EEC) –

August 2007. [14] ITU-T G.781 – Synchronous Layer Functions – September 2008. [15] M. Le Pallec, D.T. Bui, F. Dorgeuille and N. Le Sauze – Time and Frequency Distribution Over

Packet Switched Networks. Bell Labs Technical Journal 14(2), 131-154. [16] WD 60 - About the control of PDV using QoS mechanisms and the applicability of proposed PDV

metrics (MAFE and minTDEV), France Télécom contribution to ITU-T, interim meeting Q13/15, San Jose, March 2009

[17] WD 38 - Additional PDV measurements on Ethernet platform, France Télécom contribution to ITU-T, interim meeting Q13/15, Edinburgh, June 2009

[18] WD 39 - PDV measurements on Micro Wave platform, France Télécom contribution to ITU-T, interim meeting Q13/15, Edinburgh, June 2009

[19] C 668 - Performance aspects of a packet slave clock targeting 50ppb and based on a physical clock with a large time constant and PDV accumulation discussions, France Télécom contribution to ITU-T, plenary meeting SG15, Geneva, September 2009

[20] WD 36 - PDV measurements over different platforms – extract from FT presentation at ITSF 2009, France Télécom contribution to ITU-T, interim meeting Q13/15, Lannion, December 2009

[21] WD 37 - Continuation of the considerations on the performance aspects of a packet slave clock targeting 50ppb and based on a physical clock with a large time constant and PDV accumulation discussions, France Télécom contribution to ITU-T, interim meeting Q13/15, Lannion, December 2009

[22] WD 47 - Network PDV accumulation proposal, France Télécom contribution to ITU-T, interim meeting Q13/15, San Jose, March 2010

[23] C 780 - Test plan for PDV generation of a single equipment involved in IEEE1588-2008 transport, Cisco / France Télécom contribution to ITU-T, plenary meeting SG15, Geneva, June 2010

[24] C 1017 - Network PDV accumulation study, France Télécom contribution to ITU-T, plenary meeting SG15, Geneva, June 2010



[25] Study of network Packet Delay Variation for packet based methods, France Télécom article submitted to IEEE Communication Magazine, June 2010

[26] ITU-T SG15/Q13, C685 – Huawei Technologies Co. Ltd, China Academy of Telecommunication Research, MII - Packet Master Traffic Load Analysis based on ITU-T PTP profile message rates – Lannion, December 2009

[27] Taks 2.4 mid-term report on Synchronization service in Carrier-Grade Ethernet environments (M24), Feb 2010.

[28] ITU-T G.8265.1 - IEEE1588™ profile for telecom (frequency delivery without support from network nodes)– June 2010

[29] IETF RFC 5905 – Network Time Protocol Version 4: Protocol and Algorithms Specifications – June 2010.

[30] David L. Mills – Clock Disciplining Algorithm of the Network Time Protocol Version 4 - Technical Report 97-3-3, Electrical Engineering Department of Delaware, March 1997.



1 Executive Summary

The migration from TDM to Packet based (IP, MPLS and Ethernet) cellular backhaul networks

has imposed a very difficult challenge for the network designers and operators. The inherit frequency

transparency attribute of TDM was no longer available. New advanced techniques had to be developed

in order to deliver what was already obvious with TDM: A stable, accurate frequency reference to

derive the transmission clock of the base-station. These days, the challenge is further increasing, as

new, more bandwidth efficient cellular technologies (e.g. TD-SCDMA and LTE) depend on the supply

of an accurate Time-of-Day reference signal to the base-station. As a consequence, it is commonly

understood today by the designers and service providers of cellular networks that, along with the

layout of the data plan, careful attention and planning for the synchronization issues must

simultaneously take place.

This document comprises the synchronization related work, carried by the members of Task 2.4

within the TIGERII project over the past 1.5 years. The various contents within this document present

the accomplishments of Task 2.4 based on the initial objectives definition of the Task. The work items

described in this document complement and extend previous work already published in the mid-term

report document M2.4 [27].

At the beginning of the work, Synchronous Ethernet (Sync-E), developed by SG15/Q13 of the

ITU-T in 2008, was identified as the prime technology to distribute accurate frequency information

over packet based network that rely on Ethernet as their link layer protocol (indeed, this is the case for

the majority of green-field cellular backhaul networks). The major appeal, especially for the large

Telecom carriers, is the guaranteed level of performance and the seamless interwork with the legacy

TDM networks. Based on this understanding, three backhaul network deployment scenarios have

been identified:

1. Mobile Operator operating all backhaul segments and Sync-E fully deployed

2. Mobile Operator operating all backhaul segments and Sync-E partially deployed

3. Mobile Operator not operating all backhaul segments (hence, the existence of Sync-E cannot

be guaranteed)

The work plan of Task 2.4 was based upon the above scenarios, trying to answer various open issues

concerning each and every one of them, for the benefit of the industry. The work was also focusing on

problems and issues which are currently not extensively treated by the various SDOs that work on

these topics (e.g., ITU-T SG15/Q13, IETF TICTOC and NTP WGs etc.) with the intention that this

work package could be reused by the SDOs once completed to derive standardized solutions.

The work plan of Task 2.4 was comprised of the following work items that were selected in order

to cover the above mentioned scenarios:

1. Improving ToP end-to-end performance using the following approaches:

• Studying the effect of PDV accumulation on the end-to-end frequency recovery

performance and understanding how different QoS mechanisms could be used to

improve it.

• Studying new mechanisms than can be used to better control end-to-end

performance.

2. New TOD/phase distribution methodologies that rely on network support. Two

main study courses were selected for this item:

• Understanding how interworking with an underlying Sync-E frequency distribution

can be used to improve overall performance in the case where Sync-E is partially

deployed or fully deployed.



• Study a new link-by-link phase distribution scheme that is based on the Sync-E ESMC

SSM signalling channel.

Work item 1 is mainly addressing frequency distribution under the assumptions of network

deployment scenario (3), where physical-layer based frequency distribution does not exist. Hence,

frequency distribution must rely on high-layers ToP schemes that are inherently very susceptible to

Packet Delay Variation (PDV). Chapter 6 of M24 [27] embarked on this work, discussing a few

innovative mechanisms that could be used in order to introduce more control into the ToP

environment, aiming towards a more constrained end-to-end error budget. This work is now further

complemented by clause 7.2 of this report, comprising a comprehensive study of the way PDV is being

accumulated in legacy packet networks that does not have special capabilities to handle the ToP

traffic. The work presented in this clause constitutes an important milestone towards understanding

the PDV limits which are required in order to be able to guarantee a certain level of performance and

how standard QoS/CoS mechanisms could be efficiently employed in the network in order to increase

the level of expected performance.

The focus in work item 2 has turned to accurate time and phase1 information distribution in

packet networks, a very ‘hot’ topic that is currently being studied by many SDOs and Academic

institutes. The extremely tight end-to-end time/phase error budget (less than 1 usec in many

applications) has driven the entire industry to the understanding that network support, in one way or

another, is mandatory. This notion was also adopted by the work of task 2.4 as a principal guideline.

The first part of the work is presented in chapter 7 of M24 [27] that suggests a few interworking

concepts between various available technologies (e.g., Sync-E and PTP or PTP and NTP) to better

exploit these technologies and form a more overall optimized solution. This work in now extended

with two additional study items that occupy chapter 8 of this report. The first addresses the challenge

inherit by the use of IPSec tunnels for the synchronization distribution while the second discusses a

TOD distribution scheme that is confined to the physical layer (actually lower part of the link layer)

and which superfluous any higher-layer TOD distribution scheme (e.g. PTPv2), thus, saving valuable

network bandwidth and management resources.

1 Absolute phase synchronization among a group of devices means that a certain periodic event is

occurring in all the devices in the group at exactly the same time. In that scope, knowledge regarding

the specific Time-of-Day (TOD) instants at which the events are occurring is of non-importance. TOD

synchronization implies phase synchronization with the additional requirement that the specific

Time-of-Day (TOD) information related to the occurrence of the events is also distributed.



2 Introduction

In the context of a strong development in data services (the majority of applications are now

over IP), especially in the mobile networks, traditional TDM infrastructure is not any more able to

meet the following equation: offering more bandwidth and guaranteeing a viable RPU (Revenue Per

User) (as explained in the deliverable D20). This gives strong arguments to operators to migrate their

transport infrastructure towards Packet Switched Networks (PSNs) with Carrier-Graded Ethernet

(CGE) being one technology of choice. However, many technical challenges are raised for this

migration and the synchronization network requirements represent a major hurdle.

Indeed, during the migration of the transport infrastructure from the TDM towards the PSN,

synchronization is lost, as the traditional Ethernet is not being capable of distributing neither a

frequency reference nor a time/phase reference. Alternatives such as the GPS (Global Positioning

System) raise many technical issues and cost drawbacks (coverage, interference, cost of the numerous

receivers, cabling and engineering costs, radio-frequency jamming, etc) as well as political concerns

(some countries avoid from being GPS dependant).

Within this framework, the ITU-T’s Synchronous Ethernet (Sync-E) is seen as the preferred

technology to distribute a frequency reference and the IEEE‘s 1588V2 (PTPV2) is considered, by

major telecom actors, as the preferred technology to distribute a time or phase reference. Concerning

this latter aspect, the IETF’s Network Time Protocol (NTP) is also to be considered, as per existing

deployments.

In this document, studies will focus on the mobile backhauling as the target application for the

network synchronization over the PSN. Such focus is motivated by the related stringent mobile

synchronization requirements driven by an optimal use of rare radio resources under high speed

mobility constraints (e.g. interferences due to Doppler effects). Through different deployment

scenarios, potential issues will be pointed out and related network requirements will be described.

This report complements mid-term report (M24) submitted in Feb 2010. The work materialized

into M24 was mainly focusing on innovative ways to mitigate Packet Delay Variation, the most

dominant component affecting time/frequency distribution performance in packet networks. Based

on the carrier (FT) input, three typical network deployment scenarios have been introduced,

representing a cellular mobile operator migration process from a case where it does not operate the

backhaul segments (thus having only little control over the synchronization support capabilities in the

network) to a case where it completely owns the backhaul segments, thus having direct control over

the various synchronization support means within the network. The work within M24 has contributed

a few innovative ideas to improve E2E performance for the various deployment scenarios.

The work in this report complements the previous work, emphasizing on the following work

items in particular:

• Improving E2E time/frequency distribution with the use of IPSec tunnels. This work item is

addressing a unique case where classic on-path support schemes like Transparent Clocks

cannot be used dues to the use of the IPSec tunnels.

• Comprehensive study of the way PDV is being accumulated across a frequency distribution

chain in packet networks. This study in extremely important in understanding the achievable

E2E frequency distribution performance given a specific network topology as well as load

profile.

• Accurate distribution of Time of Day (TOD) information using an extension to the standard

Sync-E frequency distribution scheme. Such a TOD distribution apparatus can be especially

appealing in backhaul networks having complete Sync-E distribution capabilities as it will



superfluous any additional higher-layer TOD distribution scheme running on top of the

Sync-E (e.g. PTPv2) and save network resources.

This report is organized in the following way: Chapter 4 presents the key motivation for the work

as is expressed by the strict synchronization requirements of new mobile technologies. Chapter 5

describes the dominant technologies used for time/frequency distribution across packet networks.

Chapter 6 discusses the key network deployment cases as they were identified by France Telecom and

from which the entire work items have been derived. Chapter 7.1 comprises a brief summary on the

accomplishment in M24 addressing networks that do not offer any on-path support (2nd and 3rd

deployment scenarios) while chapter 7.2 comprise the new study on PDV accumulation. Clauses 8.2 to

8.4 discuss the new challenges imposed by the use of IPSec tunnels across the backhaul network,

presenting a few innovative ideas to overcome them. Last, clause 8.6 extends the work already partly

presented in M24, regarding extending Sync-E to distribute TOD, with the new simulation

environment and the obtained results.



3 Terminologies

In order to fully understand the content of this document, some terminological definitions are

provided below.

On-Path Support (OPS): On-path support refers to any intended feature of the path over which

synchronization (frequency, phase or time-of-day) is being distributed that may improve the quality of

the distribution.

On-Path Support Node: A node along the communication path between a “Master” and a “Slave”

that offers on-path support for the time/frequency distribution packets.

Syntonization: Frequency synchronization. Two oscillators are syntonized when they have the

“same” frequency within a defined error. The latter is defined as the syntonization accuracy. When

two oscillators are syntonized the difference between their respective phases remains constant with

time.

Phase Synchronization: Absolute phase synchronization among a group of devices means that a

certain periodic event is occurring in all the devices in the group at exactly the same time. In that

scope, knowledge regarding the specific Time-of-Day (TOD) instants at which the events are occurring

is of non-importance

TOD Synchronization: TOD implies phase synchronisation with the additional requirement that

the specific Time-of-Day (TOD) information related to the occurrence of the events is also distributed.



4 Mobile network synchronization requirements

Within the mobile network infrastructure, radio interfaces impose the most stringent

synchronization constraints as explained in Chapter 1. Two technology types are to be distinguished:

- FDD (Frequency Division Duplex) technologies only require network syntonization

- TDD (Time Division Duplex) technologies require both network syntonization and

alignment in phase (or phase synchronization).

The mobile network requirements at the radio interface for different mobile technologies are

summarized in the following table:

Radio System Frequency accuracy Time accuracy

GSM 50 ppb NA

UMTS FDD 50 ppb NA

UMTS TDD 50 ppb 2.5 µs (1.5µs*)

CDMA2000 50 ppb 3 µs (10 µs worst case)

TD-SCDMA 50 ppb 3 µs

LTE FDD 50 ppb NA

LTE TDD 50 ppb 3 µs

WIMAX 802.16e 8 ppm 0.5-25µs according to system profile

(*) some standards define max base station to base station time difference and not a difference against a common reference as

here

Table 1 – Mobile networks: synchronization requirements at the air interface level

It is to be noted that radio base stations introduce noises into the synchronization signal. Thus,

requirements are typically more stringent at the network interface level (output of the backhaul

network) than at the air interface level. As an illustration, a 16 ppb value at the network level is often

mentioned while targeting a 50 ppb frequency accuracy at the radio interface level [8].



5 Key protocols and their related dependencies

5.1 Protocol overview

The following sections shortly describe the three main synchronization protocols that will be the

study focus within the TIGER II project.

The ITU-T Synchronous Ethernet protocol is purely dedicated to the distribution of a frequency

reference within an IEEE 802.3 Ethernet-capable network. It is based on a hierarchical

synchronization method using a synchronous physical layer. As specified by the ITU-T within a

migration perspective, Sync-E is aligned with SDH/SONET) in the sense that a Sync-E clock element

can be inserted seamlessly into a SDH/SONET clock hierarchy. This approach should be efficient in

terms of performance as Sync-E is expected to be at the same performance level as SDH/SONET.

The IEEE 1588v2 protocol, also called as the Precision Time Protocol version 2 (PTPv2),

provides a standard protocol for synchronizing clocks within a PSN. PTPv2 was approved as a revision

standard by the IEEE standards Board in March 2008 with the intention to evolve the IEEE 1588v1

from its first deployed local area network (LAN) environment. Many enhancements have been added

for this purpose, including concepts of transparent clock, boundary clock, and unicast support. Some

of these elements will be detailed later in this document. Unlike PTPv1, PTPv2 is specified for

different underlying layers — 802.3/Ethernet, User Datagram Protocol (UDP)/Internet Protocol

version 4 (IPv4), and UDP/Internet Protocol version 6 (IPv6). From this perspective, the protocol is

more suitable for meeting the needs of the telecom industry than its predecessor. The PTPv2 protocol

is designed to distribute a time reference across a communication network between a master clock

and a slave clock.

Growing out of the work conducted by Dr. David Mills at the University of Delaware, the IETF’s

Network Time Protocol (NTP) has taken advantage of lengthy deployment experience with various

standards versions:

- NTPv0 (RFC 958), 1985

- NTPv1 (RFC 1059), 1988

- NTPv2 (RFC 1119), 1989 and

- NTPv3 (RFC1305), 1992.

Today, there are millions of NTP servers and stations deployed worldwide, and NTP software is

implemented on almost every workstation and server platform [1]. The current NTPv4 version is

nearly formalized as a standard IETF RFC and comparatively to the previous version, NTPv4 targets

are:

- To accommodate Internet Protocol version 6 address space,

- To extend potential accuracy to the tens of microseconds for modern workstations and fast LANs by

including fundamental improvements in the mitigation and discipline algorithms,

- To ease NTP deployment by including a dynamic server discovery scheme, so that in many cases

specific server configuration is not required,

- To specify an autonomous authentication protocol,

and

- To correct certain errors in NTPv3 design and implementation and to include an optional extension

mechanism.



5.2 Sync-E constraints and performance dependencies

Similarly to SDH/SONET, Sync-E has being designed for being independent from the traffic

load. Thus, the experience with SDH/SONET should be beneficial to Sync-E in that the former has

shown it has been capable of synchronizing very large networks with high accuracy. However, making

use of the physical layer to distribute a reference frequency imposes the Sync-E/SDH chain to be

contiguous and many issues can break this rule:

- A Sync-E equipment in the chain is not 100% compliant to the standard

- Interoperability issue between 2 consecutive Sync-E equipments

- Mis-configuration or mis-installation; e.g. an Ethernet port is peer by mistake to a Sync-E port

- ESMC discrepancy; e.g. Synchronization Status Messages (SSM) corrupted

- etc.

Apart from these technical aspects, regarding the full deployment of hardware support on

Ethernet nodes, cost is a major obstacle which could prevent from having a full migration of the

traditional Ethernet towards Sync-E, thus from having a contiguous chain of physical frequency

distribution.

5.3 Timestamp protocols constraints and performance dependencies

5.3.1 NTP and PTPV2 comparison

Unlike PTPv2, which has been specified for different underlying layers, IETF NTPv4 relies on

UDP/IP (version 4 or 6) and is thus a “pure layer 3” solution in terms of connectivity. All future

developments of NTPv4 have been officially shifted to the IETF Transmitting Timing over IP

Connections and Transfer of Clock (TICTOC) working group. This decision has been driven by the

desire to converge existing time protocols and thus to ensure — as a first step — 1588v2/NTPv4

coexistence within the same network. Similar to PTP, NTP has been conceived for a time distribution

purpose, but can be used for syntonization as well. Server-to-client time distribution methodology

over an NTP-capable network (over IP Packet Network) is conceptually close to the 1588v2

master/slave principle regarding the “2-way signalling” (synchronization message exchanges in both

Master-to-Slave and Slave-to-Master directions).

Thanks to many mechanisms introduced such as the hardware timestamping, the Transparent

Clock, the maximum message rate, etc, PTPV2 is often preferred to NTP for the distribution of an

accurate time (and thus frequency). Nevertheless, as final tasks within the TICTOC Transition, the

NTPV4 working group is specifying a “Carrier-Class” NTP, addressing the hardware time stamping

and maximum message rate requirements. With these additional features, the “Carrier-Class” NTP

should provide similar performance with regards to 1588V2 in an end-to-end scheme. Such a scheme

is presently investigated by the ITU-T SG15/Q13 within the context of the first IEEE1588V2

frequency-only Telecom profile (frequency distribution purpose without neither Boundary Clock nor

Transparent Clock support).

In the rest of this document, IEEE1588V2 will particularly be the investigation focus without

forgetting that the (Carrier-Class) NTP could be a relevant candidate as well for most of the scenarios.

5.3.2 IEEE 1588V2 constraints and performance dependencies

For a clear understanding of the document, the following figure briefly recalls the principle of

the PTPV2 protocol (very similar to NTP):



Figure 1 – PTPV2 principles

PTPV2 performance mainly depends on 2 parameters:

- The delay asymmetry: the difference between the transmission delay of the (Mater/Slave) Sync

message and the related Delay-Req message.

- The Packet Delay Variation (PDV): the difference between the transmission delay of a PTP message

and a reference transmission delay (transmission delay of the first message, theoretical minimum

delay, theoretical maximum delay, etc).

The above dependency which can be seen as the error is computed between the estimated offset

and the ideal offset. The latter offset allows the Slave to adjust its timescale exactly to the Master one:

with dMS being the transmission delay of the Sync message and dSM being the transmission delay of the

associated Delay-Req (Slave/Master). Thus,

The offset estimation error is therefore the instantaneous delay asymmetry:

idealSM

idealMS

offsetdtt

offsetdtt

−−−−++++====

++++++++====

34

12

222

)()( 3142 SMMS

estimated

SMMS

ideal

ddoffset

ddttttoffset

−−−−−−−−====

−−−−−−−−

−−−−−−−−−−−−====

2

SMMS

idealestimated

ddoffsetoffseterror

−−−−====−−−−====



This can be developed further as:

(Equation 1)

with,

Within Equation 1, the first term represents the impact of PDV on PTPV2 performance and

the second term the impact of delay asymmetry. Delay asymmetry, as defined above, while remaining

deterministic can be measured and injected into the Slave computation module in order to reach a

better time accuracy. PDV however has random components which are real issues (it is to be noted

that in the above equations, the random components of delay asymmetry has been arbitrary

integrated to the PDVs. These random components could be also integrated to the delay asymmetry as

another definition of this latter parameter).

From a general perspective, the Packet Delay Variation has different random components

[2][3], namely:

- The transmission Delay (variation): transmission delay is the result of the velocity of the signal

between two endpoints of a transmission link and the distance between these two endpoints. The

transmission delay variation is due to many factors depending on the transmission technology

employed (e.g. wireless, wire-line, etc). For example on a copper link, temperature variations can

modify the transmission delay by shrinking or extending the transmission distance.

- The processing delay (variation): the processing delay is the delay resulting from the processing of a

timing packet within a Network Element (NE). It strongly depends on the NE hardware and software.

For example, the processing delay variation can result from a specific CPU (Central Processing Unit)

configuration: a same CPU on a given port could be in charge of in-going and outgoing packets.

Consequently the traffic load in one direction can have a detrimental effect on the other direction.

- The buffering/queuing delay (variation): buffering or queuing delay is the total amount of waiting

time of the timing packet within different buffers or queues in a given NE before being processed and

finally transmitted to the next NE on the communication path. Buffering delay variation is partially

due to the “competition” between timing packet flows and other packet flows, and between different

timing packet flows themselves according to their relative arrival time at different waiting queues and

the priority policies implemented.

It is to be noted that in a telecom environment, the buffering/queuing delay (variation) is

generally the main component of the PDV.

minMSMSMS ddPDV −−−−====

minSMSMSM ddPDV −−−−====

minmin_ SMMS ddAsymmetryDelay −−−−====

2

_

2

AsymmetryDelayPDVPDVerror SMMS ++++

−−−−====



6 Deployment scenarios

Considering deployment scenarios of the synchronization network, three representative use–

cases will be distinguished within Tiger II synchronization task, thanks to two high level variables:

- Backhaul infrastructure ownership: the Mobile Operator (MO) owns the backhaul

infrastructure or not,

- Sync-E availability within the MO owned infrastructure: Sync-E fully or partially deployed.

6.1 Mobile Operator operating all backhaul segments and Sync-E fully

deployed

The first considered deployment scenario is illustrated by Figure 2 below.

Figure 2 – MO owning all backhaul segments with Sync-E fully deployed

This is an ideal scenario or a targeted architecture where Sync-E is fully deployed for a

syntonization purpose. The Mobile Operator (MO) owns all the backhaul segments and thus has the

flexibility to implement on-path support nodes in order to optimize the protocol performance with

respect to end applications. Both time and frequency references are likely to be co-located and time

distribution paths are likely to follow frequency distribution paths in order to take advantage of the

syntonization supports.



6.2 Mobile Operator operating all backhaul segments and Sync-E

partially deployed

This second considered deployment scenario is illustrated by Figure 3 hereafter.

Figure 3 – MO owning all backhaul segments with Sync-E partially deployed

This scenario is more realistic as it considers migration constraints for which Sync-E would be

partially deployed. Similarly to the previous case, the MO takes the great advantage of having control

over the entire backhaul network. In order to distribute the reference frequency towards end

applications, the MO can make use of timestamp protocols (e.g. PTPV2) in order to bound different

physical layer based frequency distribution domains together or to extend one of them for reaching

end applications. Obviously, the MO can also make use of timestamp protocols in order to distribute a

time reference.

Remark: when making use of a timestamp protocol for distributing both frequency and time,

respective time and frequency distribution topologies do not need to be congruent within the entire

network segment while considering their different respective dependencies (PDV dependency for a

syntonization purpose and both PDV and delay asymmetry for a time synchronization purpose).



6.3 Mobile Operator not operating all backhaul segments

The last considered deployment scenario is illustrated by Figure 4 below.

Figure 4 – MO not owning all backhaul segments

This scenario covers numerous cases where the MO does not own all backhaul segments or

does not have any backhaul network at all. Within leased parts of the backhaul network, the MO has

to make use of timestamp protocols in order to distribute both time and frequency. The MO has no

visibility on the potentially available on-path support that can be implemented along the leased

segments. End-to-end approach should be adopted in this case.

6.4 Operator deployment focuses

This chapter presents high level deployment priorities as foreseen/viewed by France Telecom.

The majority of the proposals in this document can be applied to a large number of deployment

scenarios. However, a particular focus will be put on the following choices when it comes to

performance comparison, and especially to simulations and/or test results:

1. For frequency-only distribution:

1.1 Physical-layer based methods (e.g. Sync-E): this is the preferred solution for distributing

frequency. Physical methods are standardized and largely known, thus will not be investigated

within Tiger II research project.

1.2 Packet-based methods:



1.2.1. Packet-based methods without any hardware support (e.g. BC/TC) from the network: it is an

interesting case especially for inter-carrier frequency distribution cases (although physical-layer

based solutions may apply also to this case, e.g. if a Sync-E service is offered as part of the

leased line). Also, this case is relevant for supporting gradual network migration towards

physical layer based technologies as a targeted architecture.

1.2.2. Packet-based methods with partial hardware support (e.g. BC/TC) from the network: regarded

by FT as less important than other approaches. Indeed, if some hardware support is required

from the network for frequency distribution, then Sync-E is seen as the preferred solution.

1.2.3. Packet-based methods with full network hardware support (e.g. BC/TC): regarded by FT as less

important than other approaches. Indeed, if some hardware support is required from the

network for frequency distribution, then Sync-E is seen as the preferred solution.

Summary regarding frequency-only distribution: regarding the deployment of new hardware

support, physical-based methods such as Sync-E are considered as the preferred approach to

guarantee the performance delivered to the end application. In other cases, the use of packet based

methods in an end-to-end approach is preferred.

2. For both frequency and TOD distribution:

2.1 PTPV2 (alternatively NTP) without Sync-E and without network hardware support (e.g.

BC/TC): this is considered by FT to be a challenging case if the purpose is to meet the stringent

LTE phase/time requirement with µs accuracy objective.

2.2 PTPV2 (alternatively NTP) without Sync-E and with partial network hardware support (e.g.

BC/TC): same remark as 2.1. This is considered as a challenging deployment.

2.3 PTPV2 (alternatively NTP) without Sync-E and with full network hardware support (e.g.

BC/TC): this is identified as possibly an interesting case where prolonged holdover is not an

issue.

2.4 PTPV2 (alternatively NTP) with Sync-E and without network hardware support (e.g. BC/TC):

same remark as 2.1. This is considered as a challenging deployment.

2.5 PTPV2 (alternatively NTP) with Sync-E and with partial network hardware support (e.g.

BC/TC): same remark as 2.1. This is considered as a challenging deployment.

2.6 PTPV2 (alternatively NTP) with Sync-E and with full network hardware support (e.g. BC/TC):

this is identified by FT as probably the best technical approach that would certainly allow to

guarantee both required TOD and frequency performance under normal operation conditions

as well as holdover.

2.7 Link-by-link TOD distribution using ESMC messages: this case is identified to be an interesting

solution for networks where Sync-E is already deployed as it will save the need to use PTP in

addition.

Summary regarding frequency and TOD distribution: according to France Telecom, physical

layer frequency distribution should be used to support time distribution (to guarantee time holdover

performance). A fully deployed hardware support is also seen as a targeted architecture in order to

meet stringent LTE phase/time requirement with µs time accuracy objective. TOD distribution can

either be accomplished end-to-end using PTPV2 (or NTP) with full network hardware support or link-

by-link using the ESMC messages (if Sync-E is already deployed).



7 Improving Timing-over-Packet (ToP)

7.1 Summary of work accomplished in M24

The work presented in Chapter 6 of [27] was focusing on ways, both from the time/frequency

distribution master, time/frequency distribution slave and the network element in the middle, to

mitigate and dampen the effect of Packet Delay Variation (PDV) on the overall time/frequency

distribution performance.

In particular, clause 6.1 discussed the dampening of network introduced PDV by applying

special means at the time/frequency distribution master, slave and the network elements in between.

At the master side, controlling the packet-rate and the packets transmission profile of a

time/frequency distribution master device in order to dampen end-to-end network induced PDV, was

discussed. A PDV decrease in one order of magnitude has been argued as accountable by such control.

At the network element, the benefit of accurate timestamping based on dedicated fast logic

hardware was presented. The performance degradation expected in last-mile technologies that do not

have such mechanism was presented.

Different network QoS mechanisms have also been mentioned as sometimes an effective way to

mitigate network PDV. In particular, by differentiating timing signalization flows from other packet

flows using QoS-build-in information within the transport header. This allows for prioritizing

synchronization packets with regards to data packets, and avoiding a direct competition of the latter

against the formers. QoS technique usually relies on parallel buffers whose filling depends on

scheduling policies. It is mentioned that, though being sometimes highly efficient for PDV mitigation,

such QoS techniques cannot entirely make the problem go away. A following in-depth discussion on

this issue is given in Chapter 7.2 of this report.

At the slave side, different filtering mechanisms (a.k.a. ranging mechanisms) that can be

applied in order to enhance the PDV filtering capabilities of slave were presented. In particular, two

traditional mechanisms, the Moving Average (MA) and the Exponentially Weighted Moving Average

(EWMA) mechanisms were discussed presenting their suitability for some PDV profiles as well as

their incompetence to deal with different PDV profiles. A new promising approach focusing primarily

on those packets that experience minimum network delay (sometimes referred to as ‘early packets’)

was also discussed along with a set of new related metrics, such as MinTDEV and MATIE that aim to

better characterize the relationship between a specific PDV profile and obtainable time/frequency

recovery performance.

Clause 6.2 of [27] extended clause 6.1 by suggesting a new Distributed Ranging Mechanism that

offers more PDV control compared to the legacy ranging mechanism that is used by current

time/frequency distribution protocols as was described in clause 5.3.

The new scheme, presented in Figure 5 below, suggests the use of special distributed

“selection/filtering” modules along the time/frequency network distribution path that have the ability

to recognize timing (e.g. PTPV2) packets and to proceed to the ranging process. For each

aforementioned filter module, a ranging process is applied in order to select packets demonstrating

low PDV (e.g. based on packets experiencing minimum delays). If the number of these selected

packets is lower than a certain provisioned threshold (e.g. tm1 for filter 1), the filter module requests

the Master for increasing the timing packet emission rate. In the opposite way, a second threshold can

be provisioned so that the filter module can request the Master to decrease the timing packet emission

rate when the number of selected packets is found to be above this second threshold. This method



guarantees that the Slave receives sufficient relevant packets during its filtering process in order

discipline its local oscillator.

An illustration of the benefits of the apparatus was also given using two representative PDV

examples based on different Probability Density Functions (PDF). Avoiding network congestion that

might be caused by the intended increase of synchronization traffic was discussed.

Figure 5 – Distributed ranging mechanism

Clause 6.4 of [27] discussed ToP as an extension for physical-layer frequency distribution. A

new approach to inter-connect remote isolated Sync-E islands using ToP in general and PTPv2 in

particular was presented. A new concept termed an ‘SSM tunnel’, that allow the PTPv2 protocol to

distribute native Sync-E SSM Quality Level (QL) information between the Sync-E capable islands was

presented (see Figure 6 below). It was argued that this solution will enable the network operators a

smoother migration towards Sync-E by bridging those network parts that still did not migrate towards

the new physical-layer frequency distribution technology.



Figure 6 – SSM Tunnel across ToP domain

It should be mentioned that since the completion of [27], the ITU-T SG15/Q13 group of experts

has defined, within the PTPv2 Telecom Profile suit for frequency distribution [28], the distribution of

native Sync-E QL information using the clockClass field in the PTPv2 Announce message. The

motivation for doing that was to form frequency distribution continuity between the native Sync-E

technology (based on physical layer timing distribution) and packet based PTPv2.

7.2 Study of Packet Delay Variation (PDV) accumulation over packet

networks, for the purpose of delivering frequency synchronization

As already discussed in Chapter 5, Synchronous Ethernet, a technology standardized in ITU-T,

specifies how to synchronize Ethernet PHY in order to achieve a physical layer timing distribution.

However, some already deployed networks do not implement the relevant hardware to support

Synchronous Ethernet. Alternative methods may therefore be required in these cases, at least

temporarily.

Packet based methods, such as Precision Time Protocol (IEEE1588/PTPv2), form a family of

methods enabling to deliver frequency synchronization over a packet based network with an

asynchronous physical layer. However, as presented in Chapter 5, their end-to-end performance is

greatly affected by the PDV introduced by the packet network. It is therefore crucial for an operator to

have access to engineering rules permitting to deploy such methods in a safe way, in order to ensure

that the relevant synchronization quality will be delivered to the network equipments which need such

reference.

7.2.1 The need for developing engineering rules for packet based methods

In the scope of this document, packet based methods refers to mechanisms enabling to

transport frequency synchronization via packets over a network which handles these packets as data

traffic and does not offer any specific timing hardware support in the intermediate transport nodes.



Those timing packets may contain timestamps, as it is for instance the case when a protocol like PTP

is used. Other types of synchronization, like phase/time, are not discussed in the scope of this section,

although some observations it contains might also be relevant in this context.

Deploying this kind of solution is not trivial and requires careful analysis of the network: the reception

of the timing packets is indeed not enough to ensure that the timing information will be recovered

with the appropriate quality.

The key parameter impacting the performance of packet based methods when there is no timing

support from the network nodes is the PDV produced by the network. Indeed, the timing packets are

not all experiencing exactly the same transit delay when transported over a network; these variations

of delays, for instance due to packet buffering, can lead to degrade the timing quality recovered by a

packet slave clock.

The reasons for the difficulties to develop engineering rules for packet based methods

deployment are mainly due to two aspects: the first one is related to the lack of knowledge on the ways

to control the PDV produced by a telecom network, and the second one is related to the fact that the

implementations of packet slave clocks are using proprietary PDV filtering algorithms.

As a consequence, engineering rules for the deployment of packet based methods are not

currently defined. Their development is related to the characterization and the control of the PDV, a

subject which is still heavily debated in standardization organisms. In particular, the maximum

number of network nodes between a packet master clock and a packet slave clock is difficult to

determine, as it depends on many parameters.

Therefore, it is essential to study the typical PDV that is seen over real telecom networks, but

also to study the mechanisms enabling to control and limit this PDV. A better understanding of the

packet slave clocks PDV tolerance and performance is also a necessary step before a safe development

of packet based methods, such as IEEE1588/PTPv2.

In addition, a methodology for building a network which meets the PDV tolerance of the

filtering algorithms implemented in the packet slave clocks is required in order to ease operational

aspects related to these technologies for the operators.

The next sub-Clauses will present some studies related to PDV aspects, with this objective of

progressing the engineering rules for deploying packet based methods. In particular, a network PDV

accumulation model will be introduced.

7.2.2 Analysis of PDV generated by telecom networks

PDV generated by telecom networks is a complex parameter to characterize. Indeed, each

network equipment is different in terms of PDV generation, and the parameters having impacts on

PDV generation are multiple and complex to control. Moreover, how the PDV accumulates over the

network still need valid models.

References [16][17][18][20][24] contain examples of PDV measurements performed over

different telecom network equipments. These results include measurements over platforms of

different technologies (fiber, micro-wave...) and different manufacturers. The main observations from

these documents will be summarized hereafter.

The first important observation from those measurements is that two equipments of the same

technology, even from the same manufacturer, can generate very different PDV. The consequence is

that testing on a case-by-case basis is necessary in order to have a precise knowledge of the PDV noise

produced by a network.

Among the parameters which may have impacts on PDV, the following can be listed:

1. Type of medium (e.g. micro-wave, copper, fiber link)



2. Hardware architecture of the equipment (small routers for instance do not necessarily implement the same hardware architecture as big core routers)

3. Features activated in the equipment (in general the more features activated, the higher the risk of PDV generation, because of shared resources)

4. Speed of the links carrying the timing packets 5. Transport technology (e.g. Optical Transport Network (OTN), Digital Subscriber Line (DSL),

packet network, …) 6. Characteristics of the background traffic load (e.g. level of load, size of the packets, …) 7. Prioritization of timing flows and background data traffic flows

A general idea often mentioned is that prioritizing the timing packets enables to solve the

problem of PDV generation. However, even if the way the packet timing flow is prioritized in the

network equipments has a strong impact on the PDV, it does not necessarily always fully solve the

problem.

Indeed, Quality of Service and Class of Service (QoS/CoS) mechanisms are generally designed

for improving the packet jitter of real time applications (such as VoIP for instance). The packet jitter

requirements of such applications are totally different from the PDV requirements of "timing

applications" (PDV in the order of µs can have an impact for timing applications, while only

amplitudes of PDV in the order of ms are generally of importance for real time applications).

Therefore, these mechanisms are not fit for purpose for timing flows.

For instance, QoS/CoS mechanisms may in some case improve the overall PDV amplitude, but

not necessarily stabilize the population of timing around a fixed delay window. As it will be discussed

further in section 7.2.3, the statistical distribution of the delays of the timing packets is in general a

more important parameter compared to the amplitude of the PDV.

One aspect that is generally observed however when assigning the highest (strict) priority to the

timing packets is that it helps improving the population of packets close to the minimum delay transit

of the network (also called floor delay in this document). It is an important observation which will be

reused in the PDV accumulation process presented later. However, certain technologies produce PDV

independently of the traffic load (e.g. DSL, micro-wave), and prioritization does not really help

increasing the population of packets close to the floor delay.

The following general observations have been made when measuring PDV over different types

of Ethernet platforms using fiber connections:

1. Delay steps, corresponding to an increase or a decrease of the minimum delay transit of the network, can be generated by some equipment, especially in case of congestion or high level of load, but not only (some equipments are generating delay steps even with reasonably low level of load).



Figure 7 - Load pattern and resulting delay steps generated during congestion periods (figures

extracted from [16])

0

10

20

30

40

50

60

70

80

90

100

0 2 4 6 8 10 12 14 16 18 20 22Time (h)

% trafic load



Figure 8 - Delay steps generated with reasonably low levels of load (figure extracted from [20] and

[24])

2. Intermediate priority assigned to the timing packets does not help in general to improve the PDV, and can sometimes degrade it compared to transporting the timing packet as best effort traffic. Only the highest (strict) priority significantly helps in general.



Figure 9 - Comparison of PDV histograms with lowest priority (first plot - peak-to-peak = 302µs), intermediate priority (second plot -peak-to-peak = 161µs), and highest priority (last plot - peak-to-

peak = 43µs) for the timing packets in case of congestion (figures extracted from [16])

3. Packet size of the background data traffic has an important impact on PDV. Large packets are generally creating more PDV, since they are longer to be transmitted over the physical link.



Figure 10 - PDV measurement when changing the size of the background traffic (figure extracted

from [16])

4. Core equipment generates very different PDV compared to access equipment. Different

hardware architectures of the equipments may explain this observation.



Figure 11 - Comparison of PDV histograms generated by a core equipment (first plot - peak-to-peak = 302µs) and by an access equipment (second plot - peak-to-peak = 1.55ms) when timing

packets are not prioritized in case of congestion (figures extracted from [16])

5. Similar types of equipment of the same technology from different manufacturers lead to different PDV generation. Again, different hardware architectures of the equipments may explain this observation.

Other observations can be made when measuring PDV over other types of transport

technologies. For instance, OTN transport does not produce important PDV for frequency

distribution. Micro-wave systems however can create some problems (see reference [18]): in

particular, strong impacts have been observed when adaptive radio modulation feature is used,

enabling to change the radio modulation according to the weather conditions (in general, delay steps

are observed when the radio modulation changes).

7.2.3 Model of a packet slave clock

The required timing quality to be delivered by a packet slave clock depends on the end

application connected to the slave. In some cases, a packet slave clock needs to control the phase error

accumulation so that Maximum Time Interval Error (MTIE) and Time Deviation (TDEV) masks

would not be exceeded. In other cases, like for instance the case of a packet slave clock integrated in a

mobile base station, only the frequency error needs to be bounded, not the phase error accumulation.

A well-known example corresponding to this latter case is the 50ppb accuracy requirement for

the air interface of mobile base stations, which can be considered as a relaxed requirement compared

to meeting MTIE and TDEV masks. Moreover, a good oscillator needs anyway to be implemented in

the base stations, in order to meet the short term stability requirement for the radio interface.

Therefore, in the mobile network context, the integration of the packet slave clock in the base station

permits to relax the challenge related to performance. This work will therefore mainly focus on the

50ppb requirement, corresponding to this case.

As explained earlier, in order to recover the timing information, a packet slave clock receives

timing packets. Not all the timing packets received by the slave are used in general for timing



recovery, since the overall PDV noise would be too important for the packet slave clock, compared to

the target performance.

Instead, in almost all the packet slave clocks, some packet pre-selection is done. This is for

instance the way a packet slave clock is modelled in Figure 12, extracted from draft ITU-T

Recommendation G.8263 under discussion in ITU-T Q13/15. In this figure, a Packet Selection block is

shown before the Phase Lock Loop (PLL). Therefore, only a subset of the timing packets is applied to

the PLL. In general, the packets are selected according to their transit delay.

Low Pass filter Oscillator

Local Time

scale

Output ClockPacket Timing

SignalPacket

Selection

Local reference

Time Scale

Comparator

PEC-S

Figure 12 - Figure A.1/G.8263 draft - Function model of a PEC-S packet-based clock (source of the data: International Telecommunication Union, ITU-T Working Party 3/15 Temporary

Document 318, G.8263 latest draft, 16 May 2010)

Deriving general rules regarding the behaviour of a packet slave clock is a difficult task, because

of the proprietary nature of PDV filtering algorithms. Some assumptions are however necessary in

order to enable deriving suitable PDV criteria to be met by a network: when testing different packet

slave clock implementations, some common observations can generally be made.

For instance, a consequence of the packet pre-selection in a packet slave clock is that the

amplitude of PDV is in general not an important parameter. Instead of using all the timing packets, a

delay window where a sufficient population of packets is located may be considered by the packet

slave clocks for timing recovery. Only the packets located in this delay window may for instance be

selected and used for timing recovery by the slave in some cases.

This delay window may be located at the floor delay (i.e. the fastest packets are selected), or at

another delay window (for instance, in case of poor population of packets close to the floor delay). A

pre-selection based on the floor delay criteria is implemented in most of the packet slave clocks (some

slaves implement other criteria in addition).

Therefore, being able to build a network enabling to always offer for any condition (e.g.

independently of the level of load) a certain population of packets in a given fixed delay window is

expected to strongly help the performance of the packet slave clocks. It is the purpose of the

methodology presented in the next section.

7.2.4 PDV accumulation study

This section will introduce a network PDV accumulation model, based on statistical analysis.



7.2.4.1 Objective and methodology

This section presents the objectives of the network PDV accumulation model. The main idea is

to define a methodology for ensuring that a network will always offer a minimum population of timing

packets located in a floor delay window of a given size, whatever are the conditions of the network

(e.g. for any load condition). More details about this methodology can be found in the references

[19][21][22].

Figure 13 shows a 60µs floor delay window; in this case, the objective is that enough timing

packets would be located in this delay window starting from the delay dmin of the fastest packet to

dmin+60µs.

Figure 13 - Illustration of a 60µs floor delay window

The key aspect leading to choose the floor delay window as the relevant window in the model

discussed here comes from the observations made when testing Ethernet platforms: prioritizing the

packet timing flow (e.g. PTPv2 flow) generally enables to reduce the amplitude of the delay steps

which occur when the network load increases, and therefore to optimize the concentration of packets

close to the floor delay during load variations.

Other models might be imagined with delay windows not located at the floor, for instance in

order to cover the technologies which natively do not experience a good population of packets at the

floor, independently of the traffic load (e.g. DSL links).

The key idea of this model is illustrated in Figure 14: at the end of a reference chain, enough

population of timing packets must remain within the floor delay window (e.g. 5% of the packets in the

100µs floor delay window) independently of the traffic load applied on the network. Each node

removes a certain percentage of the packets from this floor delay window, because of the PDV it

produces.

60µs floor

delay

window



Figure 14 - Network PDV accumulation model

The values provided in Figure 14 are taken as examples; it is possible to use other values than

100µs for the floor delay window size, and than the 5% objective. The appropriate values to be used

depends on the specific packet slave clock implementation, for instance on the quality of the oscillator.

In order to determine how the packets are removed from the floor delay window when

cascading nodes, a statistical analysis of the PDV produced by each node can be performed, using

convolution operations in order to anticipate the remaining population of packets in a given floor

delay window. This will be further explained in section 5.2.

This approach targets packet slave clocks embedded in base stations, where only 50ppb

accuracy is targeted. For packet slave clocks targeting bounded phase accumulation error, some more

complex approach could be needed.

7.2.4.2 Experimentation using statistical analysis and convolution operator

In order to study how the population of timing packets decreases from a floor delay window

when cascading nodes, a statistical analysis is used here, based on real PDV generation

measurements. All the details of this experimentation can be found in the reference [24].

Packet master clock

(e.g. PTPv2 master)

Packet slave clock

(e.g. PTPv2 slave)

Percentage of timing packets in the 100µs

floor delay window (worst case)

100%

N1%

N2%

N3%

N4%

5%

objective



The approach presented here consists in measuring the PDV produced by an equipment during

stress conditions (for instance, when loading the equipment), and to use these PDV measurements to

anticipate the PDV produced by several equipments of the same type when they are cascaded. Mixing

different types of equipments should also be possible with the same approach.

Figure 15 shows an example of the PDV produced by a single network equipment on PTP timing

packets when data traffic load of variable packet size is applied on the same output port as for the PTP

packets. The first diagram shows how the PDV generated by the network equipment is evolving when

increasing the traffic load by steps. The second diagram shows the histograms corresponding to each

step of load considered independently.



1

10

100

1000

10000

100000

0.000E+00 1.000E-05 2.000E-05 3.000E-05 4.000E-05

Histogram10% 20% 30% 40% 50% 60% 70% 80% 90% 91%

92% 93% 94% 95% 96% 97% 98% 99% 100%

Figure 15 - PDV of PTP Sync messages and corresponding histograms

Each histogram is calculated during a stationary condition (stationary load) and represents

statistically the delay that can be experienced by a timing packet when going through the node loaded

at a given level. Figure 15 shows that when increasing the traffic load, a timing packet has higher

probability to experience higher delay.

The histograms measured over one node can then be used to determine what should be the

histograms after several nodes. Indeed, assuming that the PDV noise produced by a node is not

correlated to the noise produced by the next node (e.g. in case they are loaded with uncorrelated

traffic), it is possible to use convolution operations to anticipate the histograms produced after several

nodes. Convolution can be seen as the "addition" of histograms.

Experimentation has been performed in order to verify that this theoretical assumption is

correct. In this experimentation, two similar nodes (identical to the one used for the PDV shown in

Figure 10) have been cascaded and loaded similarly at the same instant, as illustrated in Figure 16.

The PDV is measured on PTP timing packets directly at the output of the second node.



Figure 16 - Configuration with 2 nodes used during the experimentation

This configuration aims at measuring how the PDV accumulates when 2 nodes are loaded at the

same level at the same time, in case of a "hop-by-hop" data traffic (each data traffic flow T1 and T2

exists the network at the next node it enters).

The histograms of the PDV measured with this configuration have been analyzed and compared

with the theoretical results obtained when convolving with itself each initial histogram obtained with

a single node (e.g. histograms of Figure 15).

Figure 17 provides an example of this comparison in the case when the two nodes are loaded at

80%, with background data traffic of variable packet size. The first diagram compares directly the

histograms: the red curve corresponds to the initial histogram measured over a single node, the blue

curve corresponds to the convolution of the red histogram with itself, and the orange curve

corresponds to the histogram measured after having cascaded two nodes. The second diagram

compares the cumulative distribution functions (CDF) of the histograms. It shows how the population

of timing packets is evolving when increasing the size of the floor delay window. It can be observed

that the theoretical convolution calculation matches quite well the real measurement with 2 nodes in

this case.

0.000E+00 1.000E-05 2.000E-05 3.000E-05 4.000E-05 5.000E-05

Relative delay (s)

Packet Number

initial hist 1 node hist conv 1 node hist 2 nodes



0%

20%

40%

60%

80%

100%

0.00E+00 5.00E-06 1.00E-05 1.50E-05 2.00E-05 2.50E-05 3.00E-05 3.50E-05 4.00E-05

Delay Window Observation (s)

Pe

rce

nta

ge initial CDF 1 node

CDF conv 1 node

CDF 2 nodes

Figure 17 - Histograms and Cumulative Distribution Functions comparison at 80% of traffic load

Based on this experimentation, the main following points can be summarized:

- Convolution operations appear in most of the cases predicting correctly the PDV generated by 2 nodes in the configuration used (two uncorrelated data traffic with a "hop-by-hop" scheme), and seems therefore a priori a useful tool in this case. It takes into account the effect of delay steps accumulation

- In general, the predictions based on convolution operation are slightly optimistic. This

behaviour might be due to the fact that not all the components of PDV noise have been taken into account in the convolved histograms. In some other (rare) cases, the prediction with the convolution is pessimistic. An appropriate worst case histogram reflecting how the main sources of PDV in the network equipment can accumulate needs therefore to be considered.

- Identification of all the sources of PDV noise in the network equipments is critical in order to properly anticipate the PDV produced by a network. Not only the effects of the output ports need to be considered.

- The way several data traffic flows are generated seems critical when measuring the effect of cascading nodes; in case the two data traffic flows T1 and T2 applied are correlated, then the convolution operator does not provide correct estimation of the accumulated PDV.

- It is expected that all these observations can be extended to more than 2 nodes being cascaded, and to the case of nodes with different levels of load at a given instant. This has however not been confirmed at the moment with real experimentations.

- Additional testing needs to be performed to study how the PDV accumulates in the case of an "end-to-end" data traffic (i.e. the data traffic flows do not exist the network at the next node they entered). It is likely that in this case, other phenomena may be observed (e.g. delay steps), and that other tools in addition to the convolution operation might be needed to estimate how PDV accumulates.



7.2.4.3 Theoretical maximum number of nodes which can be cascaded

Based on the observations of section 7.2.4.2, this section will present a theoretical calculation of

the maximum number of nodes which can be cascaded in case of uncorrelated hop-by-hop data traffic.

In order to compute this calculation, a given criteria in terms of population of packet in a floor delay

window has to be chosen. In addition, a theoretical worst case of PDV histogram needs to be identified

for different levels of load, reflecting the worst case of depopulation of packets in the floor delay

window. This latter point is not a trivial task, and some assumptions may have to be made, for

instance based on the PDV generated by the equipment of interest during different conditions. An

initial proposal of PDV testing procedure can be found in the reference [23].

In the computations presented in this section, a minimum of 5% of timing packets to be

maintained within a 100µs floor delay window will be used as the relevant criteria to be met. A

theoretical worst case of PDV histogram has been determined for 3 levels of load: 50%, 80%, and

100% (congestion). These histograms are then convolved together as needed in order to determine the

theoretical maximum number of nodes which can be cascaded, while meeting the previous criteria.

Figure 18 shows how PDV histograms and corresponding CDFs are evolving when convolving

the theoretical worst case histogram for 50% of load. It also shows how the population of timing

packets in a 100µs floor delay window is evolving.

0.00E+00 5.00E-05 1.00E-04 1.50E-04 2.00E-04 2.50E-04 3.00E-04

Relative delay (s)

Pa

cke

t N

um

be

r

Initial 1 Conv 2 Conv 3 Conv 4 Conv 5 Conv 6 Conv 7 Conv 8 Conv

9 Conv 10 Conv 11 Conv 12 Conv 13 Conv 14 Conv 15 Conv 16 Conv



0%

20%

40%

60%

80%

100%

0.00E+00 2.00E-05 4.00E-05 6.00E-05 8.00E-05 1.00E-04 1.20E-04 1.40E-04 1.60E-04 1.80E-04 2.00E-04

Delay Window Observation(s)

Pe

rce

nta

ge

Initial

1 conv

2 conv

3 conv

4 conv

5 conv

6 conv

7 conv

8 conv

9 conv

10 conv

11 conv

12 conv

13 conv

14 conv

15 conv

16 conv

100µs limit

5% limit

Figure 18 - Analysis of successive convolutions

The first diagram of Figure 18 shows that when increasing the number of nodes, the histograms

become logically more and more Gaussian, and their amplitude increases. A timing packet has

therefore higher probability to experience higher delay. The second diagram of Figure 18 shows the

CDFs of each histogram. A black dotted line has been drowned at 100µs. A red dotted line shows the

5% limit. The last diagram of Figure 18 shows how is evolving the percentage of packets within a

100µs floor delay window when simulating cascaded nodes via convolution operation. Up to 7

convolutions, corresponding to 8 cascaded nodes loaded at 50%, 100% of the timing packets are

within the 100µs floor delay window. Beyond 8 nodes loaded at 50%, the percentage is decreasing.

The maximum number of nodes loaded at 50% which can be cascaded while meeting the criteria of

minimum 5% of timing packets within a 100µs floor delay window is 16 nodes, corresponding to 15

convolutions (8.06% of packets are remaining in the 100µs floor delay window, while after 17 nodes,

only 3.46% are remaining).



In other words, this model shows that when more than 16 nodes loaded at 50% or less are

cascaded, there is a risk that less than 5% of timing packets would remain in a 100µs floor delay

window, because of the traffic load variations.

Similar calculation might be done with other levels of load, for instance (provided that the

criteria to be met are the same):

- 16 nodes loaded at 50% can be cascaded (8.06% remaining in a 100µs floor delay window), as shown in Figure 18

- 10 nodes loaded at 80% can be cascaded (16.16% remaining in a 100µs floor delay window) - 4 nodes at loaded at 80% + 11 nodes loaded at 50% can be cascaded (7.71% remaining in a

100µs floor delay window) - 1 node in congestion (loaded at 100%) + 4 nodes loaded at 80% + 8 nodes loaded at 50% can

be cascaded (6.06% remaining in a 100µs floor delay window)

It is important to remind that those theoretical estimations are only valid for the specific

equipment which has been tested, and for which worst case histograms have been derived. Other

equipments might produce more or less PDV, and therefore the maximum number of nodes that can

be cascaded can be different. Note that in order to simplify this point, several categories of

equipments in terms of PDV generation levels might need to be defined. Anyway, this methodology,

which needs to further analysed and completed, provides an initial model for building engineering

rules for the deployment of packet based methods.

7.2.5 Conclusion (for clause 7.2)

Clause 7.2 has presented the context and the challenges associated to packet based methods,

such as PTPv2, for frequency distribution. It has also discussed general observations which can be

made with regards to the PDV generated by telecom networks, as well as how a packet slave clock can

be modelled.

A network PDV accumulation model has been explored based on statistical analysis, with the

objective of progressing the engineering rules for deploying packet based methods. Some initial

experimentation seems to confirm that this model is valid in the case of hop-by-hop uncorrelated data

traffic. However, further analysis and models might be required in other contexts, for instance in the

case of end-to-end data traffic.

Although controlling the PDV produced by a network is a difficult task, these results show that it

might be possible to do so if the purpose is to ensure that a minimum population of packets is located

in a given fixed delay window.

Testing on a case-by-case basis of the PDV produced by the network equipments is however a

prerequisite, as it has been observed that different equipments generate different levels of PDV. It is

likely that this testing would be a task for the equipment manufacturers, who have the knowledge of

the hardware architecture of their equipments. Defining several categories of PDV generation levels

might however help simplifying this aspect.

Clarification of the input tolerance of a packet slave clock in terms of PDV is also a necessary

step, in order to specify clearly which criteria the network has to meet. It is clear that this input

tolerance specification implies that all the packet slave clocks would share at least a minimum

common criteria (e.g. utilization of the packets located at the floor delay, if the population is

sufficient), in addition to other criteria which might be implemented in smarter slaves.



8 Time and Phase Distribution over Packet Networks

with Full or Partial Network Support

8.1 Summary of work accomplished in M24

Chapter 7 in [27] addresses time (and frequency) distribution in networks having full or partial

on-path-support.

The discussion in Chapter 7 begins with an overview of two of the most common representatives

for PTPv2 on-path support, namely, the Transparent Clocks (TCs) and the Boundary Clocks (BCs).

The differences in-terms of the distribution topology as well as the expected E2E performance are

discussed. Following, deployment considerations for the cases of full and partial network support are

presented. For the partial network support case, a discussion regarding the slave clock dominant error

contributors is given focusing on the packet filtering capabilities and the impact of the local oscillator.

Next a comprehensive comparison between BCs and TCs is given.

In clause 7.2 another form of on-path support is being discussed. An inter-work between a

PTPv2 time distribution scheme and an underlying Sync-E frequency distribution scheme is

presented. It is argued that awareness at the high-layer PTPv2 distribution node to the underlying

Sync-E physical layer frequency distribution, can be nicely exploited to improve overall time

distribution performance. Specifically it is shown that a collaboration scheme between the Sync-E and

PTPv2 management entities could be used to find the best network routes for the PTPv2 protocol that

will have the most interactions with Sync-E capable nodes (see Figure 19 below). The high quality

frequency reference, supplied by the Sync-E in such nodes, could be then used by the PTPv2 on-path

support function (e.g. TC or BC) to improve its performance, contributing to the E2E time

distribution. Sync-E failure events and thus a change in the Sync-E distribution topology, in such a

case, could trigger a PTPv2 distribution route change in order to re-adapt to the new Sync-E

distribution topology.

••

•1588V2 slave

••

••

••

••

•Grandmaster 1588V2 •SyncE

•PRC

•SEC

•G.811

•G.811

•G.811

•SSM

•��

•��

•��

•��•��

•��

•��

•��

•G.811

•Transparent nodes

•Syntonized

•Boundary clock

•Other 1588V2

•slave

•1588V2 BC

•Non •-•syntonized

•Boundary clock

Relevant Synchronization

path

•��



Figure 19 - Automatic computation of the PTPV2 topology thanks to an improved BMC taking into

account a local Sync-E support

Last, on clause 7.4, presents a new approach to the collaboration between two ToP schemes: the

PTPv2 and the NTP. The cooperation suggested is driven by the fact that although a solution based on

PTPv2, that is the newer time distribution scheme, usually comprises important features like HW

timestamping and on-path support (e.g. TCs) that a typical NTP solution usually lacks, NTP is much

more widely spread and has been massively deployed over the past 30 years to facilitate a very wide

range of applications. Thus, if the old NTP scheme could somehow benefit from the goodies offered by

the new PTPv2, major performance enhancement could be achieved.

The novel idea is to use the 1588V2 signaling for carrying the NTP one. This latter could be

performed by taking advantage of the flexible 1588V2 structure through a Type Length Value (TLV)

field and by ensuring a protocol inter-working at the intersection of time domains, as depicted in

Figure 20.

The proposed solution is based on 2 complementary building blocks:

- The first one relies on NTP ADD/DROP elements bordering 1588V2 transparent clocks domains

guaranteeing the coherence and continuity between NTP and 1588V2 (particularly for timestamps

and correction fields).

- The second building block is an auto-discovery/advertising mechanism dedicated to inform a

management or a control plane of the position of these ADD/DROP elements to ease the (dynamic)

establishment of the best paths between a NTP server and a NTP client.

NTP ADD/DROP element clocks have local requirements for the correction of (queuing) times

and thus should be far cheaper than ’gateway clocks’ (which stability has to be ensured during the

two-way signaling). This enables a dense deployment of these formers, yielding to a high flexibility in

terms of deployment (especially for sharing expensive NTP servers and 1588V2 masters), providing an

optimal allocation of all synchronization resources.

Figure 20 – NTP ADD/DROP concept



8.2 LTE IPSec challenging environment

8.2.1 Introduction

As discussed in Chapter 5, the IEEE Standard 1588TM-2008 (called hereafter as IEEE 1588V2)

suffers from Packet Delay Variation (PDV) and packet Delay Asymmetry (DA) induced by the Packet

Switched Network (PSN).

In order to maintain both of the aforementioned adverse effects within controlled boundaries

and to reach high performance in term of synchronization accuracy, the IEEE 1588V2 standard has

specified Boundary Clock (BC) and Transparent Clock (TC) concepts. Those two mechanisms are

called as On-Path Support (OPS) as referred to the terminology already defined in [27].

The respective strengths and weaknesses of BCs and TCs have also been discussed with [27].

The conclusion drawn from this discussion is that they have complementary roles and that they are

welcome for addressing network dependencies within a high-performance IEEE 1588V2 deployment.

Nevertheless, there are still several scenarios where their deployment becomes very complex or

even not applicable (e.g. TC in the middle of an IPSec tunnel). This Chapter illustrates such scenarios,

via an important study case represented by the LTE standard. Addressing security concerns, this latter

recommends the use of IPSec tunneling which strongly complicates the deployment of any BC and/or

TC.

In such a context, this Chapter proposes two innovative OPSs which better suit ciphering

environments. The first one addresses the Delay Asymmetry (DA) concern whereas the second one

addresses PDV effects. Both solutions present an advantage of being protocol-agnostic, meaning that

they could be reused to improve the performance of other packet time distribution protocols including

the well established NTP (Network Time Protocol).

8.2.2 LTE IPSec environment description

The LTE standard gives way to a full deployment of PSN for its backhaul, by adopting a flat IP

transport architecture instead of the legacy ATM-over-TDM one. Also, the use of IP as the transport

protocol potentially allows LTE traffics to share the same backhaul infrastructure with other

applications. This opens the way to an unified transport architecture.

However, this raises new concerns in term of security. The LTE standard [4][5][6] particularly

recommends to deploy ciphering and authentication means in order to protect both LTE signaling

plane and user plane traffics.

Figure 21 illustrates such a configuration where IPSec tunnels are established respectively

between the LTE Core elements and the evolved NodeBs (eNBs) and between different eNBs in order

to ensure the integrity (i.e. authentication) and the confidentiality (i.e. ciphering) of all encapsulated

packet flows. The LTE standard [4] especially recommends the use of IPSec ESP (Encapsulating

Security Payload) tunnelling mode as defined by the RFC 4303, with strong ciphering algorithm as

defined by the RFC 3602. This particularly applies when eNBs belong to untrusted locations/domains.



Figure 21 - IPSec tunnel deployment in LTE

8.2.3 Traditional OPS deployment discussion

Figure 22 illustrates the IPSec ESP tunnel wherein the original (or encapsulated) IP packets are

entirely ciphered. In the case where IEEE 1588V2 messages are transported within these latter, it

becomes impossible for a BC or a TC to operate (e.g. modifying the Correction Field in the case of a

TC) in the middle of such an IPSec tunnel.

The deployment of BCs and TCs within the LTE secured environment is however still possible.

Figure 23 represents an example of such a possibility where a parallel IPSec tunnel architecture is

built to transport IEEE 1588V2 flows, thanks to a distributed tunneling scheme.

Figure 22 - IPSec ESP tunnel mode



Figure 23 - BCs deployment scenario within a LTE IPSec environment

Nevertheless, this dedicated security architecture adds complexity to the network (e.g. new

functionalities should be added onto intermediate network nodes in order to start or to terminate the

IPSec tunnels). It also implies additional management burdens (e.g. an additional certificate

hierarchy) and finally additional costs (e.g. higher OPEX).

To keep the transport infrastructure as simple as possible without sacrifying the traffic security

requirements, the natural way would be to transport IEEE 1588V2 packets using the same security

architecture as for LTE traffics. And, in this specific case, there should be new alternative OPSs with

regards to BCs and TCs concerns.

In the following Clauses of this Chapter, new agnostic OPS solutions will be presented. These

latter particularly rely on the use of a dedicated DSCP value transported within the IPSec tunnel IP

header in order to segregate IEEE 15588V2 traffics from LTE traffics. Indeed, the DSCP field within

the IPSec tunnel header is apparent to (i.e. can be read by) network nodes positioned in the middle of

the IPSec tunnel.

8.3 New protocol-agnostic mechanism to monitor the DA

8.3.1 Principle description with a first proposed implementation

Within the context of time distribution, delay asymmetry is a vector of synchronization

inaccuracy. Typically the achievable time accuracy is one-half of the delay asymmetry.

The new proposed DA monitoring mechanism, as described in this chapter, aims at minimizing

delay asymmetry induced by each network node. As such, this is an alternative to the measurement of

IEEE 1588V2 packet resident time performed by TCs, with the advantage of being protocol-agnostic

and capable to operate within an IPSec tunneling environment.

The principle of this solution consists in managing timestamp packets at the network node level,

so that Master-to-Slave packets experience similar queuing and switching effects as packets in the

opposite direction (i.e. Slave-to-Master). This solution particularly addresses the low frequency

variation of the delay asymmetry which is typically hard to filter by PTPV2 Slave algorithms [7].

A first way to implement the new concept is illustrated by Figure 24a and Figure 24b. Figure

24a illustrates the proposed path of a timing packet traversing the network node in the Master-to-



Slave direction. Figure 24b illustrates the proposed path of a timing packet traversing the network

node in the Slave-to-Master direction.

Figure 24 - First way to implement the new DA monitoring mechanism

As depicted in Figure 24a, an incoming timing packet (from the Master) experiences the

following sequence of queuing and switching events (the assumption here is that the input and output

ports are located in different interface cards; otherwise, switching operations would not be required):

P1 → P2 → P3 → P4 → P5 → P6 → P7

Note that paths P2 and P6 do not represent a path external to the switching matrix. Instead, they

represent a path through the switching matrix that results in the timing packet being deposited at the

input of the respective buffers.

It is important to stress here the new behavior of queuing buffers with regards to the traditional one:

- Within a common packet handling behavior, the queuing buffers are high speed memory

space dedicated to packet storage located upon a particular port card and once a packet has

been switched to a respective port card, it is at the suitable location to be placed in the

indicated buffer without further transitioning of the network node switching matrix.

- Within the new proposal, it particularly requires buffers presenting multiple inputs and/or

outputs (regarding their connectivity with other buffers). Moreover, this feature is only

applied to a selected set of packets, especially timing packets (e.g. IEEE 1588V2) identified



thanks to the information carried by the IPSec tunnel header (DSCP parameter) as previously

discussed.

As depicted, the transition delays related to a timing packet traversing the network node in the

Master-to-Slave direction is the sum of all elementary event delays:

dM→S = d(P1) + d(P2) + d(P3) + d(P4) + d(P5) + d(P6) + d(P7)

Similarly, Figure 24b depicts the path of a timing packet coming from the Slave side of the network.

The path of the timing packet traversing the network node through input/output buffers and

switching matrix is as follows:

P’1 → P’2 → P’3 → P’4 → P’5 → P’6 → P’7

The related network node transition delay of a timing packet traversing the network node in the Slave-

to-Master direction is thus:

dS→M = d(P’1)+d(P’2)+d(P’3)+d(P’4)+d(P’5)+d(P’6)+d(P’7)

As the result, the delay asymmetry produced by the network node is given by:

DA = dM→S - dS→M

DA = (d(P1)+d(P2)+d(P3)+d(P4)+d(P5)+d(P6)+d(P7))

- ((d(P’1)+d(P’2)+d(P’3)+d(P’4)+d(P’5)+d(P’6)+d(P’7))

Referring to Figure 24a and Figure 24b, transition delays d(P2) and d(P6) are respectively

similar to d(P’2) and d(P’6) as they are related to similar queuing events. In the same way, d(P4) and

d(P’4) refer to similar switching events.

Limiting the discussion to the low frequency variation component of the DA over time <DA>

(e.g. the result of DA after a low-pass filter or an averaging), the above transition delays have the

resulting effect which can be considered as a small constant value in the expression of <DA>

(asymmetry related to the structural hardware design of the network node). Therefore, the above

expression may be simplified by replacing the transition delays associated with traversing the

switching matrix and queuing events. This results in:

<DA> = ε+ (<d(P1)>+<d(P3) >+<d(P5) >+<d(P7) >) - (<(d(P’1) >+<d(P’3) >+<d(P’5) >+<d(P’7) >),

with ε being a constant value.

By further inspection of Figure 24a and Figure 24b it may be noted that certain of the respective

elementary delays refer to the transition delay within the same buffer, namely: P1 and P’3, P3 and P’1,

P5 and P’7, and, P7 and P’5. Even though d(P1) are different from d(P’3) with regards to their

respective instantaneous values, their average value (i.e. low frequency component) over a long

observation time should be the same as it consists of the same buffer characteristic. Thus, the low-

frequency component of transition delays for the respective queuing buffers effectively cancels out,

resulting in: <DA> ≈ ε.



The low frequency component of the delay asymmetry is thus reduced to a constant

value (over time). Such a static asymmetry can eventually be measured and manually

provisioned at the Slave level, if it happens to be significative with regards to the

targeted synchronization accuracy.

8.3.2 Alternative implementation of the Delay Asymmetry mechanism

An alternative implementation of the new DA monitoring mechanism is illustrated in Figure

25a and Figure 25b below. Similarly to the first implementation, path P4 (resp. P’4) in Figure 25a

(resp. Figure 25b) does not represent a path external to the network node switching matrix. Instead, it

represents a path through the switching matrix (i.e. input and output ports on different cards) that

results in the timing packet being deposited at the input of the respective buffer.

The main difference with regards to the first implementation is that paths P2 and P6 (resp. P’2

and P’6) do represent a path external to the switching matrix in that these paths represent the timing

packet being deposited in the appropriate buffer on the same port obviating the need for a switching

operation. This behavior could be performed by using an external module connected to all ports.

Figure 25 - Alternate implementation of the new DA monitoring mechanism

8.3.3 Conclusion

The above new protocol-agnostic OPS allows for canceling the low frequency component of the

DA which represents a strong benefit as slave algorithms are typically not able to filter such a

component (high frequency component of the DA can be filtered out by the Slave somehow thanks to,

for instance, averaging techniques).



The depicted implementations demonstrate that the selection of a specific switching sequence at

the network node level can potentially allows for providing a more manageable behavior of the DA,

and thus a better end-to-end IEEE 1588V2 time synchronization performance.

In term of realization, traffic redirections from an output port towards another port on a same

network node, represents the biggest challenge related the new concept. Indeed, this requires a

forwarding table implemented at the output port, which is not the case for available network nodes.

8.4 New protocol-agnostic mechanism to dampen PDV

8.4.1 Packet Delay Variation (PDV) root causes

Packet Delay Variation (PDV) is essentially generated by buffering events within the network

nodes (e.g. switches or routers) along the communication path between the Master and the Slave. As

shown in Figure 26 (extracted from the ITU-T G.8061 Figure I.2), the network node can cope with low

traffic load so that packets (or frame) can be forwarded/routed within without additional delays

(except the minimum “propagation” delay). As the traffic load increases, while the processing capacity

of the node is not exceeded, the instantaneous packet rate may exceed the available processing

capacity, causing the need for a buffering sequence. Finally, at some traffic load threshold, the

processing capacity of the node is exceeded causing the delay to be increased further until packets are

dropped. Our study is rather focused on the network node state before such a processing overload

situation.

Figure 26 – Variation of delay with loading, as referred to G.8061 (Figure I.2)

Aiming at minimizing the PDV induced by queuing, a commonly used heuristic is to assign a

high priority Quality of Service (QoS; e.g. Premium class as compared to Best-effort one) to timing

packets. However, even in such a context, PDV experienced by timing packet is still impacted by the

following root causes:

- Root cause 1: a QoS assignment does not prevent the system from having PDV as the

transmission of a “Premium” packet can be delayed by an early departing “Best-effort” packet

which occupies the transmission link. And, as the size of the best-effort packet is not fixed and



predictable, this yields to an undeterministic PDV for the ‘premium’ timing packets. Such an

issue is illustrated by the Figure 27 below.

Figure 27 – “Premium” packet transmission delayed by a “Best Effort” packet

- Root cause 2: in the same way, competition between different synchronization flows (within

the Premium QoS) for the same output transmission resource induces unpredictable PDV for

each of these flows. In particular, when timing packet flows are sent with very similar

frequencies, the competition between the different flows for the access of the transmission

medium generates a “beating effect” for each of them [8]. Figure 28 below illustrates such a

phenomenon.

Figure 28 - Beating effect as referred to G.8261 (Figure 20)

8.4.2 Case of single synchronization flow

In order to avoid the aforementioned root cause 1, the new PDV dampening mechanism

consists in systematically delaying each individual “Premium” timing packet (transmitted

through a network node) by a fix delay before serializing it onto the transmission medium. The

delay is set to ∆∆∆∆MS and ∆∆∆∆SM for respectively the Master-to-Slave and the Slave-to-Master

direction.

In a first implementation, ∆∆∆∆MS can take one of the following values:



- )(MTUimeserializeTMS ====∆∆∆∆ : the time to serialize a packet of MTU (Maximum

Transmission Unit) size. For example, in 1Gbps Ethernet technology, the MTU size is 1518

octets, and sMTUimeserializeTMS µµµµ12)( ========∆∆∆∆

- ))(( etSizesBestEfPackMaximeserializeTMS ====∆∆∆∆ : the time to serialize a “Best effort” packet

of maximum size.

The above values allow to make sure that a timing packet will not be further delayed (by

the serialization of a best-effort packet) when it comes the moment to serialize it onto the

transmission medium.

During the “waiting” period (∆∆∆∆MS) of the timing packet, and in the absence of another

timing packet to transmit, the network node can complete the serialization of “Best effort”

packets. The network node should make sure about the size of these “Best effort” packets so

that their serialization onto the transmission medium will not further delay the next waiting

timing packet. This strategy can lead to wasted bandwidth when packet fragmentation is not

possible (e.g. IPSec tunnel case). However, this wasted bandwidth is minimal as the timing

packet bandwidth is typically small with regards to the overall consumed bandwidth. Indeed,

as reported by [9] for IEEE 1588V2 Telecom profile typical message rates, a Slave generates at

worst 0.3 Mbps for an FE link of 100 Mbps. As the monitoring point comes closer to the

Master, this consumed bandwidth increases with the aggregation of IEEE 1588V2 flows for

multiple slaves. Assuming that a Master is deployed at each LTE core element to support an

average of 1000 of LTE eNBs and each eNB represents a Slave clock, then the total IEEE

1588V2 consumed bandwidth is at worst 300 Mbps. As the core links will most likely have 10s

of Gbps bandwidth capacity, the IEEE 1588V2 traffic still remains a very small portion of the

overall bandwidth.

In a second implementation, ∆∆∆∆MS can take one of the following values in order to

optimize transmission bandwidth especially when (“Best effort”) packet fragmentation cannot

be activated:

- )%( MTUximeserializeTMS ××××====∆∆∆∆ with 100<<<<x

- ))(%( etSizesBestEfPackMaxximeserializeTMS ××××====∆∆∆∆

with 100<<<<x

In this case, there is not 100% guarantee that the waiting timing packet will not be further delayed.

Indeed, the serialization of an MTU-size “Best effort” packet during the waiting time ∆∆∆∆MS raises an

issue related to a conflicting usage of the transmission medium when the waiting timer expires.

However, the x

value can be chosen to statistically minimize such conflicting situations. In any case,

the PDV magnitude is strongly reduced in proportion to x−−−−1 value.

8.4.3 Case of multiple concurrent synchronization flows

In order to avoid root cause 2, as raised above, within the context where multiple Master

clocks transmit timing messages periodically with the same frequency mf , the new PDV

dampening mechanism consists in delaying systematically individual packet of different

“conflicting” (as per the aforementioned “beating effect”) timing streams. The detection of the

“beating effect” and the computation of the different delays require a learning period which

could be as small as mm fT /1==== .



The whole procedure is illustrated by Figure 29 below. The value for ∆∆∆∆MS can be for

example )(MTUimeserializeT

or a multiple of that time depending on the “conflicting”

situation and the size of different timing packets.

It is to be noted here that different timing streams can be segregated, for instance, using

their respective tunnel destination IP addresses for the Master-to-Slave direction and their

respective tunnel source IP addresses for the opposite direction.

This delaying mechanism should be performed at each network node along the communication

path. Each of these nodes requires a theoretical convergence time in order to analyze

“conflicting” situations and to compute different appropriate delays: the first node convergence

time is mm fT /1==== , the second node convergence time is

mT××××2 and the Nth node

convergence time is mTN ×××× . However, thanks to the periodical characteristic of the packet

streams, the delaying mechanism may only be required at the first node.

Figure 29 - Timing flows delayed by a multiple of the elementary delay ∆ms

The different computed delays i∆∆∆∆

applied by the network nodes to different synchronization

streams i

can be reported to the appropriate Slave clock via a centralized management system. This

strategy is possible as i∆∆∆∆

values do not vary rapidly with time which is different from the TC case

where individual residence times must be determined for each packet and then communicated to the

slave. The important difference is that i∆∆∆∆ is bound to the whole stream and not to each individual

packet within this stream.



8.4.4 Discussions on implementation and conclusion

Implementation-wise, the mechanism can be easily realized thanks, for instance, to a

simple ordered time descriptor chain. Each descriptor would point to the associated timing

packet and indicate the time when this packet should be transmitted. Figure 30 below provides

with the illustration of such an implementation. The transmitter can simply read the chain

sequentially and transmit the associated packet at the time instant indicated by the descriptor.

Figure 30 – Possible implementation of the new PDV dampening mechanism

As a conclusion, the new methodology as described in this chapter should simply allow for

dampening PDV effects by providing more consistent network transit delays. This new OPS could

particularly be useful for the distribution of a frequency reference. A performance evaluation of such a

methodology is under progress.

8.5 Conclusions and outlook (Clauses 8.2 to 8.4)

Facing stringent LTE synchronization requirements, OPSs are largely recommended for

reaching the targeted performance with PTPV2. Within this context, BC and TC concepts introduced

by the IEEE 1588V2 standards represent a straightforward solution. However, stringent security

requirements in LTE environments raise strong concerns related to their implementation and

deployment, becoming either too complex or even not applicable. In such cases, Clauses 8.2 to 8.4

propose two protocol-agnostic OPSs allowing for addressing all security constraints within the single

and fully deployed IPSec protocol. Considering an advantageous management of the IEEE 1588V2

traffic as part of the LTE application one, the two presented mechanisms particularly make use of an

apparent DSCP code. They allows for efficiently managing timing packets by performing network

node operations which aims at dampening network dependencies.



The first mechanism focuses on PDV dampening whereas the second proposes an innovative

way to minimize the low frequency component of DA, typically hard to manage by IEEE 1588V2 Slave

algorithms. These two OPSs could be activated altogether or separately according to deployment

scenarios. Thus, in a time distribution scenario, where IEEE 1588V2 is deployed with the frequency

support from Sync-E, DA is the only issue, and only the new DA dampening needs to be activated.

Suppressing the Sync-E support in the previous scenario makes PDV also a concern. Both new

mechanisms then are recommended.

In any case, synchronization is presently not a source of revenue for operators and the

advantage of the IEEE 1588V2 protocol is that it can be deployed using the existing or the common

transport infrastructure. This makes IEEE 1588V2 very attractive to operators.

Nevertheless, the introduction of BCs and TCs certainly raises the initial costs of the IEEE

1588V2 implementation within the network. Thus, there are needs to further investigate other OPSs

that can be shared amongst different protocols in order to amortize synchronization deployment

costs.

8.6 Link-to-link TOD distribution using the Sync-E’s ESMC messaging

channel

8.6.1 Context description

As more and more traffic is being delivered in Ethernet format, carriers are realizing the

advantages to converging on a pure Ethernet infrastructure, but are not able to do so with Ethernet as

presently defined. New ITU-T Recommendations, such as Y.1731 for Ethernet OAM and G.8031 for

Ethernet protection switching, provide features essential for operational maintenance of Ethernet

networks. Recently proposed mechanisms for configuring point-to-point connections in Ethernet

networks close another gap between Ethernet and SONET/SDH networks.

The final difference between conventional TDM-based networks and Ethernet is that the former

also transport frequency information, needed for some applications, while Ethernet does not. A

comprehensive solution for that problem has been achieved when the ITU-T has developed

Synchronous Ethernet that locks the timing of the Ethernet physical layer, much the same way that

SONET/SDH does. This enables simple recovery of the physical layer timing, which can be used

directly by locking downstream networks and applications, or indirectly by using the physical layer

timing as a common reference.

Nevertheless, today end applications (e.g. 3G and WiMAX cellular base-stations) are becoming

more and more dependent on the distribution of a precise TOD or phase reference in addition to

accurate and stable frequency reference. This ruling is even truer when it comes to future applications

such as LTE cellular base stations or within Single Frequency Networks (SFN) such as the Terrestrial

Digital Video Broadcasting (DVB-T) standard.

Existing packet based time distribution protocol such as IEEE 1588v2 (PTP) and the Network

Time Protocol (NTP) have been markedly improving over the recent years and have been shown to be

able to deliver the required level of performance in some network scenarios. Nevertheless, being

‘upper layers’ type of time distribution protocols (in the general case) they suffer from ‘upper layer’

type of network interferences such as Packet Delay Variation (PDV) and network re-route delay

changes. At the end of the day, these methods cannot be trusted to always perform as expected under

changing network scenarios.

A known fact is that in order to be able to assure a certain level of performance under (almost)

all network conditions the information has to be carried in the physical layer (or very close to it). This



can be accomplished either by applying precise HW timestamping very close to the physical layer to

support the upper layer synchronization distribution protocol (e.g. TCs and link-by-link BCs in the

case of PTP) or by applying the synchronization distribution directly at the physical layer (or very

close to it). An FT contribution to Q13/SG15 of the ITU-T [10] purposed exactly that: a link-by-link

phase distribution scheme based on link layer OAM messages.

The apparatus purposed in this Clause presents some modifications and improvement to the

scheme presented in [10]. Specifically, the Ethernet Synchronization Messaging Channel (ESMC),

currently used to carry Ethernet Synchronization Status Messages (ESSM) across Synchronous

Ethernet capable links, is suggested as a container to carry the time information between neighboring

nodes.

8.6.2 The ESMC

The Ethernet SSM is an ITU-T defined [11] Ethernet slow protocol that was defined in order to

distribute Quality Level (QL) information across Synchronous Ethernet capable Ethernet links.

Synchronous Ethernet SSM or ESSM distributes the same SDH/SONET 4-bits QL words. In

SDH/SONET, these 4-bits are carried in a fixed location in the frame header. In Synchronous

Ethernet, on the other hand, these 4-bit words are distributed using a dedicated unidirectional IEEE

802.3 Organization Specific Slow Protocol (OSSP) termed Ethernet Synchronization Messaging

Channel (ESMC).

In order to comply with clock selection procedures given in ITU-T Recommendation G.781 (e.g.

maximal time to generate a clock source switch over when the current selected source QL changes)

two ESMC message types have been defined. Information 1 second ‘heart beat’ messages are used to

continuously inform the downlink node of the current QL value. Additionally, in order to minimize

short-term interferences and large phase excursions during transition to holdover or clock source

switch-over, an event message is immediately send towards the downlink element to signal it on the

changed QL. To protect against possible failure, the lack of the messages is considered to be a failure

condition. The protocol behaviour is such that the SSM value is set to DNU if no SSM messages are

received after a five second period.

ESMC frames bear an ITU-T organizationally unique identifier (OUI) 00-19-A7 and a slow

protocol subtype 0x0A that distinguish it from other OSSPs. The SSM quality level is carried in a type

length value (TLV) field, which is contained within the ESMC PDU. Two types of ESMC PDU frames

are defined and are distinguished by the C flag. These are the ESMC information PDU and the ESMC

event PDU mentioned above.

The ESMC PDU format is shown in Table 2. The QL TLV is shown in Table 3. The same QL TLV

is used for both the information and event messages. To allow for potential hardware

implementations, the SSM TLV is always sent as the first TLV in the data/padding field. This means

that the QL indication always remains fixed in the PDU. Any padding must occur after the SSM TLV.

Octet number Size/bits Field

1-6 6 octets Destination Address =01-80-C2-00-00-02 (hex)

7-12 6 octets Source Address

13-14 2 octets Slow Protocol Ethertype = 88-09 (hex)

15 1 octets Slow Protocol Subtype =0A (hex)

16-18 3 octets ITU-OUI = 00-19-A7 (hex)



19-20 2 octets ITU Subtype

21

bits 7:4(note 1) Version

bit 3 Event flag

bits 2:0(note 2) Reserved

22-24 3 octets Reserved

25-1532 36-1490 octets Data and Padding (See point J)

Last 4 4 octets FCS

Note 1: Bit 7 is the most significant bit of Byte 21. Bit7 to bit 4 (bits 7:4) represent the four bit version

number for the ESMC.

Note 2: the three least significant bits (bits 2:0) are reserved.

Table 2 – ESMC PDU format (Table 11-3/G.8264)

ESMC PDUs have the following fields in the order specified above:

a) Destination address (DA): This is the IEEE-defined slow protocol multicast address. The format is defined in Annex 43B of [IEEE 802.3].

b) Source address (SA): The source address is the MAC address associated with the port through which the ESMC PDU is transmitted.

c) Slow protocol Ethertype: ESMC PDUs must be type encoded and carry the slow protocol type field value.

d) Slow protocol subtype: Assigned by the IEEE and fixed with a value of 0x0A.

e) ITU OUI: Organizational unique identifier assigned by the IEEE registration authority.

f) The ITU subtype is assigned by ITU-T. The value of 00-01 applies to all usage defined in this Recommendation.

g) Version: The four-bit field indicates the version of ITU-T OSSP frame format. This field shall contain the value 0x1 to claim compliance with version 1 of this protocol.

h) Event flag: This bit distinguishes the critical, time-sensitive behaviour of the ESMC event PDU from the ESMC Information PDU. A value of 1 indicates an event PDU, a value of 0 indicates an information PDU.

NOTE 1 – The behaviour of the event PDU is similar to the critical event defined for Ethernet OAM in clause 57

of [IEEE 802.3]. Event messages need to meet processing times defined in [ITU-T G.781].

i) Reserved for future standardization (27 bits). These fields are set to all zero at the transmitter and are

ignored by the receiver.

j) Data and padding: This field contains data and necessary padding to achieve the minimum frame size

of 64 bytes. The PDU must be an integral number of bytes (octets). Padding characters are not defined

and are ignored by receivers.

NOTE 2 – The recommended maximum size for the ESMC PDU is 128 bytes as per Annex 43B of [IEEE 802.3].

However, PDU sizes greater than 128 bytes may be permitted.

k) FCS: Four-byte frame check sequence as defined in clause 4 of [IEEE 802.3].




1 8 bits Type:0x01

2-3 16 bits Length: 0x0004

4 bits 7:4 0 (unused)

bits 3:0 SSM code

Note 1: Bit 7 of Octet 4is the most significant bit. The least

significant nibble, bit 3 to bit 0 (bits 3:0) contain the four

bit SSM code.

Table 3 – QL TLV format (Table 11-4/G.8264)

According to [11], additional TLVs comprising future extensions to the ESMC messages are

possible. Such extensions might include ‘trace-route’ like mechanisms to better mitigate timing loops.

Another possible extension to the ESMC messages is the support for phase/TOD distribution over

Ethernet links.

8.6.3 Proposals

The proposal comprises 3 major parts:

The first part of the proposal is to enhance the current ESMC messages format

with an additional TLV that will carry TOD information across an Ethernet link.

It is therefore a fact that unidirectional ESMC messages carrying QL information are distributed

across both directions of any Sync-E capable line2. Hence, Sync-E capable Ethernet networks already

possess a very bandwidth efficient communication channel between neighbouring nodes. As already

discussed in the motivation part of this Clause, extending this messaging channel with a complete

TOD distribution scheme seems, therefore, something of great value.

In order to make it happen and allow ESMC messages to convey TOD information two

additional information entities needs to be added to the current standard ESMC messages format:

(1) Timestamps (TS) that will carry the TOD information from one side of the link to the other.

As will be presented shortly, two timestamp fields will be added to each ESMC message using

the TLV format presented in Table 3 above.

(2) A Sequence number (SN) that will mark the ESMC messages according to their chronological

transmission order. As will be presented shortly, the sequence number will be used to

correlate a timestamp to the specific ESMC message it belongs to.

One possible way of incorporating these changes in the current standard ESMC message format

is given in Table 4 and Table 5 below. The changes comprise an additional 24-bit Sequence Number

introduced into currently reserved 3 octets (22 to 24) in the message header. This will be a standard

running sequence number that will reset to 0 once reaching its maximal value (223-1). The two

2 According to [11] there might be Synchronous Ethernet links with partial support. In other words, where the physical layer

clock as well as the QL distribution are being distributed in one direction only or where ESMC messages are not being

transmitted at all. Nevertheless, these cases can be regarded as exceptional and are only relevant for last mile Ethernet links.



timestamps will be introduced into a new dedicated TOD TLV that will be placed immediately after

the currently standard QL TLV (starting at octet number 29). Each of these timestamps will have the

standard PTP 80-bit Timestamp format [12] to allow for better interworking with PTP as will be later

on explained and demonstrated (alternatively these timestamps could have been designed to bear the

standard NTP format of 64-bits. Nevertheless, as PTP seems to be getting more traction from telecom

service providers, especially in the cellular backhaul arena, and a lot of discussions and work had been

taking place concerning better PTP/Sync-E interaction, it seems only natural to adopt the PTP

Timestamp format).

The first Timestamp (octets 4-13) records the precise time (according to the local node time

counter) at which the last periodic ESMC message was received in the specific node, and is used to

feedback the other end of the link with that information. Following (octets 14-16), the sequence

number of the specific received periodic message that the timestamp refers to, is also sent to allow the

receiver at the other end to correlate between transmission and reception time events on this specific

direction of the link.

The second Timestamp (octets 17-26) records the precise time (according to the local node time

counter) at which the last ESMC message was transmitted by the specific node (alternatively this

Timestamp could have been made to comprise the precise time at which the current transmitted

ESMC message left the node. However, this would impose on-the-fly HW timestamping mechanism at

each participating node. Such a mechanism might increase the overall cost of the scheme, but more

important, will unnecessarily degrade the accuracy of the timestamps by some amount).

As will be explained in the next clause, each link end (node), independently sending and

receiving ESMC messages, will have all the information it needs to recover TOD information from the

other end. This concept is very much in the spirit of standard Sync-E frequency distribution where

(except for some specific Ethernet interfaces) the frequency information can, potentially, go both

directions and correct distribution direction is determined by the overlay clock selection procedure.


1-6 6 octets Destination Address =01-80-C2-00-00-02 (hex)

7-12 6 octets Source Address

13-14 2 octets Slow Protocol Ethertype = 88-09 (hex)

15 1 octets Slow Protocol Subtype =0A (hex)

16-18 3 octets ITU-OUI = 00-19-A7 (hex)

19-20 2 octets ITU Subtype

21

bits 7:4(note 1) Version

bit 3 Event flag

bits 2:0(note 2) Reserved

22-24 3 octets 24-bit Sequence Number (SN)

25-1532 36-1490 octets Data and Padding (See point J)

Last 4 4 octets FCS

Table 4 – TOD enhanced ESMC PDU format (proposed changes are highlighted)




1 8 bits Type:0x02

2-3 16 bits Length: 0x0017

4-13 80-bit Timestamp of last received

periodic ESMC message

14-16 24-bit Sequence Number of last

received periodic ESMC message

17-26 80-bit Timestamp of last transmitted


Table 5 – TOD TLV format (proposed changes are highlighted)

It should be emphasized that the above TOD extensions need only be applied to periodic (Event

flag = ‘0’) ESMC messages with the average rate of 1 PPS. Event driven ESMC messages (Event flag =

‘1’) need not incorporate these modifications as they will not contribute anything to the TOD

distribution scheme. Hence, as far as TOD distribution is concerned, ESMC messages with Event flag

= ‘1’ should be ignored.

The second part of the proposal is to define a new apparatus exploiting the new

proposed ESMC messages TOD TLV extension to distributed TOD information across

the Ethernet link.

Lets us begin by assuming two neighboring Sync-E capable nodes X and Y that are, in the

general case, part of a Sync-E frequency distribution chain according to [11], [8] and [13]. Each of

these nodes will transmit standard QL ESMC messages to the other Naturally, depending on the

decided timing information distribution direction, one node will transmit QL=’DNU’ to the other side.

As will be discussed in the next clause, this QL information could also come very handy for the TOD

distribution process, but for now we will assume that the frequency distribution and the TOD

distribution process are completely orthogonal.

By extending the ESMC messages format as explained in the previous clause, each node will

now be transmitting the following TOD information to its neighbor:

(1) Precise local time of last received periodic ESMC message marked by: TSR(k-1).

(2) Precise local time of last transmitted periodic ESMC message marked by: TST(n-1),

where k and n are the different periodic ESMC messages indexes on both directions.

Additionally, at each node, the timestamp of the precise local time at which the last periodic

ESMC message was transmitted (TST(k-1)) as well as the timestamp of the precise local time at which

the last periodic ESMC frame was received (TSR(n-1)) are recorded and saved. A graphical

representation of the above text is given in Figure 31.

At each node, every 1 second after the completion of a successful ESMC message reception, the

exactly same following four timestamps will be available: TST(k-1), TSR(k-1), TST(n-1) and TSR(n-1).

Using these four timestamps each node could now calculate an estimation of the time error between

himself and its neighbor using the following, very known, time error estimation formula:



The calculation time error estimation could, at this point, be used to time-lock each node to the

other. The only left issue to be resolved is the correct direction (based on some correctness criteria as

will be discussed in the clause) of the TOD information flow. In other words, which node is going to

play the role of the Master and which one will be Slave, for this specific link (not setting an

unambiguous master-slave relationship for the specific link might results in a “time-loop” situation

where, in a very similar manner to timing-loops in frequency distribution, both ends will

simultaneously try to maintain lock to the opposite, resulting in unstable, unpredicted behavior).

Clearly, the above suggested apparatus introduces a 1-second delay (at least) in the time error

estimation procedure, and therefore, also in the TOD recovery process. At first glance, this fact might

be seen as problematic (1 second in the time error estimation procedure means that the time error

could be reduced up to that residual time error, stemming from the time error development over that

1-second calculation delay). Nevertheless, taking into account that the aforementioned residual time

error is negligibly small if the time counters on both nodes are timed by the Sync-E physical layer

recovered clock, this does not appear to be such a big sacrifice at the benefit of avoiding the

timestamping accuracy degradation that accompanies any on-the-fly timestamping mechanism.

Many time (clock) recovery apparatuses can be envisaged to perform the time lock procedure

based on the calculated time error estimations. This issue is currently left for further study.

Figure 31 – TOD ESMC messages exchange

The third part of the proposal is to define the TOD selection process that will be

used in each node to select the best TOD information source among possible multiple

sources. This TOD selection process is analogues to the frequency selection process

defined in [14].

Simplifying the matter a little bit, TOD distribution can be broke-down into the following

elements:

(1) A frequency distribution scheme that maintains the local time counters along the distribution

node in the correct pace.

(2) An additional occasional phase correction information that align the phase of the clocks along

the distribution chain, removing any developed bias.

In that view, the performance of a link-by-link TOD distribution scheme is very much impacted,

in the general case, from the same factors that impact the performance of a link-by-link frequency



distribution scheme. In other words, similar to the frequency distribution case, the specific free-run

quality of the local clocks along the distribution chain plays an important role. On the other hand, as

this is a link-by-link TOD distribution scheme, other attributes such as path asymmetry and path PDV

are of less importance. In other words, TOD distribution chains that comprise stable and accurate

local oscillators have better chance of delivering good TOD distribution performance.

Another reinforcement of that approach is the fact that the other capabilities of the conventional

frequency selection process such as the hold-off and wait-to-restore functions as well as the various

user configurable parameters (priority tables, force/manual switch commands etc.) seems to be also

desired functionalities for the TOD distribution case.

It therefore seems logical that the same standard clock selection process already established for

the act of frequency distribution could be largely reused for TOD distribution.

Two strategies of incorporating the fundamentals of frequency clock selection into TOD clock

selection can be envisaged:

(1) Using the exact same clock selection process for both frequency and TOD. This basically

means staying with the existing clock selection process, using it to select both frequency and

TOD. Working in such a strategy ensures that the same reference clock will always be selected

for both frequency and TOD.

(2) Using a similar but orthogonal clock selection process for TOD. The existing clock selection

process will be modified to better suit TOD source selection (for example, the specific values

of the hold-off and wait-to-restore timers might change or a different user priority table might

be used). The new TOD selection process will run in parallel to the frequency selection

process, which might result in different reference clocks selected for frequency and TOD. This

obviously means different network paths for frequency and TOD.

The first strategy is very simple and straightforward. No change (almost) to existing clock

management agents in live products will be needed. The frequency and TOD distribution will always

go together hand by hand following the same network path, a fact, that will simply overall network

management. Nevertheless, such a scheme offers little flexibility for the user to force its own TOD

distribution preferences, which might be different from his frequency distribution preferences, into

the selection process.

The second strategy, on the other hand, gives the user the freedom to design and configure its

TOD selection preferences independently from his frequency distribution preferences. Nevertheless, it

doubles the required management efforts.

8.6.4 The Simulation Environment

A complete closed simulation environment for testing the concept of TOD distribution using the

Sync-E ESMC messages was created. The simulation was created in SIMULINK of MATLAB where

participating key SIMULINK blocks where written in C. SIMULINK was selected due its nice GUI

properties as well as its inherit simulation time axis that makes it a very convenient simulation

environment to characterize time varying processes.

The simulation environment was based on the following fundamental assumptions:

(1) Either linear or ring topologies must be supported.

(2) The Ethernet physical layer’s frequency is always synchronized using the underlying standard

Sync-E frequency distribution apparatus. In other words, the amount of time error introduced by



the frequency instability of the physical-layer is very small and can be regarded as negligible in

practice. This issue is very important and will be addressed again later in this chapter.3

(3) The delay introduced by the Optic-fiber lines is completely symmetric (the same in both

directions). This assumption is known not to hold true in real fiber-optic installation where there

is always a slight difference of the length of the fiber between both directions. Nevertheless, as this

source of error is quite deterministic in nature and can only be compensated by applying a-priori

delay measurements on the fiber lines, there is no real point in simulating it4.

(4) Queuing delays at the output buffers of the participating network elements do not have any

impact on the time distribution performance. The reason for that is that no ‘on-the-fly’

timestamping in involved in the suggested scheme so the timestamping of the egress packets can

be directly taken at the output of the transmission queue just before the first byte of the packet is

being transmitted over the link. See Figure 31 for more information.

The simulation environment is comprised of a few prime functional blocks and is driven by the

interaction between these blocks. The connection between the prime blocks is defined by pre-defined

APIs so that any combination of blocks or network topology can be created and simulated for. By

extending these common APIs, the simulation can be rather easily extended, in the future, to cover

additional aspect and simulate additional effects. The following clauses will describe the prime

functional blocks and their relevant APIs.

8.6.4.1 Node block

8.6.4.1.1 Input/output ports

The node block comprises the fundamental prime functional block of the simulation (see Figure

32). It characterizes a basic 3-ports network element (a L2/L3 switch). All the ports are bidirectional.

Two of the ports, termed ‘L’ (Left) and ‘R’ (Right) are used to connect the block to its neighboring

node blocks, thus, forming the required network topology. The 3rd port termed ‘X’ (eXternal) is used

to input/output information (e.g. TOD information) from/to an external source/destination.

Figure 32 – node prime functional block inputs/outputs

3 The simulation was designed in such a way that it could be easily extended to take imperfections of

the underlying physical layer frequency distribution into account. 4 We could simply argue that the resultant introduced time error over a given link would exactly equal

half of the delay asymmetry.



The L and R ports are defined by the following API, called the ESMC signal API, and that is

comprised by the following signals:

Signal number Size [octets] Content

1 1 (char) QL � Quality Level indication

carried by the ESMC message

according to [11]

2 2 (unsigned

integer)

Sequence Number of current

transmitted periodic ESMC

message

3 4 (2 x unsigned

integer)

Timestamp of last transmitted


4 2 (unsigned

integer)

Sequence Number of last

received periodic ESMC message

5 4 (2 x unsigned

integer)

Timestamp of last received


6 double Physical-layer instantaneous

frequency (currently taken to be

constant)

Table 6 – Signal structure of the ESMC signal API

By connecting the ‘L’ port output of one node block to the ‘R’ port input of the next node block and

vice-versa, ESMC messages with the above content can flow between the blocks, delivering the

required information back and forth. It should be noted that, for the sake of the simulation, the ESMC

message rate was selected to be fixed at 1 message per second (as defined by [11] for the ESMC

periodic messages), with the ESMC event messages (that are transmitted immediately on the line

should the QL value change) not taking any active role in the TOD distribution process.

The ‘X’ port (input/output) is defined by a different API, termed Time/Frequency reference

API, and that is comprised by the signals:

Signal number Size [octets] Content

1 1 (char) QL � Quality Level indication of

the incoming/outgoing TOD

reference

2 4 (2 x unsigned

integer)

Accurate TOD representation of

external clock source (input) or

the node itself (output)



3 double Physical-layer instantaneous

frequency of the external clock

source (input) or the node itself

(output) (currently taken to be

constant)

Table 7 – Signal structure of the Time/Frequency reference signal API

The external API allows a specific node block to be timed (derive time and frequency) by an external

clock source or supply its own time and frequency to an external end-application (e.g. a time error

measurement device). This interface can thought of, in practical aspects, as a frequency reference

signal (e.g. a 2.048MHz clock) accompanied by a TOD reference signal that can realized, for example,

by a 1 PPS signal (transmitted over a dedicated BNC interface) accompanied by a matching standard

TOD code, such as NMEA 0183, transmitted over an RS-442 serial link. As these dedicated reference

interfaces are usually used to carry the time/frequency information over a relatively very short

distance (not more than a few meters usually), no inherit time error is assumed for this interfaces.

8.6.4.1.2 Internal structure

In terms of internal structure, the node prime functional block resembles a standard Sync-E

capable device with the additional functionality of the time recovery. The internal structure is depicted

in Figure 33.

The ESMC L/R Tx/Rx processes are derived by the local frequency, time and QL representations

marked by the frequency synthesis, time register and QL memory blocks. The ESMC blocks exchange

ESSM messages with their neighboring nodes and derive physical-layer frequency, time error and QL

from each. Additionally, a frequency, time and their associated QL might be available from some

external reference source.

Based on the incoming QL information from all 3 sources, the G.781 based clock selection block

selects the one with the highest QL to be the reference for the frequency and time recovery blocks. The

frequency recovery block is based on the G.8262 EEC characteristics while the time recovery block

was taken to follow the clock discipline algorithm of the NTPv4 standard [29] (see more in next

clause). The resultant frequency and time recovered information is then used to ‘close the loop’ and

derive the frequency and time synthesis blocks as well as the QL memory that, on their turn, will

derive, again, both the ESMC blocks as well as any external device. It should also be highlighted that,

in between updates, the node’s local time register is advanced by the node’s recovered clock. This is

marked by the small arrow connecting the two blocks.

Note: As was already mentioned before, the current simulations carried assumed perfect frequency

recovery by the node block with the understanding that for most scenarios the contribution of this

noise source would be miniscule. Nevertheless, as can be seen in Figure 33, it should be quite easy to

introduce imperfect frequency synchronization into the simulation environment simply by replacing

the currently perfect G.8262 based frequency recovery block with a more realistic functional block

that has a finite PLL bandwidth as well as a non-zero phase noise local oscillator.

Having no introduced noise on the account of background traffic (output Queue) nor due to

frequency recovery imperfections, the only noise source contributing here is the timestamping

quantization noise. In real implementations this noise is generated by the imperfect (usually HW



realized) timestamping procedure embedded within the ESMC Tx/Rx processes. The timestamping

quantization noise was realized in this simulation environment as the uniformly distributed process

with distribution where σ is the noise standard deviation that can be a-

priory configured.

Figure 33 – node prime functional block internal structure



8.6.4.1.3 Time recovery block

The time recovery block implements the NTPv4 clock disciplining algorithm as it appears in

[29]5. The NTPv4 clock disciplining algorithm was chosen for this purpose for the following reasons:

i) It is a rather simple algorithm that is based on well-known FLL/PLL designing principals.

ii) It is the only truly standardized time recovery algorithm that exists today.

The above reasons makes NTPv4 an almost perfect candidate for the simulation as it demonstrates

that using the suggested approach, the time recovery function, that in built to be quite complex in

many other time distribution apparatuses, can be maintained quite simple and still meet very good

performance.

The NTPv4 clock disciplining algorithms was used in this simulation with the following

parameters:

STEPT . 128

WATCH 900

PANICT 1000

PLL 10

FLL 1

AVG 4

ALLAN 1500

LIMIT 30

MAXFREQ 500e-6

PGATE 4

PRECISION -18

IGNORE 0

These parameters were chosen to yield fast time recovery convergence while still maintaining good time-stamp quantization noise filtering capabilities. For more in-depth information regarding the NTPv4 clock disciplining algorithm please refer to [29] and [30].

8.6.4.2 Primary Reference Time Clock (PRTC) block

The PRTC block is a complementary prime block to the node block. Its purpose, within the

simulation environment, is to supply a time/frequency reference to a given node block, thus

originating a time/frequency distribution chain (see Figure 32). It is comprises one ‘QLin’ input port,

used by the simulation environment to control its specific QL value, as well as one ‘Refout’

Time/Frequency reference signal API (as already defined) output port.

Figure 34 – PRTC prime functional block

5 It should be noted that only the clock disciplining algorithm (time servo-loop) of [29] was

implemented and not the entire NTPv4 time recovery suit that comprises many additional processes

and algorithms dedicated for mitigating impairments introduced by the packet network (i.e. filtering,

selection, clustering and combining algorithms).



8.6.4.3 Time error analyzer block

The Time error analyzer block is another complementary prime block that constitutes the “end-

application” of the simulation environment. As can be seen in Figure 35 below, the block has two

input ports of type Time/Frequency reference signal API. The ‘Rec in’ input port is used to receive the

time/frequency information from the DUT (the recovered time/frequency information) whereas the

‘Ref in’ input port is used to receive the time/frequency reference from which the comparison as well

as the error calculation would be derived.

Figure 35 – Time Error Analyzer prime functional block

The measured time error through the entire simulation time is recorded by the block and can be

monitored via its inherit scope function. The time-error scope is presented in Figure 36.

Figure 36 – Time error scope



The upper subplot presents the QL of the incoming clock (European SSM codes) while the bottom

subplot presents the time error, in every simulation instant compared to the time reference input.

8.6.5 Simulation Results

Two important properties of the new apparatus were examined by the simulation:

1. Steady-state: The steady-state time error performance across an entire distribution path

comprised of a number of nodes as well as the time to reach a complete steady-state

conditions across the entire distribution path.

2. Restoration condition: The time-error performance while doing network time distribution

restoration (as a result of some configuration change or fault detected) as well as the time to

reach a complete network restoration.

In order to test the steady-state property, the ring network topology presented in Figure 37 was

simulated. 10 nodes are connected in a ring topology. The link between node10 and node1 is left

disconnected in order to avoid an Ethernet infinite loop6.

Figure 37 – Steady-state test setup

6 Such a property is usually achieved by employing some type of Ethernet ring protocol (e.g. G.8032)

that makes sure that at least one link within the ring is left disconnected at any given time.



The PRTC connected to node1 with QL PRC is starting the TOD distribution chain at simulation time

0. The instantaneous QL and TOD associated with each node along the distribution chain is

monitored by the time error analyzer. In order to introduce some randomness into the time

distribution process the initial TOD of each node was randomly uniformly selected in a time window

of ±1 msec compared to some common time epoch. The timestamping quantization noise was selected

to be approximately 5.7 nsec yielding a quantization noise magnitude of ±20 msec, a customary noise

level in today’s HW implementations.

The obtained results are presented in Figure 38 and Figure 39 (each color represents a different node

according to the below given key). As can be seen from Figure 38, the TOD convergence process for

the entire distribution chain of 10 nodes take about 160 seconds to complete, while the QL

propagation time is, as was expected, 10 seconds (one second for each participating node). According

to Figure 39, steady-state performance is reached after ~300 seconds with a time error lower than ±10

nsec.

Figure 38 – Steady state test: distribution chain convergence process (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan,

node9=magenta, node10=yellow)



Figure 39 – Steady state test: distribution chain steady-state performance (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan,


Next, ring restoration was simulated based on the setup presented in

Figure 40.

Figure 40 – Ring restoration test setup



This time a second PRTC, with the same TOD setting as the first PRTC, was connected to node10. The

QL of the second PRTC was configured to be SSU-A (‘0100’ or 4) while the QL of the first PRTC was

initially configured to be PRC (‘0010’ or 2). After 500 seconds of simulation runtime, the QL of the

first PRTC was degraded to SSU-B (‘1000’ or 8). The obtained results are presented in Figure 41. The

QL change group of events as a result of the ring restoration is clearly evident. Zooming-in on this

group of events, the specific nature of the QL change across the entire distribution chain can tracked.

Once the QL of PRTC1 drops from PRC (2) to SSU-B (8), the new QL starts to propagate across the

distribution chain node after node. When it reaches the last node10, the clock selection process within

node10 selects PRTC2 with QL SSU-A (4) to be its clock source (marked by the yellow line climbing

from 2 to 4 at time=509 sec). From here, the restoration process begins where the new QL SSU-A

starts to propagates towards node1. The entire QL restoration process lasts ~18 seconds.

Figure 41 – Ring restoration process (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan, node9=magenta, node10=yellow)

The time-error performance during the restoration process is presented in Figure 42. No time error

transient of any sort is evident. This can be explained by the fact that the two PRTC were configured

for the exact same TOD settings. Should some small mismatch between the TOD setting of the two

PRTC exists, a small transient process might have occurred. Nevertheless, more clever hitless

mechanisms (of the sort that are extensively being used for frequency distribution) would easily solve

this problem on the price of longer holdover periods while in the restoration process. Nevertheless,

having a synchronized physical-layer (thus, a very accurate holdover frequency) this should not

become a major issue.



Figure 42 – Steady state test: distribution chain steady-state performance (node1=red, node2=cyan, node3=magenta, node4=yellow, node5=blue, node6=green, node7=red, node8=cyan,


8.6.6 Conclusions

Link-by-link TOD distribution can be achieved by extending the Sync-E ESMC messages to

distribute TOD information in addition to the SSM information its distributes today. The benefits of

this approach are:

• A “light” TOD distribution scheme that demand very low network resources (one message

per second on every link) but still able to give state-of-the-art level of performance.

• Based on existing clock selection algorithm. No need to re-invent the wheel.

• Saves the need for an additional higher-layer TOD distribution protocol such as PTPv2,

imposing further resources allocation on the network and requiring an additional

management plane and increasing the operator CAPEX and OPEX costs.

• Complete inter-work with the physical layer frequency distribution (the clock selection

process taking place at the node actually applies for both). No need for special discovery

mechanisms and complicated management considerations.

• Can be easily extended using any higher-layer standard time distribution protocol (e.g.

PTPv2 or NTP). In order to do so, one only needs to implement a special “boundary clock”

comprising an ESMC TOD recovery function (a slave device) feeding any higher-layer

master/server device. The opposite case can also be considered (PTPv2/NTP to Sync-E

TOD).

The suggested TOD distribution scheme that was described in this clause will need to be re-

visited after the current work, ongoing in the ITU-T SG15/Q13, to define a new time distribution

PTPv2 telecom profile will be completed. When that happens, the proposed scheme could be re-

evaluated in order to learn it can fully comply with the requirements of the new telecom profile.



Comprehensive comparisons to existing higher-later TOD distribution scheme could then be

conducted again to learn about its true value.



9 Conclusion

The field of frequency and time distribution over packet switched networks is rapidly moving

forward with new improved solutions to meet the stringent requirement of new end-applications

constantly appearing. The rapid, ever growing demand for new services together with the strong

competitiveness between service-provides and their desire to increase their value offering by

introducing more and more cutting-edge technologies, result in a technology migration process that is

not always smooth and bump less.

The result is quite a wide spectrum of new offered technologies where each one it tuned to solve

a specific problem in a very good way. Nevertheless, these technologies do not often interact or co-

exist in a smooth way to give the service provider a global optimized solution for his network.

This report summarizes the work carried over the past 1.5 years with the attempt to define a

portfolio of tools and solutions that will enable a network designer/service provider to make better use

and planning of the existing technologies he got. Whether these are new mechanisms to better

interwork PTPv2, NTP and Sync-E, new insights to how QoS can be efficiently employed to improve

overall end-to-end performance, new ideas to enable accurate time and frequency distribution over an

IPSec tunnel or a new apparatus to extend a widely-spread well known frequency distribution

technique to additionally support time distribution, the common denominator that binds them all

within this framework is the attempt to bridge across these multitude of new technologies perhaps

allowing for a smoother more orderly migration from one to the other.

A few years from now, networks will offer full support, allowing distributing end-to-end

frequency and time with formidable accuracies to service even the most demanding application.

Technology wise, the industry will probably converge into only a few winning technologies that will

easily interwork and bring great value to the service provides. Nevertheless, we are not there yet and

the work included in this report aims at making this journey an easier one.

* * *

d2.4 synchronization service in carrier-grade ethernet...

Documents