ip-tdm converged optical industrial networks_issue 1

IP-TDM Converged Optical Industrial Networks

White Paper Authors: Volker Meschonat & Stefaan Verhoeven Date: March 26, 2010 Document no.: AA-W050-E-1

of 28 White Paper • AA-W050-E-1 © OTN Systems NV • All rights reserved

OTN Systems NV Atealaan 34B B-2200 Herentals Belgium www.otnsystems.com [email protected] Disclaimer: OTN Systems NV has written this document with the greatest care. Nevertheless, information published here may be incomplete, outdated or no longer correct. OTN Systems NV may amend published information at any time, without being required to give notice. Liability: Users may not derive any rights from information published in this document. OTN Systems NV does not accept any liability for the contents and use of this document, or for any damage or losses resulting from its use.

http://www.otnsystems.com/

mailto:[email protected]

AA-W050-E-1 of 28 © OTN Systems NV • All rights reserved

Table of Contents 1 Introduction 5

2 Current Ethernet and TDM Networks 7

2.1 Rise of Ethernet from the LAN to the MAN and WAN 7

2.2 Support of Ethernet in TDM networks 7

3 Keeping control of the network 9

3.1 Network control is going to protocols 9

3.2 Dedicated connections instead of Any-To-Any Connectivity 10

3.3 Having control instead of being controlled 10

3.4 Redundancy paths in Ethernet networks could lead to everlasting loops 10

3.5 Loop protection 12

3.6 Service recovery times 12

3.7 Non-Stop behavior of OTN Rings 12

4 Quality of Service (QoS) 13

4.1 General 13

4.2 Congestion 13

4.3 QoS implementation in Ethernet switches 14

4.4 Connection-oriented Ethernet via the OTN S-LAN, exhausted queuing 15

4.5 Network Performance: wire-rate, wire-speed or frame/packet loss, delay, and jitter 16

4.6 Frame Losses and Buffer Sizes 17

4.7 Extensive Ethernet testing: BER, RFC2544 17

4.8 Peace of mind 18

5 Network Security 18

5.1 Ethernet technology: Denial-of-Service Attacks possible 18

5.2 No DoS attacks possible on OTN Networks 19

6 Network and Service provisioning 20

6.1 Provisioning and change of services, extension of networks 20

6.2 Considerable education and certification/recertification efforts 21

6.3 OTN Management System (OMS) Point-N-Click, End-To-End provisioning – Service Enabling

Architecture 21

7 Network uptime and reliability 22

7.1 Fault correction in Ethernet Networks 22

7.2 Availability given for hardware only 23

7.3 OTN System MTBF available, 10-year delivery guarantee 23


7.4 Safety Integrity Level 3 (SIL 3) – certified 24

7.5 Fault clearance and re-commissioning after heavy incidents in OTN 24

8 OTN overview and its characteristics 25

8.1 OTN Systems’ nodes: purpose-built products 25

8.2 Support of legacy and Ethernet interfaces plus embedded video control 25

8.3 Customer-specific interface cards, customer-specific nodes possible 26

8.4 Extended Ethernet architectures to come soon 27

9 Conclusions 28

List of figures Figure 1-1 Keeping the assets while migrating to a new core network 6

Figure 3-1 Broadcast storm 11

Figure 3-2 Alternative path for redundancy 11

Figure 4-1 Congestion caused by rogue user 13

Figure 4-2 Example of a Poor QoS implementation 14

Figure 4-3 Example of a Good QoS implementation with overbooking 14

Figure 4-4 Example of a Good QoS implementation without overbooking 15

Figure 4-5 OTN S-LAN concept to guarantee bandwidth per service and to -perform exhausted queuing 16

Figure 5-1 Ethernet network elements open for attacks 19

Figure 6-1 Example of Cisco IOS management system interfaces 20

Figure 6-2 Ethernet-based networks can cause huge education efforts 21

Figure 6-3 Point-N-Click service provisioning by OTN network Management System 22

Figure 6-4 OTN network Management System OMS able to control huge topologies 22

Figure 7-1 Real-life “System-MTBF” available 23

Figure 7-2 SIL 3 - Reducing the probability of dangerous failures 24

Figure 8-1 OTN support of legacy, Ethernet, plus hybrid video 26

Figure 8-2 Overview of the different OTN Node families 26

Figure 8-3 Customer-specific, purpose-built OTN node 27

Figure 8-4 Ethernet extension by additional rings or spokes 27


1 Introduction

Telecom network divisions of industrial companies that are active in Oil & Gas, Metros, Railways, Mines, Energy, Motorways, and other industrial segments in all regions over the world face a fundamental challenge today: on the one hand side, their applications based on the IP traffic continue to grow rapidly, but on the other hand there are still a lot of legacy applications in the field not yet based on IP (and some of them are even not foreseen to be at IP layer for a long time to come).

For years now, the basic telecommunications needs for these industrial companies have been unchanged. People still appreciate and demand for easy to manage and fully controllable telecom networks that do not let them down, even under the most harsh conditions.

What has changed over the last few years is that Ethernet has become the dominant networking protocol in a lot of applications (in the LAN or the Local Area Network), mainly because of its potential for cost effectiveness. The challenge to the industrial environments is the combination of TDM-based (like SONET/SDH or OTN Systems) transport equipment with the ever growing need for Ethernet/IP bandwidth. Network managers need to cope with both worlds in a reasonable manner while keeping the emphasis on their specific requirements and applications.

To meet these needs, said industrial companies need to enter into a new phase of TDM-to-packet-network convergence. In essence, these networks must achieve the following:

• Optimize communication transport costs within the organization

• Accommodate all existing legacy services

• Enable new emerging services

• Be open for upcoming services and interface/data formats

• Deliver operational simplicity

• Offer deterministic behavior

• Provide the required performance and resilience

• Minimize OPEX for running and maintaining the networks

• Guarantee a future proof investment

The products from OTN systems are designed to accommodate all these requirements in one platform portfolio. As such these products offer the perfect IP-TDM Optical Industrial Network convergence to accommodate the new emerging Ethernet / IP applications while offering a reliable deterministic transport behavior at the same time.

Already for many years, OTN Systems addresses the convergence of the new packet-based networks with the reliability and deterministic behavior of TDM technology.


OTN Systems added the first Ethernet card (ET-10) to its portfolio in 1995, allowing customers to smoothly integrate Ethernet-based IP applications into their networks. In recent years OTN Systems has extended its portfolio with a growing family of Ethernet cards with the unique S-LAN (Segmented/Secured LAN) technology as the perfect combination between both worlds: reliability, deterministic behavior, easy end-to-end provisioning and management, a broad range of interfaces and services and Ethernet-based applications with the highest QoS.

This white paper addresses the challenges and opportunities in modern industrial networks. It provides a means to assess the different technologies that are around today, how to exploit them at the highest benefits and to integrate the applications on a common convergent optical industrial network. It will become clear that OTN Systems is really able to offer the best of both worlds in one common platform. The convergence of the economies of keeping legacy plus Ethernet switching on the one hand and the smooth traffic flow, steady bandwidth and easy network management offered by the TDM-based SLAN technology on the other hand proves to be the ideal way forward for new IP-TDM Converged Optical Industrial Networks.

Figure 1-1 Keeping the assets while migrating to a new core network


2 Current Ethernet and TDM Networks

2.1 Rise of Ethernet from the LAN to the MAN and WAN

During the last 10 years, Ethernet networks have increasingly gained market traction and have been deployed massively in the enterprise LAN networks. Also in industrial networks, Ethernet is becoming more and more popular due to the applications that make use of Ethernet and IP technology.

It is clear that IP has become the dominant protocol for most high-bandwidth demanding end-user services and applications. The main drivers can be seen in the computerized offices with their overall IT-centric applications such as office appliances, storage of mission critical information and documentation and increasing usage of internet and intranet applications for information sharing and distribution.

Every enterprise in the world relies for the greatest part on its IT environment. This IT environment typically uses the IP protocol to connect computers with servers and with other office applications such as printers and storage disk arrays. The universal way to transport these IP streams is definitely Ethernet.

A few typical TDM services are shifting in the direction of IP as well. Examples are classical voice applications, moving to Voice over IP (VoIP) and video applications moving to IP-based video services using modern compression algorithms such as MPEG2, MPEG4 and H.264. Also in the SCADA world, IP is becoming more and more important with the emergence of IP-based PLCs.

More recently, Ethernet has been evolving from its enterprise LAN roots towards a MAN/WAN scope, with higher bandwidths, fiber optic technology and advanced features needed for wide-area networking. New standards are emerging to turn IP and Ethernet devices into carrier class equipment (e.g. IP-MPLS). Besides its classic connectionless and best effort character, connection-oriented services possible as well. Although IP-MPLS is currently already widespread in telecom carrier core environments, it has not yet found its way into industrial networks due to its excessive complexity to design, operate and maintain.

Fortunately, we might add, since the carriers have experienced the high CAPEX and OPEX costs involved in maintaining IP-MPLS, they are looking today for more cost optimized IP/Ethernet transport solutions to be used in their future networks (e.g. T-MPLS, PBB-TE, MPLS-TP, PWE3…). As these technologies are still in their infancy, in the lab and test phase at major vendors or even still need to be standardized, it will still take a few years before the first reliable MPLS-transport solutions will be available at reasonable cost for industrial networks.

2.2 Support of Ethernet in TDM networks

Modern SDH/SONET TDM-based networks such as OTN (Open Transport Network) or NG-SDH technology have an increasing support and integration of Ethernet and IP as interfaces towards the transport system. Next-generation SDH incorporating Ethernet-over-SDH (EoSDH) capabilities makes SDH a better platform for data requirements, as it combines efficient data transport with the carrier-class protection inherent in SDH and the stability of a proven, widely deployed technology.


Although NG-SDH (also called Multiservice Provisioning Platforms/MSPP, or Multiservice Transport Platforms, MSTP) has been widely deployed in telecom carrier environments for transport of Ethernet over SDH, Industrial Networks still need an additional level of flexibility and simplicity, one of the major strengths of OTN Systems.

One of the major issues here is the way in which Ethernet interfaces are mapped into SDH in the current MSPP platforms. The implementation of the newest features such as GFP (Generic Framing Procedure), VCAT (Virtual Concatenation) and LCAS (Link Capacity Adjustment Scheme) have brought a much better flexibility to the SDH world as opposed to the standard SDH solutions. However it implies a higher degree of complexity to design and operate the networks.

The Ethernet traffic received on the interfaces in NG-SDH is mapped into a number of Virtual Containers (VCs) and is transported in these VCs to the destination, where the data is unmapped and sent out on the Ethernet interface. Thus EoSDH mapping provides point-to-point connections where the traditional client interfaces (for instance E1) are replaced with Ethernet interfaces.

The virtual concatenation addresses the flexibility and scalability problem in traditional SDH networks. The flexibility problem comes into play in the fact that until now capacity, through the SDH network has been assigned in steps from VC-12 to VC-3 to VC-4. To map the Ethernet interfaces in concatenated containers, one still needs to calculate in a multiple of the container capacities to try to come close to the desired Ethernet bandwidth. This is not a very flexible bandwidth assignment as it wastes capacity and is hence not satisfying for customers.

Since OTN is specifically designed to be used in industrial networks, this flexibility barrier and complexity has been overcome. With OTN Systems, network operators don’t have to worry about these complex mappings. Instead, via the Network Management System, the needed bandwidth is programmed in the most flexible way without any loss or inefficient usage of bandwidth capacity on the network.

Specifically with OTN, the network operator of Industrial networks benefits from the following:

• Direct connection of Ethernet to the network as opposed to the traditional PDH/SDH

• E1, E3 multiplexing solutions

• Flexible bandwidth granularity – the bandwidth of the Ethernet services can be programmed in steps of 1 Mbps as opposed to the traditional SDH steps of E1, E3 and STM-1. This is especially important to easily provide higher bandwidths such as with Gigabit Ethernet interfaces

• Service bandwidth upgrades and downgrades are easily configured in just a few minutes by the Network Management System

• A strict separation of different Ethernet segments over the network is provided (SLAN)

• Electrical 10/100/1000Base-T Ethernet ports as well as SFP-based 100Base-FX and 1000Base-X ports provide ultimate connectivity


• Perfect mix of low-speed data, analog and digital voice, analog and digital video and Ethernet interfaces on the same box and on the same network

This way, the benefits of both network technologies are combined: a perfect match of the versatility and widespread use of Ethernet technology and the reliability, deterministic behavior and predictability of TDM.

3 Keeping control of the network

One of the questions network operators in industrial networks are faced with is why we don’t use only an Ethernet network instead of a TDM network. This question is a valid one, since a simple Ethernet Network is cheap, a huge portion of bandwidth to be transported is IP anyway and it has already implemented Ethernet for the office applications.

In the following paragraphs, we will explain where the pitfalls of this approach are and provide a clear explanation on how and why you need to keep your network under control in all circumstances.

3.1 Network control is going to protocols

An important characteristic of IP and Ethernet technologies is their connectionless nature. Packets are routed or switched on an individual basis and statistically multiplexed onto shared links, allowing for very efficient use of bandwidth, but implying a loss of network control.

Due to their ability to learn the way devices are connected to the network, Ethernet switches and their inherent protocols simplify the life of the administrator on the one hand – this is Plug-N-Play provisioning – but on the other hand this means that these protocols and behaviors take over the control of the network.

This becomes more and more the nightmare of network operators, especially in Industrial Networks. Indeed, it is of the utmost importance to guarantee the correct delivery of information from one point to another, keeping timing, delay and jitter under strict control. Signaling and automation applications cannot live with packets or Ethernet frames coming in different orders, with different delays and jitter. Also with video and voice applications over IP, correct timing, fixed delay, order and jitter are important to avoid glitches in the voice communications and freeze frames in video applications.

This is why in Industrial Networks, the TDM network is by far the most preferred technology to build the network, while providing a means to transport the increasing number of Ethernet / IP applications over the same reliable deterministic TDM backbone.


3.2 Dedicated connections instead of Any-To-Any Connectivity

There are typically two types of networks: Any-To-Any-Connectivity networks like in LAN environments or huge Telecommunication carrier networks; and on the other hand networks, where a Control Center has to be connected to a high number of points with which it communicates, or which it controls over a long period of time (Point-To-Point or multipoint, connection oriented connectivity). The nodes of OTN Systems have traditionally best fit in the latter type of networks, which require a number of dedicated connections/services. The overall set-up of these services was connection-oriented in the past and will continue to be so in the future.

Beside classical Any-To-Any technologies like Ethernet, IP or IPv6 there is a strong tendency at large Telco carriers to regain the control over the network (core) with solutions like P-OTS, PBB-TE, or MPLS (and derivates). These technologies however not only increase the cost again, but also, and what is more important, the complexity to design, operate and maintain. The key to keep control lies in the management system, not so much in the underlying transport technology.

3.3 Having control instead of being controlled

This is where OTN Systems perform at their best. The control lies with the network operator and the tool he uses for that is the Network Management System. Once provisioned, a connection will always be available, without any interference of other applications, no matter what.

The user of OTN Systems technology does not need to know much details about the underlying technology – he just provides or changes the services. There is no protocol to take over the control or to select the connection.

3.4 Redundancy paths in Ethernet networks could lead to everlasting loops

Any-to-any connections are learned by the Ethernet network itself. If a higher redundancy requirement exists, the network itself will look for an alternative/redundancy path to reach the target

The Ethernet frame format includes no TTL (time-to-live) field. In case of loops, everlasting frames (e.g. broadcasts to ask for a MAC address) may occur and fill the network, and consequently you will have no user traffic and no remote access to the box (if it is in-band) either. You have to make sure that no loops occur.


Figure 3-1 Broadcast storm

Loop protection can be built in by using a standardized algorithm, called the Rapid Spanning Tree Protocol and its inherent rules (a.k.a. RSTP, included in IEEE standard 802.1D-2004). A loop-free tree is built by the network elements; one of them will become the root element. If a fault in the network occurs, RSTP is typically able to respond to changes within three (3) hello times (typically six (6) seconds) or less.

The maximum number of network elements counted from the root bridge is typically seven (7). This does not help for medium and large Ethernet environments, where segments have to be separated by routers, and the network administrator arrives at layer 3 of the OSI model.

Figure 3-2 Alternative path for redundancy


3.5 Loop protection

It could be that the network administrator decides to separate the traffic by means of VLANs or stacked VLANs. For these topologies there are standardized loop protection mechanisms available as well , which make the loop protection thinking more complex.

Several vendors have developed proprietary loop protection protocols . Realized failover times are sub-second, sometimes sub-50ms. Ethernet Ring Protection Switching, or ERPS, is currently an effort at ITU-T to provide sub-50ms protection for Ethernet traffic in a ring topology while at the same time ensuring that there are no loops formed at the Ethernet layer. These protocols allow for a higher number of network elements. Today, compatibility between different vendors implementing these kinds of protection schemes is hard to find.

3.6 Service recovery times

After detection of a topology change by RSTP (max. 6 seconds) the new topology will be established and the learning on the ports will start again. The speed of learning depends on the learning rate of the switches. Once relearned the services are up again. Compared to TDM-based layer 1 systems this means a huge delay between link recovery and service recovery.

3.7 Non-Stop behavior of OTN Rings

OTN is the backbone for mission critical applications. Therefore it is designed for reliable and non-stop 24/7 operation during a lifetime of over 10 years. One of the features that guarantee the non-stop behavior of OTN includes the following: dedicated and guaranteed bandwidth for each individual connection. This excludes unwanted interference when new services are added or if existing services are overloaded.

To improve the reliability of the fiber optic backbone, a redundant fiber optic ring topology is used. This allows the OTN network to restore itself automatically in case of an optical fiber cable break. Multiple of these redundant OTN rings can be coupled to provide the optimal network topology. The redundant ring concept automatically restores network connectivity in case of an optical fiber break. No configuration effort or network management action is needed for this. The reconfiguration is 100% predictable and once the network path has been restored, all applications will be restored at the same time.

The node design is largely hardware-based, requiring little or no updates. In the event an update is needed, this can be carried out centrally, without the need to go on site. Critical components, such as power supplies or common logic cards (BORA) can be protected by a redundant unit. Thanks to the plug-and-play concept, the Mean Time To Repair (MTTR) can be kept very short.


4 Quality of Service (QoS)

4.1 General

QoS is the ability to treat frames or packets differently as they traverse a network device, based on the packet contents. Without QoS, all packets on the network need to use the same pool of resources, and when congestion occurs, high priority traffic is heavily impacted by low priority traffic. QoS can be used to address issues such as slow applications, jerky video and poor sound quality. QoS can be expressed in terms of packet loss, delay, and jitter.

4.2 Congestion

Typically, Ethernet & IP networks operate on a best-effort delivery basis, which means that all traffic has equal priority and an equal chance of being delivered in a timely manner. When congestion occurs, all traffic has an equal chance of being dropped. Congestion may occur in LAN switches when more than one input port is contending for a single output port or when a high-speed input port is forwarding traffic to a lower speed output port. If the level of congestion is high and lasts for a period of time, packets may be lost due to overflowing buffers of the port under stress.

Attacks from rogue stations let the switches sweat. Any latest PC is able to pump easily up to 800 Mbps (PCI bus 100 MBps) into the networks. You can easily imagine what will happen if only two of these PCs want to reach the same target at the same time and this is all unwanted traffic?

Figure 4-1 Congestion caused by rogue user


4.3 QoS implementation in Ethernet switches

Ethernet offers the possibility to handle different colored traffic differently, and to give these types of traffic bandwidth per definition. Low-cost Ethernet switches do not have good QOS mechanisms at all. A general rule is that the better the QOS implementation, the higher the cost will be. Please find below some examples of different real-life QoS implementations in Ethernet switches from different manufacturers, measured in our labs.

Figure 4-2 Example of a Poor QoS implementation

Figure 4-3 Example of a Good QoS implementation with overbooking


Figure 4-4 Example of a Good QoS implementation without overbooking

To guarantee the bandwidth per channel/VLAN, an high-end egress (outgoing) side must be able to shape the egress rate of each switch and has to be programmed on each switch and per VLAN separately. The guaranteed bandwidth must be configured on each of the ports for the egress side. This functionality is much more than the standard VLAN that is found on the simple VLAN switches and will definitely increase the complexity to design and maintain the network.

Regular switches offer eight (8) queues per port by hardware. Only high-end switches offer more. This has to be taken into account while engineering.

With a proper QoS implementation the switch can also offer a Committed Information Rate (CIR), but only for the number of hardware queues it offers. Exhausted queuing (fine granularity per service CIR until reaching the maximum available bandwidth) is offered by very high-end switches only.

This affects the Service-Level-Agreements as well. SLAs can be given for all services but can be guaranteed for a small number of services only; the number of these is limited by the number of available queues per network element.

4.4 Connection-oriented Ethernet via the OTN S-LAN, exhausted queuing

Every service has been getting its dedicated bandwidth in OTN since the early days. This basic philosophy of OTN is adoptedfor Ethernet services. Every port or VLAN-tagged input (service) receives its bandwidth as Committed Information Rate (CIR) at the highest QoS level.

We call this pipe inside the backbone a Segmented LAN (S-LAN). The S-LAN provides OTN with a packet-heavy functionality in its ET100AE/DAE card family. In a S-LAN, there is no need for policing, queues or priorities. Each application or Ethernet connection gets its own ‘lane’ or layer in the network with its own guaranteed bandwidth, independently of any other Ethernet services. The number of S-LANs is limited only by the total available bandwidth.


Thanks to this S-LAN concept, OTN can guarantee total network virtualization that can be controlled and provisioned by a few simple clicks in the Network Management System. This way, your SCADA, CCTV, voice telephone and any other applications run smoothly on OTN, without interfering with each other.

Figure 4-5 OTN S-LAN concept to guarantee bandwidth per service and to -perform exhausted queuing

4.5 Network Performance: wire-rate, wire-speed or frame/packet loss, delay, and jitter

In Ethernet-based networks there will be a variety of different applications on the same network. This fuels the need for Quality of Service (QoS) to ensure reliability of services for each type of service they offer. These networks are “Best Effort Networks” per definition and do not guarantee the same level of guaranteed performance as would be expected from a TDM network such as OTN.

The ideal would be no packet loss (layer 3),no frame loss (layer 2) (wire-rate), zero latency of the signal, zero delay by the network element (wire-speed), and no jitter.

As network operators introduce new real-time Ethernet-based services such as IP-video and voice-over-IP (VoIP), latency and frame loss can have a significant impact on these applications.

Packet jitter is usually caused by queuing and routing across a network, or buffering in switched transport networks. Although the variation in packet delay should be minimal in a core 10GigE network, jitter may be introduced if bandwidth demand increases and more buffering occurs. Jitter should not occur in core networks at low utilization rates.

But Ethernet on its own does not provide end-to-end monitoring or point-and-click provisioning mechanisms or a default QoS better than “Best Effort”, all of which is possible in TDM networks.


While additional features can be added to Ethernet networks to improve their performance, each one must be implemented and maintained by the network operator separately and this on a network-wide scale.

4.6 Frame Losses and Buffer Sizes

Another traffic engineering issue is the use of “pause packets”, a flow-control protocol defined by IEEE 802.3x, to slow down the ingress traffic on a point-to-point service when the internal ingress buffers are close to saturation. When a threshold in the buffer, set by the network operator, is passed, this scheme sends a pause packet to the downstreaming network element in order to stop the transmission of traffic until the local buffers are ready to receive again.

This threshold must be set manually at the right level to maximize throughput while not losing any frames. The effect of the pause frame is largely determined by the time taken for it to be transmitted and received at the downstream end. If the pause threshold is set:

• too high, the reaction time will be too slow and the local buffers will overflow, causing frames to be lost,

• too low, the pause frames will be sent too often, reducing the overall throughput and increasing packet jitter.

4.7 Extensive Ethernet testing: BER, RFC2544

TDM Networks such as OTN Systems, carry Ethernet applications in a transparent way. Ethernet-based networks should transport the applications in the same manner, but by nature they cannot. Errors of both types of networks can be measured on Bit Error level with a Bit Error Rate (BER) test that checks a pseudo-random bit sequence inside an Ethernet frame.

If the network to be tested is packet-switch-based the network processing element will discard frames or packets if an error is found and hence most errors will never reach the test equipment. These lost frames are difficult to translate into a BER value. Therefore, a number of network parameters should be tested to ensure that a deployed Ethernet system delivers an acceptable QoS and to show the users that the network complies with the performance metrics needed to run their applications.

The common way to test a network during installation and commissioning in order to evaluate the performance criteria is to use RFC2544 (benchmarking methodology for network interconnect devices) test procedures. These incorporate four major tests:

• The throughput determines the maximum rate at which the device or network can operate without dropping any of the transmitted frames;

• The back-to-back (a.k.a. burst ability) value is the number of frames in the longest burst that the network or device can handle, without losing any frames;


• The frame-loss is the percentage of frames that is not forwarded by a network device under steady-state (constant) loads due to lack of resources;

• The latency measures the time taken by a frame to cross a network or device - either in one direction or round-trip.

Packet jitter tests should be performed under heavy load to provide enough information to fully understand the behavior of the network and to foresee any QoS issues at maximum utilization. The maximum acceptable level of jitter is:

• 30 ms for IP-video applications and

• 10…20 ms for VoIP

4.8 Peace of mind

As outlined in this chapter, ensuring the quality of a network requires much more tests and control in an Ethernet-based network compared to a TDM Network.

Where TDM networks work in a transparent way, with fixed delay and virtually no jitter, simple BER tests will ensure proper end-to-end service testing. Traffic engineering is much simpler in OTN Systems compared to any Ethernet-based network and allows the operators of OTN Networks to save on OPEX when installing and commissioning their networks.

5 Network Security

5.1 Ethernet technology: Denial-of-Service Attacks possible

In Ethernet networks (like in all other networks higher than OSI layer 1), the bearer traffic frames and the protocol traffic frames (e.g. for RSTP) are transported via the same connection. Each PC, which is connected to a network element, would be able to sniff the net and to inject such protocol packets (BPDUs) into the network to disturb its regular operation - being always open for Denial-of-Service/Distributed-Denial-of-Service (DoS/DDoS) attacks.

A DoS attack is characterized by an explicit attempt by attackers (hackers) to prevent legitimate users of a service from using that service. Attacks can be directed at any network device using in-band protocols. A DoS attack can be issued in a number of ways.

The basic types of attack are:

• Consumption of computational resources, e.g. bandwidth or processor time

• Disruption of configuration information, such as switching/routing information

• Disruption of state information, such as unsolicited resetting of TCP sessions


• Disruption of physical network components

• Obstruction of the communication media between the intended users and the victim so that they can no longer communicate adequately.

Figure 5-1 Ethernet network elements open for attacks

A DoS attack may include execution of malware intended to:

• Max out the switch processor's usage, preventing any work from occurring

• Trigger errors in the microcode of the machine

• Trigger errors in the sequencing of instructions, so as to force the switch into an unstable state or lock-up

• Exploit errors in the operating system, causing resource starvation and/or thrashing, i.e. to use up all available facilities so no real work can be accomplished

• Crash the operating system itself.

5.2 No DoS attacks possible on OTN Networks

By definition, an OTN network cannot be attacked. User data and control traffic is physically separated, and it is just transporting information in a proper manner with a predictable behavior.


6 Network and Service provisioning

6.1 Provisioning and change of services, extension of networks

In Ethernet Networks, every path and service installs itself according to protocols. Access to the network element typically takes place via the Command-Line-Interface (CLI). Configurations are carried out on a port/interface/machine basis. End-to-End view is possible via an overlaid network management system only. Tool-based provisioning of Point-to-Point, Point-to-Multipoint, Multipoint-to-Multipoint, Broadcast, or Multicast services is provided in complex and expensive management consoles only.

Figure 6-1 Example of Cisco IOS management system interfaces

One can find below an example on how to program an interface policer via CLI (the policer marks traffic and limits the bandwidth utilization):

Bridge (config-pmap-c)# police [flow] bits_per_second normal_burst_bytes [maximum_burst_bytes] [pir1 peak_rate_bps] [[[conform-action {drop | set-dscp-transmit2 dscp_value | set-prec-transmit2 ip_precedence_value | transmit}] exceed-action {drop | policed-dscp | transmit}] violate-action1 {drop | policed-dscp | transmit}]

It will be clear that knowing all CLI commands and keeping a clear view on the characteristics and configurations of the network elements and their associated ports is a job for highly skilled specialists and therefore very difficult to manage by most network operators.


6.2 Considerable education and certification/recertification efforts

Considerable education efforts with regard to network installation, commissioning, driving, changing, addressing, security, loop protection, etc. have to be taken into account. The operator’s staff has to be educated and to this end take appropriate training courses as to both amount and quality. Experience shows that we are talking about weeks of being missed at the job, which is also expensive for the employer.

Said training is normally necessary not only to engineer and drive the network but also to get access to the vendor’s helpdesk or Technical Assistance Center (TAC) in case of problems while using the equipment. To gain this access the staff has to have a valid certification, which normally outdates after a certain period of time (2-3 years). Training and certification is one of the hidden cost parts in the Total Cost of Ownership (TCO) that comes with complex Ethernet networks.

Figure 6-2 Ethernet-based networks can cause huge education efforts

6.3 OTN Management System (OMS) Point-N-Click, End-To-End provisioning – Service Enabling Architecture

It is as simple as saying hello. Just define where the service comes in and where it should be dropped or multi-dropped, by point-N-clicking in the OTN Network Management System OMS.


Figure 6-3 Point-N-Click service provisioning by OTN network Management System

Figure 6-4 OTN network Management System OMS able to control huge topologies

7 Network uptime and reliability

7.1 Fault correction in Ethernet Networks

After occurrence of a fault, the correction measures have to cover both layers of the OSI stack, i.e. the physical and the data link layer (1&2), in the right sequence. The loop protection mechanism in each switch, its pre-setting and ID to influence the position of the root in the network etc. must be checked. In medium or large networks this could be very long lasting.


7.2 Availability given for hardware only

Outage times of services depend on needed boot time and learning rate after a failure, the stability of the used software, and the reliability of hardware and the failover time of the loop protection protocol.

High availability of switches is needed. MTBF values are available and given for hardware only. Software availability is not taken into account to calculate the entire system availability. Also note that there are many different software versions around supporting or not supporting certain features.

Software updates are very frequent and often lead to excessive (but planned) downtimes.

Having double switching fabrics and a controller for redundancy and software switchover purposes, and at the same time having a simplifying management, provisioning and error detection system is possible but will inevitably lead to higher costs.

7.3 OTN System MTBF available, 10-year delivery guarantee

For OTN systems networks, next to very high MTBF figures for each of the components, also the total system availability is provided. MTBF figures are calculated based on the international standards, but OTN Systems goes one step further. The MTBF figures have been constantly adapted and challenged based on real-life verification of networks out in the field for more than 10 years.

Figure 7-1 Real-life “System-MTBF” available

OTN Systems also guarantees availability of the same or of a better product up to ten (10) years following the Last-Order-Date (LOD).


7.4 Safety Integrity Level 3 (SIL 3) – certified

Safety is the measure of being free from unacceptable risk of physical injury or of damage to the health of people, either directly or indirectly as a result of damage to property or to the environment. Functional safety is part of the overall safety. Functional Safety, as defined by IEC standard 61508, is the safety that control systems provide to an overall process or plant. Neither safety nor functional safety can be determined without considering the systems as a whole and the environment with which they interact.

Figure 7-2 SIL 3 - Reducing the probability of dangerous failures

A SIL is a measure of safety system performance, in terms of probability of failure on demand (PFD). Four (4) SILs are defined, with SIL4 being the most reliable and SIL1 being the least.

Under the test report reference #U0SX0001 the accredited and independent test labs of Nokia Siemens Networks have certified, that networks of OTN Systems node type N42 fulfill SIL3 for services spanned the following way:

ingress Ethernet-Interface -> ingress node -> backbone -> egress node -> egress Ethernet-Interface.

7.5 Fault clearance and re-commissioning after heavy incidents in OTN

Murphy never sleeps, and even without terroristic attacks and with SIL3 there could be breakdowns of the network. The fault clearance of an OTN systems network is a four (4) step job after reading the defect shown in OMS:

Switch-off the defect card pull it out plug in the spare card switch it on.

This can be done under heavy stress at any time of the day – not having to worry about protocols or address space. The defect is instantly repaired and all services are restored at the same time.


8 OTN overview and its characteristics

• Each network, no matter which technology,

• has to be purchased (price)

• has to be installed (initial design effort)

• has to be powered (running energy and cooling costs, you can even think green)

• has to be able to offer new applications and services later on (refreshing and redesign costs, upgrade costs for new features)

• has to accommodate new devices and users (generates efforts and costs for these kind of adds, moves, or changes)

• has to be operated and monitored (administrative costs).

Each aspect should be thoroughly analyzed to verify whether the pure CAPEX and initial OPEX do not hide a lifetime penalty..

From the very first beginning until today all OTN Systems product families were designed to offer no hidden costs, to be open for legacy and new services and to keep the pace of technology in terms of speed, reach, and interface types.

8.1 OTN Systems’ nodes: purpose-built products

OTN Systems’ nodes (3rd decade) are used in specifically addressed markets. Our customers with their industrial network environments care about the control of their network and about the support of legacy and new applications. OTN Systems’ portfolio is built just for those reasons. OTN Systems allows full control, supports the legacy applications, makes DoS attacks impossible, and allows for a perfect mix and match of all these different applications on the same network with speeds up to 10 Gbps to offer plenty of capacity.

8.2 Support of legacy and Ethernet interfaces plus embedded video control

OTN is committed to further support all the legacy interfaces, which are already available (see below), independent of the underlying transport technology in the backbone core of the network.

Data transport: RS232, RS422, RS485, 64kbps (G.703); LAN: Standard, Fast, Gigabit Ethernet; Telephony: E1/ T1, FXS,FXO: 2-wire (a/b) analog telephony, 2/4-wire E&M analog voice, S0 digital (ISDN) telephony, UP0(E) digital (Siemens Hicom) telephony; Public Addressing: High quality music (including stereo) or voice messages; Video applications (CCTV, video distribution): PAL (B/G), NTSC (M, CVBS), H.264, MPEG2/4 and M-JPEG compression standard. Distributed audio/video matrix functionality integrated.


Figure 8-1 OTN support of legacy, Ethernet, plus hybrid video

Figure 8-2 Overview of the different OTN Node families

8.3 Customer-specific interface cards, customer-specific nodes possible

In addition to Ethernet or typical legacy interfaces OTN Systems offers the possibility to integrate customer-specific developments or rare standard interfaces into the system.

Example: OTN Systems N2021 node with a dedicated set of interfaces integrated in a 1HU 19” chassis


Figure 8-3 Customer-specific, purpose-built OTN node

8.4 Extended Ethernet architectures to come soon

OTN Systems already offers the possibility to accommodate up to 96 dedicated Ethernet interfaces in the biggest (core) node type. It is planned to extend OTN networks via rings and spokes into smaller sites by introducing more compact Ethernet-only nodes.

Figure 8-4 Ethernet extension by additional rings or spokes


This way, the capabilities and network architecture of OTN Networks are extended to collect and groom Ethernet traffic from remote sites and bring these services to the nearest OTN Node to be transported in a reliable way over long distances via the OTN Backbone.

All OTN Ethernet Access Devices implemented in these networks are managed through the OMS as well, greatly reducing the engineering, installation and commissioning effort as well as simplifying the life of the network operators.

9 Conclusions

OTN Systems develops technology that simply is “committed to get your information across”. Packet-heavy IP, Ethernet or every kind of legacy, or video – it doesn’t matter.

While the ports on Industrial networks are increasingly becoming Ethernet ports, legacy applications are still much around in the real world and will be so for a long period of time. A lot of industrial Ethernet applications need an embedded control of the provisioning, redundancy paths, traffic flows, Quality of Service and a deterministic behavior.

This is where OTN Systems comes in: it provides numerous Ethernet and legacy interfaces with the services defined to be carried, while being able to guarantee bandwidth and service availability on a per port basis at the same time, even for Ethernet traffic.

Using OTN Systems products, no matter if legacy-supporting or packet-heavy, offers customers the following:

• control over their networks

• easiness to work with dedicated services

• exclusion of attacks

• SIL 3 services availability

• No necessity of overlay video networks thanks to the built-in Video codecs - video can be analog, digital, IP or hybrid

• enough capacity while the core speeds of up to 10Gbps.

The IP-TDM converged Optical Industrial Network Solutions from OTN Systems ensure a future-proof investment for all customers who prefer to keep their network and applications under control.

_____________________________________________________ Want more information? Contact us via [email protected]

mailto:[email protected]

ip-tdm converged optical industrial networks_issue 1

Documents