four-fold increase in productivity and quality

2001-03-19 ETX/D/XPS-01:002 Uen 1

Ulf Wiger <[email protected]>Senior System ArchitectStrategic Product & System ManagementEricsson Telecom ABData Backbone & Optical Networks Division

Four-Fold Increase in Productivity and Quality

—Industrial-Strength Functional Programmingin Telecom-Class Products(http://www.erlang.se/publications/Ulf_Wiger.pdf)

FEmSys 2001Deployment on distributed architectures

2001-03-19 ETX/D/XPS-01:002 Uen 2

The setting

1995 PC Week Study of software projects:– 16% successful

– 53% operational (but less than successful)

– 31% cancelled

Butler Group 1997 on large software projects– 5 out of 6 large projects fail

– In >$300M companies, 9 out of 10 large projects fail

How to approach this?– Use high-level modeling tools & generate code?

– Raise the level of programming language?

– Fight all causes for project failure!

2001-03-19 ETX/D/XPS-01:002 Uen 3

The design of the AXD 301 switch

Situation 1996

– Late entry into the ATM market

– Needed to leapfrog the competition

– Project members used to "traditional" large projects,wanted to try something new

– No interesting 3rd party middleware platform

Target: a backbone ATM switch with "Telecom Profile"

2001-03-19 ETX/D/XPS-01:002 Uen 4

Executive summary

AXD 301: A Telephony-Class, scalable (10-160 GBps) ATM switchdesigned from scratch in less than 3 years

AXD 301 Success factors:– Highly pragmatic, holistic approach

– Competent organisation

– Efficient process

– Excellent technology (e.g. Erlang/OTP)

More than just technology...– Consider all factors together from the start

– Erlang was a perfect match for our approach

2001-03-19 ETX/D/XPS-01:002 Uen 5

ENGINE:Migrating today's vertical networksinto a single multi-service backbone

AXD 301 in the marketplace

Central component in Ericsson's ENGINE offering

Several major operators– British Telecom– Vodaphone– Worldcom– Telia– Diveo– ...

2001-03-19 ETX/D/XPS-01:002 Uen 6

Briefly about the term Carrier-Class

To us, "Carrier-Class", "Telephony-Class" and"Telecom Profile" are synonymous

The quality we've come to expectfrom public telephony networks

The trend towards multimedia servicesrequires Carrier-Class in more systems

More than just duplication of hardware:– Fault-tolerant software– In-service hardware expansion– In-service software upgrade– Load tolerance– Flexibility (frequent changes + long service life)

Target: 99,999% ("five nines") availability, including planned outages

There's no such thingas "almost Carrier-Class"!

2001-03-19 ETX/D/XPS-01:002 Uen 7

Optional Processors

Mandatory Mated Processor Pair

ATM

Control Processors

L3F

CE

Switch

Core

FR

ATB

ATB

ATB

ATB

Server Device

LineTermination

ATM Termination

Clock &Synchronization

Clock &Synchronization

IO

IO

IO

IOCP

CP

CP

CP

Device Processoron Each Board

Telecom-Class System Architecturesimple wire-speed logic complex soft-real-time logic

2001-03-19 ETX/D/XPS-01:002 Uen 8

Fault tolerance: distributed recovery

Failover: one processor dies; its mate restarts the lost applications

Takeover: the failed processor returns; the O&M/Standbyapplications migrate

O&M or Standby application

Call-handling application

"Luke-warm" standbysufficient to

maintain quality of service

2001-03-19 ETX/D/XPS-01:002 Uen 9

Call Troughput in % of maximum sustainable call handling (set-up + release) capacity

Offered Loadin % of maximumcall troughput

100%

40%

100% 150% 1000%

RejectedCalls

95%

115 call/s sustained200 call set-up/s or

Load from handlingof rejected calls

Load tolerance

Call Handling Throughput for one control processor

AXD 301 release 3.2Traffic Case: ATM SVC UNI to UNI

2001-03-19 ETX/D/XPS-01:002 Uen 10

History of Erlang

1984:EricssonComputerScience Labformed

1984-86:ExperimentsprogrammingPOTS with several languages

1998:Open SourceErlang

1987:Early ErlangPrototype projects

1991:First fastimplementation

1993:DistributedErlang

1995:Severalnew projects

1996:Open Telecom Platform(research on verification...)

No language well suitedfor telecom systems

development

2001-03-19 ETX/D/XPS-01:002 Uen 11

Erlang Highlights

DeclarativeConcurrencySoft real-timeRobustnessDistributionHot code loadingExternal interfacesPortability

Functional programming languageHigh abstraction level

Pattern matchingConcise readable programs

2001-03-19 ETX/D/XPS-01:002 Uen 12

Erlang Highlights


Either transparent or explicit concurrency

Light-weight processesHighly scalable

2001-03-19 ETX/D/XPS-01:002 Uen 13

Erlang Highlights


Response times in theorder of milliseconds

Per-process garbage collection

2001-03-19 ETX/D/XPS-01:002 Uen 14

Erlang Highlights


Simple and consistenterror recovery

Supervision hierarchies"Program for the correct case"

2001-03-19 ETX/D/XPS-01:002 Uen 15

Erlang Highlights


Explicit or transparent distributionNetwork-aware runtime system

2001-03-19 ETX/D/XPS-01:002 Uen 16

Erlang Highlights


Easily change code in a running system

Enables non-stop operationSimplifies testing

2001-03-19 ETX/D/XPS-01:002 Uen 17

Erlang Highlights


"Ports" to the outside worldbehave as Erlang processes

2001-03-19 ETX/D/XPS-01:002 Uen 18

Erlang Highlights


Erlang runs on any UNIX,Windows, VxWorks, ...Supports heterogeneous

networks

2001-03-19 ETX/D/XPS-01:002 Uen 19

Erlang Examples 1

Basics - Factorial function

n! =1

n*(n-1)!

n = 0

n 1

Definition

-module(ex1).-export([factorial/1]).

factorial(0) -> 1;factorial(N) when N >= 1 -> N * factorial(N-1).

Implementation

Eshell V5.0.1 (abort with ^G)1> c(ex1).{ok,ex1}2> ex1:factorial(6).720

2001-03-19 ETX/D/XPS-01:002 Uen 20

Erlang Examples 2

A few very high-level constructs - QuickSort

-module(ex2).-export([qsort/1]).

qsort([Head|Tail]) -> First = qsort([X || X <- Tail, X =< Head]), Last = qsort([Y || Y <- Tail, Y > Head]), First ++ [Head|Last];qsort([]) -> [].

Eshell V5.0.1 (abort with ^G)1> c(ex2). {ok,ex2}2> ex2:qsort([7,5,3,8,1]).[1,3,5,7,8]

"all objects Y taken from the list Tail,

where Y > Head"

2001-03-19 ETX/D/XPS-01:002 Uen 21

Erlang Examples 3

Concurrency - Simple timer

-module(ex3).-export([send_after/3]).

send_after(Time, Receiver, Message) -> spawn_link(fun() -> receive after Time -> Receiver ! Message end end).

Eshell V5.0.1 (abort with ^G)1> c(ex3). {ok,ex3}2> ex3:send_after(1000, self(), hello).<0.42.0>3> flush().Shell got hellook

Anonymous functionpassed as input

argument

2001-03-19 ETX/D/XPS-01:002 Uen 22

Erlang Examples 4

Concurrency - Finite State Machine

ringing_a(A, B) -> receive {A, on_hook} -> back_to_idle(A, B); {B, answered} -> A ! {stop_tone, ring}, switch ! {connect, A, B}, conversation_a(A, B) after 30000 -> back_to_idle(A, B)end.

back_to_idle(A, B) -> A ! {stop_tone, ring}, B ! terminate, idle(A).

Selective receive

Asynchronous send

Optional timeout

2001-03-19 ETX/D/XPS-01:002 Uen 23

Erlang Examples 5

1

P1 PnP2 ...

One for one supervision:If any child dies it is restarted

a

P1 PnP2 ...

One for all supervision:If any child dies, all childrenare terminated and all arerestarted

1

1

a 1

a

Supervision tree:Process links+ special supervisor processes

The infrastructurehandles recovery

- program for the correct case"let it crash!"

2001-03-19 ETX/D/XPS-01:002 Uen 24

Erlang/OTP

Open Telecom PlatformMiddleware for Erlang developmentDesigned for fault tolerance and portabilityBehaviors: A formalization of design patternsComponents

– Error handling, reporting and logging

– Mnesia, distributed real-time database management system

– CORBA

– IDL Compiler, Java & C Interface Support

– HTTP Server

– SNMP Agent

– ...

2001-03-19 ETX/D/XPS-01:002 Uen 25

OTP Behaviors

"A formalization of design patterns"– A behavior is a framework

+ generic code to solve a common problem

– Each behavior has built-in support fordebugging and software upgrade

– Makes it easier to reason about the behavior of a program

Examples of OTP behaviors– application defines how an application is implemented

– supervisor used to write fault-tolerant supervision trees

– gen_server for writing client-server applications

– gen_event for writing event handlers

– gen_fsm for finite state machine programming

2001-03-19 ETX/D/XPS-01:002 Uen 26

Project Highlights

Early phases under one roof

Powerful architecture team

Clear chain of command

Rapid decisions (within days)

Organize according to product structure

2001-03-19 ETX/D/XPS-01:002 Uen 27

Product Council

Techn Mtg Subsyst

Block Responsible

SystemSystemManagementManagement

Techn Mtg AXD Proj Mgmt Mtg

Subsyst Proj MgrProfilingProfiling

Task ForceTask Force

Project Structure

Different forums for technical and project Issues

– But partly the same people

System Management = “glue”– “Owners” of the architecture– Keeps a system perspective– Writes design rules– Holds seminars– Profiling Task Force– Acts as Designer Help Desk

Human Interaction EssentialProximity = “Under One Roof”

2001-03-19 ETX/D/XPS-01:002 Uen 28

Process highlights

"Lightweight" development process

Careful functional decomposition

Formal change management of interfaces

Incremental design

Prototyping of critical parts

Only essential documents

2001-03-19 ETX/D/XPS-01:002 Uen 29

ATS

Basic ConnectionControl Support

ATM resourceManagement

PVC Control

ATM Signallingand call control

LSS FRS AVS

LabelSwitchRouting

FrameRelay

DynamicTrunking

AHS

NGSNarrowbandServicesforAXE

SWS OMS CPS

Switch Controland Maintenance

OAMsupport

OperatingSystem

Base Services Add-on ServicesAdd-on Services

Not released as part of AXD 301

Tools

Project

Process

Identify Subsystems

2001-03-19 ETX/D/XPS-01:002 Uen 30

PROXY UNI ILMI

IISP AINI PNNI

BICI SPVC

CHG ANA

SAAL REH

CAC

TRH

DTC VPNALIM OSPF RIP

IPS FIM

FTM

SCBDP CLKDP ATBDP ET622DP ET155DP

ET45DP ET34DP ABRDP ET2CEDP DPG

DPOS

RCM

CPC

ATS AVS

SWS CPS OMS

LSS

PNNIR

ATSD AVSD

LDP

LSSD

GSMP

CNH PRM SWSD

DPC NSY PRSW

ABR DNA CECP

EVA WMS

PERF STOEQM

DPSLMS OMSD

SWM

SYS

CPM MCNC

OTP

CSS CPO

CPSD

Tools

Project

Process

Identify Block StructureReal-time critical

Non-real-time critical

2001-03-19 ETX/D/XPS-01:002 Uen 32

Documentation

Documentation must evolve with the incremental design

Focus on a few vital document types, e.g.:– Requirement Specification

– Implementation Proposals

– Function Specifications

– Block Descriptions

– Interface Descriptions

Highly readable programs reduce the need for documentation(...but do not eliminate it)

2001-03-19 ETX/D/XPS-01:002 Uen 33

Programming languages (control system)

Erlang: ca 1 million lines of code– Nearly all the complex control logic

– Operation & Maintenance

– Web server and runtime HTML/JavaScript generation

C/C++: ca 500k lines of code– Third party software

– Low-level protocol drivers

– Device drivers

Java: ca 13k lines of code– Operator GUI applets

2001-03-19 ETX/D/XPS-01:002 Uen 34

Experiences from AXD 301 SW Design

Using Erlang in Complex Systems– Fits very well with the incremental design method

– High programmer satisfaction

– Outstanding support for robustness and concurrency

– Very few side-effects easier to add/change single components

– Small directed teams can achieve impressive results

Productivity estimates– Similar line/hour programmer productivity

– 4-10 fewer lines of source code (compared to C/C++, Java, PLEX) 4-10x higher programmer productivity

– Similar number of faults per 1000 lines of source code 4-10x higher quality

four-fold increase in productivity and quality

Documents