8/3/2019 FG2 Reliability Sept13

1/27

Network Reliability and Interoperability Council

Focus Group 2

Network Reliability

PJ Aduskevicz, AT&T

Ross Callon, JuniperWayne Hall, Comcast

8/3/2019 FG2 Reliability Sept13

2/27

Network Reliability and Interoperability Council

Focus Group Membership PJ Aduskevicz, AT&T

Jeff Barrows, Firefly Networks

Ron Bath, VoiceStream

Dan Benyon, Cisco

Ken Biholar, Alcatel USA

Ken Buckley, Federal Reserve Board

Jane Builder, VoiceStream

Rex Bullinger, NCTA

Ross Callon, Juniper

Rick Canaday, AT&T

Kevin Cavanagh, AT&T Wireless

William Check, NCTA

Wayne Chiles, Verizon

Joe Craig, Qwest

Victor Devito, AT&T

Bernie Farrell, NCS

David Fears, Cox Communications

Perry Fergus, Booz Allen & Hamilton

Lee Fitzsimmons, Nextel Ron Garrett, Tekelec

Brian Goemmer, Western Wireless

Jeff Goldthorp, FCC

Chris Gunner, Avici Systems

Ed Hall, ATIS

Thomas Hall, MFN

Wayne Hall, Comcast

Rick Harrison, Telcordia

Paul Hart, SBC

Jake Hartinger, Cisco

John Healy, FCC

Dean Henderson, Nortel Networks

Bob Holley, Cisco

Bruce Johnson, Verisign

Lynn Johnson, Boeing

Rick Kemper, CTIA

Jay Kitchen, PCIA

Bill Klein, ATIS

Bernie Ku, WorldCom

Jim Lankford, SBC Bill LaPerch, MFN

8/3/2019 FG2 Reliability Sept13

3/27

Network Reliability and Interoperability Council

Focus Group Membership Greg Larson, Exodus/CWUSA

Mike Lecocke, SBC

Chris Liljenstolpe, CW

Virgil Long, Tekelec

Chris MacFarland, Allegiance

Spilios Makris, Telcordia

Joseph Maloney, Hughes

Archie McCain, BellSouth

Mike Meyers, Microsoft Corp.

Art Menko, BPI Telcodata

Brian Micene, AT&T Wireless

Denny Miller, Nortel

Kent Nilsson, FCC

Eamon OLeary, AT&T Wireless

Gary Pellegrino, CommFlow Resources

Bonnie Petti, Verizon Wireless

Sam Phillips, BITS

Tom Purcell, Nextel

Karl Rauscher, Lucent Arthur Reilly, Cisco Systems

Ira Richer, The Telesis Group

Gary Roboff, BITS

Jim Runyon, Lucent

Sonia Sakovich, Sprint

Harold Salters, Voicestream

Dan Schutzer, BITS

Andy Scott, NCTA

Pete Shelus, Telcordia Technologies

Ken Silva, Verisign

Don Smith, NCS

Ron Stear, C&W

Sandy Stephens, Focal

Jeff Swinton, Conexion Corp.

Whitey Thayer, FCC

John Todd, NCS

Rachel Torrence, Qwest

Robert Vogel, PanAmSat Corp.

Scott Wallhermfechtel, Metromedia

Doug Williams, Comcast Cable Heather Wyson, BITS

8/3/2019 FG2 Reliability Sept13

4/27

Network Reliability and Interoperability Council

Focus Group Mission Statement Define reliability measurements (units) for commercial communications networks

(i.e., wireline and wireless transport networks, including satellite and cable) and forthe Internet by March 22, 2003.

Define reasonable, measurable customer-affecting outage reporting thresholds forcommercial communications networks (i.e., wireline and wireless transportnetworks, including satellite and cable) and for the Internet by March 22, 2003.

Conduct voluntary outage reporting trial, collect data, analyze results, and reporton the validity, usefulness, and timeliness of the process and information obtained,and make recommendations for improvement.

Based on trial results (including information on services affected by an outage),evaluate and report on the reliability of public communications network services in

the United States. Should the Commission initiate an inquiry or rulemaking with respect to any of the

above-mentioned issues, the Focus Group will provide input to the NRIC, whichmay make formal recommendations as a part of such proceeding(s).

Evaluate, and report on, the reliability of public telecommunications networkservices in the United States.

8/3/2019 FG2 Reliability Sept13

5/27

Network Reliability and Interoperability Council

Focus Group Status

Initial work on measurements andthresholds in technology specific teams

Voluntary Outage Reporting Trial processand data criteria to be based on previouswork of NRIC V (modifications/adaptations

are being identified) Expected levels and conditions of

participation in the Voluntary OutageReporting Trial are being assessed

8/3/2019 FG2 Reliability Sept13

6/27

Network Reliability and Interoperability Council

Measurements and Thresholds TeamsTeam Leader

Internet Ross Callon (data) /

Dean Henderson (voice)

Wireless Dean Henderson

Cable Wayne Hall

Satellite Bob Vogel

Wireline Wayne Chiles /

PJ Aduskevicz

8/3/2019 FG2 Reliability Sept13

7/27

Network Reliability and Interoperability Council

NRIC V Process

Outage Occurs

Determine if

Outage Meets Vol.

Trial Criteria

Fax to NCS/NCC

On (703/607-4998 )

Create Initial Report

within 3 Days

Mark Company

Proprietary

ConductRoot Cause

Analysis

Identify

Best Practices

Fax Final Report

with in 30 D ays

NCS/NCC

Logs Report

Scrub Data per

Criteria Established

by 2.B1

Share Scrubbed

Data (i.e. NIPC)

Handoff scrubbed

data required by 2.A1 &

2.A2 (Best Practices

for Packet)

Does 2.B2 want

to use data?

Analyze Scrubbed

Data:

By Type/Segment

Impact

Frequency

Root cause for

Commonality

Provide status at

NRIC V Council

meetings

Make

Recommendations

Monitor Progress

Concur or Provide

Input on

Recommendations

Develop F inal

Report to Include

Recommendations

Service Provider NCS/NCCSubcommittee

2.B 1NRIC V

If yes

If noLocal Root

Cause

Analysis

Initial Final

Aggregate

And

Send Data

Initial Final

File Final Report

Determine

whether request to

share data should

be made

Take No

Further

Action

If yes

If no

Grant

Permission

to Share

Data?

If no

If yes

8/3/2019 FG2 Reliability Sept13

8/27

Network Reliability and Interoperability Council

NRIC V Data CriteriaFinal Report field Scrub Action

Reporting Carrier/Service Provider

Delete and create outage tag

(Industry segment identifier, i.e.,

CMRS, Satellite,Cable Telephony,

Data Services, and ISPs, plus a

unique numerical identifier)

Contact Person Delete

Start Date Unchanged

Start Time of Impact Unchanged

Geographic area affected Delete

Es timated number of cus tomers affec ted Unchanged

Types of Services Affected (i f appl icable) Unchanged

Durat ion of Outage (hours and minutes ) Unchanged

Apparent or known cause UnchangedName of Equipment Involved [OPTIONAL] Delete

Type of Equipment Involved [OPTIONAL] Delete

Specific Part of Network Involved Unchanged

Methods Used to Restore Service [OPTIONAL] Unchanged

Steps Taken to Prevent Recurrence Unchanged

Root Cause and Trouble Found [OPTIONAL] Unchanged

Applicable Best Practice [OPTIONAL] Unchanged

8/3/2019 FG2 Reliability Sept13

9/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC Reports

The Network Reliability Steering Committee (NRSC) wasestablished under the auspices of the Alliance for

Telecommunications Industry Solutions (ATIS) to monitornetwork reliability utilizing major outage reports filed withthe Federal Communications Commission (FCC) pursuantto Part 63.100 of the FCC Rules.

The NRSCs mission is to analyze network outage data

reported by companies, to identify trends, makerecommendations aimed at improving network reliability,and make the results publicly available, and whereapplicable refer matters to other industry fora for further

action.

8/3/2019 FG2 Reliability Sept13

10/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportAnnual Control Chart for Outage Frequency

157160

169

154

184181

185

174 176

120

140

160

180

200

220

1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofOutages

RED

YELLOW

GREEN

8/3/2019 FG2 Reliability Sept13

11/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportAnnual Aggregated Outage Index Control Chart

1580

1978

1566

1907

160816281653

13351330

1000

1200

1400

1600

1800

2000

2200

2400

1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

Ag

gregatedOutageIndex

RED

YELLOW

GREEN

8/3/2019 FG2 Reliability Sept13

12/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportNumber Of Outages By Failure Category

61

27

32

17

68

6

72

33

14 14

4 5

89

21 22

15 14

2

6

86

35

12 11

2

8

86

32

19 1917

8

4

76

1921 20

11

2

26

20

13

20

9

5

63

30

22

8

62

11

28

17

21

5

10

18

20

32

83

22

31

8

0

10

20

30

40

50

60

70

80

90

100

Facility Local Sw itch CCS Tandem Sw itch CO Pow er DCS Other

Num

berofOutages

2001

Baseline Years(1993 - 2000)

8/3/2019 FG2 Reliability Sept13

13/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportAnnual Frequency Control Chart for Procedural Error Outages

4447

63

49

767675

56

63

30

40

50

60

70

80

90

1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

Nu

mberofOutages

RED

YELLOW

GREEN

8/3/2019 FG2 Reliability Sept13

14/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportAnnual Aggregated Outage Index Control Chart for Procedural ErrorOutages

370

820

619

497

423423

275

420

741

0

100

200

300

400

500

600

700

800

900

1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

AggregatedOutageIndex

RED

YELLOW

GREEN

8/3/2019 FG2 Reliability Sept13

15/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual Report

For outages reported in 2001:

Local Switch outages had their lowest annual

frequency and aggregated outage index to date. The annual aggregated outage index of Procedural

Errors was the highest to date.

The number of outages was lower than in any year todate. Nonetheless, the aggregated outage reached itssecond highest value to date.

The average outage index per outage was the highest todate.

8/3/2019 FG2 Reliability Sept13

16/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2001 Annual ReportSignificant trends noted over the course of the nine-year

data history include:

Local switch outage annual frequency and aggregated outage

index are decreasing at the rate of 6% and 13% per yearrespectively.

Central Office (CO) Power outage frequency is increasing at a rateof 12% per year.

The frequency of Facility outages has decreased in the last two

years, and the Facility aggregated outage index is decreasing at arate of 6% per year.

The frequency of outages with a Procedural Error as a root causeis increasing at a rate of 6% per year.

The frequency of Common Channel Signaling (CCS) outages hasincreased in the last two years.

8/3/2019 FG2 Reliability Sept13

17/27

Network Reliability and Interoperability Council

NRSC Timing Outages Task GroupImpact of Timing Outages

2000-01 Total Outages

9%91%

Timing Outage

Other Outage

2000-01 CCS7 OUTAGES

67% 33%Other Outages - CCS7

Timing Outages - CCS7

33% of all CCS7

Outages were

Timing Outages

9.4% of all Outages

were from TimingOutages

8/3/2019 FG2 Reliability Sept13

18/27

Network Reliability and Interoperability Council

NRSC Timing Outages Task GroupThree New Best Practices

Network Operators and Service Providers should insure thatengineering, design, and installation processes address how newnetwork elements are integrated into the office synchronization

plan.

Network Operators and Service Providers should developmanagement and records keeping tools that accurately track thediversity of internal wiring for office synchronization, includingtiming leads and power.

Network Operators and Service Providers should conduct periodicverification of the office synchronization plan and the diversity oftiming links, power feeds and alarms.

8/3/2019 FG2 Reliability Sept13

19/27

Network Reliability and Interoperability Council

NRSC Timing Outages Task Group

Office Inspections and New Procedures

Upgrade all BITS clocks to models capable offull A/B Power redundancy

Verify that BITS is on fully protected power (UPS) with generator, and fedseparately (A/B)

IfD4 channel banks are used for transporting common channel signaling, there arespecial timing considerations:

Redundant SS7 links should be timed from redundant timing sources (e.g., from differentBITS timing output cards).

Typically, all D4 Shelves (e.g., six) can be daisy chained with same BITS clock lead. As such, theredundant SS7 Links should terminate on Bays or Shelves with different timing sources

Periodic tests for BITS switchover should be executed where applicable

Power (A/B)

Input (redundant Clock cards) Output (redundant Timing Output cards)

Alarms (e.g., power, input, output, fuse)

A one-time physical audit of timing redundancy, with special attention to SS7 linkdiversity should be conducted

Any outages, which are determined to have the BITS clock as a contributing cause;whether supplier/service provider/other attributable, should be shared with the

BITS clock supplier to assist that supplier in improving the quality of their product

8/3/2019 FG2 Reliability Sept13

20/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 ReportFCC Reportable Service Outages(by number of events)

38

54

3229

4041

48

38

35

4240

45

4240

47 48

4447

44

35

4852

48

44

4341

34

46 47

52 5552

43

38 39

2928

32

15

25

35

45

55

65

1Q93

3Q93

1Q94

3Q94

1Q95

3Q95

1Q96

3Q96

1Q97

3Q97

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

Quarter

NumberofOutages

8/3/2019 FG2 Reliability Sept13

21/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 ReportFCC Reportable Service Outages(by outage index)

430

265

318

622

299

618

427

470 555

364

305

481

345

414422

326223

519510

362

411

287

458

246

249

401

376376

440

331

412

300301 286

346

553

675

439

0

100

200

300

400

500

600

700

800

1Q93

3Q93

1Q94

3Q94

1Q95

3Q95

1Q96

3Q96

1Q97

3Q97

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

Quarter

OutageIndex

8/3/2019 FG2 Reliability Sept13

22/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 ReportIncidents by Failure Category(Common Channel Signaling)

8

5

7 7

3

2 2

5 5

2

3

2

5

14

5

12

1

5

44

5

4

77

66

7

10

14

88

9

77

8 10

0

2

4

6

8

10

12

14

16

1Q93

3Q93

1Q94

3Q94

1Q95

3Q95

1Q96

3Q96

1Q97

3Q97

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

Quarter

NumberofOutages

8/3/2019 FG2 Reliability Sept13

23/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 ReportIncidents by Failure Category(CO Power)

2

5

9

1

6

3

0

2

13

1

23

55

4

3

5 5

32

1

2

0

2

87 7 7

4 4

7 7 78

6

3

4

0

2

4

6

8

10

12

1Q93

3Q93

1Q94

3Q94

1Q95

3Q95

1Q96

3Q96

1Q97

3Q97

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

Quarter

NumberofOutage

s

8/3/2019 FG2 Reliability Sept13

24/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 ReportProcedural Error Attributed Outages(by number of events)

101111

14

2019

19

15

29

1617

11

15

18

25

16

1510

26

20

19

13

18

1213

1213

11

13

9

11 6

16

1413 13

9

20

0

5

10

15

20

25

30

35

1Q93

3Q93

1Q94

3Q94

1Q95

3Q95

1Q96

3Q96

1Q97

3Q97

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

Quarter

N

umberofOutages

8/3/2019 FG2 Reliability Sept13

25/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC 2Q02 Report

Based upon analysis of all outages reported from1Q93 through 2Q02, the NRSC notes that:

There is an increasing trend in the number ofCO Power and CCS outages.

There is an increasing trend in the frequency

and aggregated outage index of ProceduralErrors as the root cause of outages.

There is a decreasing trend in the number ofLocal Switch outages.

8/3/2019 FG2 Reliability Sept13

26/27

Network Reliability and Interoperability Council

Reliability ReportingNRSC Reports

The NRSC urges all service providers

and equipment vendors to review allbest practices for application in their

operations.

These Best Practices may be found at:http://www.nric.org/

8/3/2019 FG2 Reliability Sept13

27/27

Network Reliability and Interoperability Council