huawei optical network maintenance reference-wdm ason-20140826-c
TRANSCRIPT
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
1/116
Huawei Optical NetworkMaintenance Reference - WDMASON
Issue 02
Date 2014-08-26
Huawei Technologies Co., Ltd.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
2/116
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
i
Huawei Technologies Co., Ltd. provides customers with comprehensive technical support and service.
Contact our local office or company headquarters.
Huawei Technologies Co., Ltd.
Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: http://www.huawei.com
Email: [email protected]
Copyright © Huawei Technologies Co., Ltd. 2011. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior
written consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, expressed or implied.
Acknowledgement
This document is prepared and reviewed by the ASON R&D Maintenance Team, Information Development
Dept, Customer Support Dept, and Technical Support Dept together.
Editor:
Feng Junjie, Zhu Fei, Feng Haoyu, Wang Chaokai, Feng Chao, Liu Yuan, Zhang Meng, Zhou Yuxing,Zheng Fan, Li Weiping
Others:
ASON R&D Maintenance Team: Jiang Yi, Bai Zhongqiang, Li Qingsong
Information Development Dept: Fan Xiaoke, Pei Xin
Technical Support Dept: Dou Yongtan, Xie Bing
Customer Support Dept: Zhang Junguang, Fu Ming, Ma Qingquan
Quality Assurance Dept: Xue Xiuhua
Special acknowledgements to Jin Yuzhi, Mu Jianhong, Feng Zhigang, Wu Gang, Niu Shouchang, and
Chen Bin
http://www.huawei.com/http://www.huawei.com/mailto:[email protected]:[email protected]:[email protected]://www.huawei.com/
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
3/116
Huawei Optical Network Maintenance Reference -
WDM ASON About this Document
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
ii
About this Document
Overview This document mainly covers four aspects: preventive maintenance, quick service recovery
guide, fault diagnosis, and maintenance cases. It consists of the following chapters:
SOP for Maintaining NG WDM ASON DevicesThis chapter provides the inspection items and operation methods for preventivemaintenance of NG WDM ASON devices. The standard operating procedure (SOP)helps maintenance personnel discover and eliminate network risks during preventive
maintenance, therefore ensuring network stability and security.
Quick Recovery Guide for NG WDM ASON Services
This chapter provides handling measures for quickly recovering ASON services on NGWDM devices.
Fault Diagnosis for ASON Service Interruptions on NG WDM Devices
This chapter provides major methods and measures for diagnosing ASON service faults
on NG WDM devices.
Typical ASON Service Troubleshooting Cases
This chapter provides the typical ASON service troubleshooting cases for reference toguide maintenance personnel through troubleshooting of NG WDM devices.
FAQs
This chapter provides frequently asked questions about NG WDM ASON operation and
maintenance.
Change History
Issue Date Description
02 2011-08-26 Added “Precautions” section in “1 SOP for Maintaining NG
WDM ASON Devices”.
01 2011-09-29 This issue is the first official release.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
4/116
Huawei Optical Network Maintenance Reference -
WDM ASON Contents
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
iii
Contents
About this Document ..................................................................................................................... ii
1 SOP for Maintaining NG WDM ASON Devices .................................................................... 1
1.1 Introduction to SOP for ASON Maintenance ........................................................... ........................................ 2
1.2 ASON Device SOP Checklist ............................................................. ............................................................ 12
2 Quick Recovery Guide for NG WDM ASON Services ....................................................... 13
2.1 Overview ............................................................ .............................................................. .............................. 13
2.2 ASON Service Recovery Process ....................................................... ............................................................ 13
2.2.1 Intended Audience ................................................................................................................................ 13
2.2.2 Standard Operations for Routine Maintenance ............................................ ......................................... 13
2.2.3 Classification of Fault Causes .............................................................. ................................................. 14
2.2.4 Rerouting and Non-rerouting Conditions ........................................................ ...................................... 14
2.2.5 ASON Alarms ........................................................................... ............................................................ 15
2.2.6 General Principles for Recovering ASON Services .............................................................................. 16
2.2.7 Key Information for Recovering ASON Services .............................................................. ................... 16
2.2.8 General Flowchart for ASON Service Recovery ................................ .................................................. 17
2.3 AQuick Recovery Process for ASON Services ...................................................................................... ........ 17
2.3.1 Quick Recovery Process for ASON Services (Optical-Layer Service) ................................................. 17
2.3.2 Quick Recovery Process for ASON Services (OTN Electrical-Layer Service) .................................... 20
2.3.3 Quick Recovery Process for ASON Services (VC-4 Services) ..................................................... ........ 24
2.4 Troubleshooting Processes for Typical ASON Service Faults ..................................................... ................... 25
2.4.1 Troubleshooting Process for Route Computation Failures ............................................................ ........ 25
2.4.2 Troubleshooting Process for Rerouting Failures ................................................................................... 27
2.4.3 Troubleshooting Process for Unreachable NEs ....................................................... .............................. 28
2.4.4 Troubleshooting Process for Interruption of a Single-wavelength length ASON Service .................... 30
2.4.5 Troubleshooting Process for Interruption of Multi-Wavelength Services ............................................. 31
2.4.6 Troubleshooting Process for Malfunctioning SCC Boards ..................................... .............................. 33
2.4.7 Troubleshooting Process for a Failure to Recover Interrupted ASON Services Because of Add/Drop
Channel Faults ............................................................................................................................................... 34
2.5 Appendix: ASON Node Troubleshooting Process ............................................................ .............................. 35
2.5.1 General Principles for Handling ASON Node Faults ......................................................... ................... 35
2.5.2 Fault Description ............................................................ ............................................................... ........ 35
2.5.3 Procedure for Restoring Databases Using the Real-Time Database Backup Function ......................... 35
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
5/116
Huawei Optical Network Maintenance Reference -
WDM ASON Contents
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
iv
2.5.4 Restoring the Configurations Manually .................................................................. .............................. 38
2.5.5 Restoration Process If No Backup Database Is Available ..................................................................... 40
3 Fault Diagnosis for ASON Service Interruptions on NG WDM Devices ....................... 41
3.1 General Fault Diagnosing Procedure ................................................. ............................................................ 41
3.1.1 Overview .............................................................. .............................................................. ................... 41
3.1.2 Flowchart .............................................................................................................................................. 42
3.1.3 Troubleshooting Procedure ........................................................................................................... ........ 42
3.1.4 Exception Handling Process ................................................................................................................. 46
3.2 Diagnosing the Fault that a Service Is Unavailable After Being Successfully Rerouted ............................... 46
3.2.1 Overview .............................................................. .............................................................. ................... 46
3.2.2 Flowchart .............................................................................................................................................. 46
3.2.3 Troubleshooting Procedure ........................................................................................................... ........ 47
3.2.4 Exception Handling Process ................................................................................................................. 48
3.3 Diagnosing the Fault that Service Rerouting Fails ........................................................... .............................. 48
3.3.1 Overview .............................................................. .............................................................. ................... 48
3.3.2 Flowchart .............................................................................................................................................. 49
3.3.3 Troubleshooting Procedure ........................................................................................................... ........ 49
3.3.4 Exception Handling Process ................................................................................................................. 50
3.4 Diagnosing the Fault that a Service Is Not Rerouted ................................. .................................................... 50
3.4.1 Overview .............................................................. .............................................................. ................... 50
3.4.2 Flowchart .............................................................................................................................................. 51
3.4.3 Troubleshooting Procedure ........................................................................................................... ........ 51
3.4.4 Exception Handling Process ................................................................................................................. 52
3.5 Common Operations Involved in Fault Diagnosis ......................................................................................... 52
3.5.1 Diagnosing OPA Adjust Failures .......................................................... ................................................. 52
3.5.2 Diagnosing Incorrect Attenuation Delivery of OPA ............................................................................. 54
3.5.3 Confirming Information About a Faulty Service ........... ............................................................... ........ 55
3.5.4 Obtaining Performance Data ................................................................ ................................................. 57
3.6 Information to Be Collected ........................................................................................................ ................... 57
3.7 References .......................................................... .............................................................. .............................. 58
4 Typical ASON Service Troubleshooting Cases .................................................................... 59
4.1 ASON-Specific Operations and Configurations ................................................................ ............................. 59
4.1.1 Case 1: Creating an ASON Service Fails Because the Wavelength for the Service Is Reserved .......... 59
4.1.2 Case 2: ASON NEs Reset Because of a Duplicated OSPF IP Address ................................................. 61
4.1.3 Case 3: An Attempt to Start the TE Link Management Window Times Out ......................................... 61
4.1.4 Case 4: Service Protection Level Fails to Change After a Line Board Participating in the Protection Is
Moved to Another Slot ............................................................ ............................................................... ........ 63
4.1.5 Case 5: Deleting Inter-NE Fiber Connections Fails When Separate Optical and Electrical NEs Are
Configured ..................................................................................................................................................... 65
4.1.6 Case 6: Route Computing Fails During Creation, Optimization, or Rerouting of an ASON Service or
During an Upgrade of a Static Service to an ASON Service ........................ ................................................. 66
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
6/116
Huawei Optical Network Maintenance Reference -
WDM ASON Contents
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
v
4.1.7 Case 7: A Newly Created ASON Service Fails to Traverse a Node or an Existing ASON Service Fails
to Traverse a Node During Trail Optimization or Rerouting ........................ ................................................. 68
4.1.8 Case 8: LMP Protocol Check Fails Due to DCN Errors and Consequently Service Deployment Fails 69
4.1.9 Case 9: ASON Services Fail to Be Deployed Because Line Attenuation Is Excessively High ............. 71
4.1.10 Case 10: An Error Message Is Displayed When Users Attempt to Create Virtual TE Links .... .......... 72
4.1.11 Case 11: Route Computation Fails After Explicit Resources Are Specified to Create or Optimize an
ASON Service ............................................................... .............................................................. ................... 75
4.2 ASON Service Restoration .......................................................................................................... ................... 77
4.2.1 Case 1: An ASON Service Is Interrupted Because OPA Fails ....................................................... ........ 77
4.2.2 Case 2: An ASON Service Is Interrupted Because Protection Switching Fails After a Second Fiber Cut
....................................................................................................................................................................... 78
4.2.3 Case 3: An ASON Service Is Interrupted After Being Rerouted Because of Incorrect Fiber Connections
....................................................................................................................................................................... 79
4.2.4 Case 4: An ASON OCh Trail in a Slave Subrack Is Interrupted but Not Rerouted After the Slave
Subrack Is Powered Off ................................................................................................................................. 81
4.2.5 Case 5: An ASON Service Is Frequently Rerouted Among Multiple Trails.......................................... 82
4.2.6 Case 6: An Optical ASON Service Fails to Be Automatically Restored in Case of a Wavelength-Level
Fault ............................................................................................................................................................... 83
4.2.7 Case 7: An ASON Service Enabled with Scheduled Reversion Fails to Be Reverted to Its Original
Trail After the Scheduled Reversion Time Elapses ........................................................................................ 84
4.2.8 Case 8: Route Computation Fails When an ASON OCh Service Traverses a Regeneration Board ...... 84
4.2.9 Case 9: An ASON OCh Service Is Interrupted but Not Rerouted ......................................................... 86
5 FAQs .............................................................................................................................................. 90
5.1 Operations on the NMS ........................................................... ............................................................... ........ 90
5.1.1 How to Distinguish Between ASON Services and Traditional Services on the NMS? ......................... 90
5.1.2 How to Identify the First and Last Nodes of an ASON Service? .......................................................... 91
5.1.3 How to Identify the Original Trail and Preset Restoration Trail of an ASON Service? ........................ 92
5.1.4 How to Manually Optimize an ASON Service? ...................................................... .............................. 93
5.1.5 How to Manually Optimize ASON Services in Batches? ....................................... .............................. 94
5.1.6 How to Change a Wavelength to Optimize Trails? ..................................................................... .......... 95
5.1.7 How to Obtain Fiber Connections of Boards that an ASON Service Traverses? .................................. 95
5.1.8 How to Quickly Locate the Board Where an OCH_SER_INT Alarm Is Generated? ........................... 95
5.1.9 How to Quickly Query the Current Trail or Preset Restoration Trail of an ASON Service that Traverses
a Specific NE or Board? ................................................................................................................................ 96
5.1.10 How to Check Whether a Preset Restoration Trail Is Available? ........................................................ 96
5.1.11 How to Check Whether an Optical Cross-Connection Is Successfully Created for an ASON Service?
....................................................................................................................................................................... 96
5.1.12 How to Quickly Restore an Interrupted Service? ............................................................. ................... 96
5.1.13 How to Quickly Create Fiber Connections Between Sites? ........................................................ ........ 97
5.2 Configuration Rules ....................................................................................................................................... 97
5.2.1 What Are Common Attributes and Recommended Configurations for ASON Services? ..................... 97
5.2.2 What Are the Risks if ODU0, ODU1, and ODU2 ASON Services Are Concurrently Configured? ..... 97
5.2.3 What Are the Basic Rules for Configuring Preset Restoration Trails? .................................................. 97
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
7/116
Huawei Optical Network Maintenance Reference -
WDM ASON Contents
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
vi
5.2.4 What Are the Recommended Configurations for the Preset Restoration Trails and Revertive Attributes
of SDH/OTN/WDM ASON Services? .......................................................... ................................................. 98
5.2.5 What Are the Restrictions on Regeneration Boards When Optical NEs and Electrical NEs Are
Separated and Why?....................................................................................................................................... 98
5.2.6 How to Ensure the Rerouting Function When No OA Board Is Configured Between an FIU Board anda WSS Board? ............................................................... .............................................................. ................... 99
5.2.7 Why Are TN52SCC Boards Instead of TN51SCC or TN11SCC Boards Recommended for ASON NEs?
....................................................................................................................................................................... 99
5.2.8 What Are the Rules for Configuring Node IDs for ASON NEs? .......................................................... 99
5.2.9 Why Must the Node ID and IP Address of an NE Be in Different Network Segments? ..................... 100
5.2.10 Why Does the LMP or OSPF Protocol Need to Be Disabled on Electrical Links on OTU Boards
Adding or Dropping WDM ASON Services? .............................................................................................. 100
5.2.11 What Are the Application Scenarios and Configuration Method for Resource Reservation? ........... 102
5.2.12 Why Does the LMP Protocol Need to Be Disabled for Optical Ports on Tributary Boards? ............ 103
5.2.13 How to Disable the LMP Protocol for the Optical Ports that Are Not Used by ASON Services? .... 103
5.2.14 How to Disable the OSPF Protocol for the Optical Ports that Are Not Used by ASON Services? ... 103
5.2.15 How to Split a Large DCN Subnet into Smaller DCN Subnets? ....................................................... 104
5.2.16 Do Diamond, Gold, Silver, and Copper ASON Services Support Hitless Conversion? ................... 105
5.3 ASON Principles ........................................................... .............................................................. ................. 105
5.3.1 What Overheads Are Used by Control Channels on the Control Plane? ............................................. 105
5.3.2 What Is the Difference Between the Menu Items "Revert To Port" and "Revert to Channel"? .......... 105
5.3.3 What Is the Difference Between Trail Overlap and Trail Sharing? ..................................................... 106
5.3.4 What Is Associated Sharing? ................................................................ ............................................... 106
5.3.5 What Is the Relationship Between SRLGs and Associated Services? .......................................... ...... 106
5.3.6 Why Is a Revertive Service Reverted to the Original Trail 5 Minutes After Rerouting and How to
Revert the Service to the Original Trail Within 5 Minutes? ........................................................ ................. 106
5.3.7 Does an OPA Adjust Failure Affect Rerouting of ASON Services? .................................................... 106
5.3.8 Why Cannot Revertive Services Be Downgraded After Rerouting? ................................................... 107
5.3.9 What Is the Difference Between the Function of Downgrading ASON Services in an NE Explorer and
the Function of Downgrading ASON Services in the Trail Management Window? .................................... 107
5.3.10 Do Service Optimization Must Be Performed at the First Node? ..................................................... 107
5.3.11 Why Do Services Fail to Be Reverted to the Original Trail? ............................................................ 107
5.3.12 Why Does Synchronization Between the NE and NMS Need to Be Performed During Each Query of
ASON Service Information? .................................................................................................................. ...... 108
5.3.13 Does a CPW_XXX_INT Alarm on the Control Plane Mean Service Interruption? ......................... 108
5.3.14 What Is the Difference in Database Backup and Restoration Between an ASON Network and a
Non-ASON Network? ............................................................. ............................................................... ...... 108
5.3.15 Why Do I Need to Periodically Check Whether Rerouted ASON Services Are Reverted to the
Original Trails? ............................................................................................................................................ 108
5.3.16 What Are Residual Cross-Connections and How to Delete Residual Cross-Connections? .............. 108
5.3.17 What Do the CPW_OCH_SER_INT and CPW_ODUk_SER_INT Alarms Mean?.......................... 109
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
8/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
1
1 SOP for Maintaining NG WDM ASONDevices
Precautions
ASON networks must be appropriately planned and designed so that the network isrobust and services can survive against multiple fiber cuts. For the detailed process, seeFigure 1-1.
The expansion of links, subracks, wavelengths, or ODUk channels of an ASON networkmust be planned, designed, and deployed according to the related creation procedure (seeFigure 1-1). Otherwise, the robustness of new services cannot be ensured, and even thesecurity of the existing services may be affected.
Normal ASON network running and service security depend on meticulous routine
maintenance and periodical comprehensive assessment and optimization.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
9/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
2
Figure 1-1 ASON network delivery and maintenance process
Purpose
This chapter provides the SOP for preventive maintenance personnel so that they can discover
and eliminate network risks, thereby ensuring network stability and security.
Intended Audience
System maintenance personnel
ApplicationDevice maintenance personnel perform the standard operations and activities for preventive
maintenance at suggested intervals.
1.1 Introduction to SOP for ASON Maintenance
Table 1-1 lists the SOP for preventive maintenance of ASON.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
10/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
3
Table 1-1 SOP for preventive maintenance of ASON
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
ASON
databases
Checking
backupdatabasesfor ASON
NEs
5 min/NE NE
databasesmust be backed up
once
networkchanges
occur.
Manual 1. Verify that
the NMS cansuccessfully back up NE
databases at
specifiedintervals.
It is
recommendedthat the NMS
start backingup databasesfor
network-wide
NEs at 2 a.m.every day.
2. After a
networkchange
occurs, for
example,many services
are deployedor reroutingoccurs,
request thecustomer to
arrange a time
window formanually
backing updebases for all
ASON NEs
on thenetwork.
Minor To ensure
that userscandownload
the NE
database torecover the
node once a
node faultoccurs onthe
network.
ASON
resources
Checking
the status
of control
links
5
min/100
links
Monthly Manual 1. On the
NMS client,
navigate to
the ASONcontrol link
managementwindow andsynchronize
the controllink
information
network-wide.
Major To ensure
that control
links are in
normalstate. If the
control linktopology isincorrect,
you maynot be able
to create,
optimize,
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
11/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
4
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
2. Checkwhether
isolated NEsare present in
the controllink topology
view. If there
are isolated NEs, pinpoint
the cause andrestore the NEs to
normal state.In addition,
check for
alarms on theisolated NEs
and clearthem one by
one.
3. Check
whetherabnormal
alarms are
generated onthe controllinks. If thereare abnormal
alarms, locate
the boardsthat report the
alarms andclear thealarms one by
one.
4. Export the
control linkinformationinto an excel
file. Comparethe controllink
informationwith this
information
next time you perform the preventive
maintenance,
and check
delete, orreroute an
ASONservice.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
12/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
5
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
whether thecontrol links
are the samein the two
inspections.
Checkingvirtual TE
links
2 min/onelink
Monthly Manual 1. On the NMS client,
navigate tothe ASON TE
linkmanagement
window and
synchronizethe TE linkinformationnetwork-wide.
2. Check the
value of
Extend Type of TE links. Ifthe value isnot
Automaticall
y Discovered,check for
alarms on the
source andsink boards ofthe TE links.If alarms have
been
generated onthe source and
sink boards,
clear thealarms
according tothe NMS
online help.
Major To ensurethat virtual
TE links areavailable.
Checking
status of
TE links
5
min/100
links
Monthly Manual 1. On the
NMS client,
navigate tothe ASON TE
link
managementwindow andsynchronizethe TE link
information
Major To ensure
that TE
links are up.If alarms
are
generatedon TE links,recovery ofASON
services
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
13/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
6
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
network-wide.2. Check
whetherabnormal
alarms are
generated onthe TE links.
If there are
abnormalalarms, locate
the boardsthat report the
alarms andclear thealarms one by
one.
3. Check thevalue of Link
Status of the
TE links. Ifthe value is
not Up, check
for alarms onthe boards
where the TElinks aredown. Then
clear thealarms.
4. Export the
TE linkinformationinto an excel
file. Comparethe control
link
informationwith this
informationnext time you perform the
preventivemaintenance,
and checkwhether the
TE links are
the same inthe two
inspections.
will beaffected.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
14/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
7
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
Checkingfor
residualcross-conn
ections
5min/onecross-con
nection
Monthly Manual Checkwhether a
CPW_XXXX _TEL_PATH
MIS alarm is
generatedthroughout
the network.
If there is aCPW_XXXX
_TEL_PATHMIS alarm,
clear itaccording tothe NMS
online help.
Major To ensurethat there
are noresidual
cross-conne
ctions. TheCPW_XXX
X_TEL_PA
THMISalarm may
affectrecovery of
ASONservices.
ASON
services
Checking
ASON
services
10 min
/network
Daily Manual 1. On the
NMS client,
navigate tothe ASONtrailmanagement
window; then
synchronizeASON trail
information.
2. Check
whetherASON trails
are activated.If an ASON
trail is
displayed asInactive,
check whethera clientservice is sent
to the ASONtrail. If no
client service
is sent to theASON trail,
no furtheraction is
required. If a
client serviceis sent to the
ASON trail,
contact the
Critical To ensure
that ASON
services arenormal.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
15/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
8
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
customer forfurther
confirmationand take
records of theconfirmation
result.
3. Check
whetherrerouting
lockout isdisabled for
ASON trails.If it isdisabled,
contact thecustomer to
check why itis disabledand take
records.
4. Check
whetheralarms are
generated onthe ASONtrails. If yes,
clear themaccording to
the NMSonline help.
5. Export theASON
informationinto an Excel
file and save
it for futurereference.
Alarms Clearing
control
planealarms
5 min/one
alarmMonthly Manual Check
whether an
alarm startingwith "CP" or
"CPW" isgeneratedthroughout
the network.
If there issuch an alarm,
clear it
Major To ensure
that there is
no control plane alarm.
A control plane alarmmay disable
ASON
servicesfrom
running
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
16/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
9
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
according tothe NMS
online help.
normallyand may
evendirectly
affectASON
trails.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
17/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
10
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
ASONevents
Checkingfor
abnormalASON
events
20min/50
ASONservices
Monthly Manual 1. Checkwhether an
ASON servicererouting
failure has
occurredlately. (To
verify this
information,click new
events icon inthe main
topology ofthe NMS,then browse
the events inthe Browse
Events Logs
– [New
Events]
window.)
2. Check
whether anASON service
re-creationfailure hasoccurred
lately. (Toverify this
information,click new
events icon inthe maintopology of
the NMS,
then browse
the events inthe Browse
Events Logs
– [New
Events]window.)
Critical To monitornetwork
operation. Ifan ASON
rerouting or
re-creationfailure is
reported
frequentlyduring a
time period,identify the
cause andtakecorrespondi
ngmeasures.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
18/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
11
PreventiveMaintenance Item
TimeRequired
Frequency
InspectionMethod
Procedure Priority Purpose
Item Sub-Item
Inspectionof potential
risksaccording to
pre-warning
notices
Checking potential
risks based on
pre-warni
ng notices
30 min
/network
Monthly Manual Download pre-warning
notices fromthe
http://support.
huawei.comwebsite and
perform the
workarounds, preventive
measures, orsolutions
provided inthe notices.
Major To remove potential
risksaccording to
officially
released pre-warning
notices.
Preventive
maintenance inspection
Preventive
maintenance
inspection
60
min/NEMonthly Tool +
manual
1. Perform
preventivemaintenance
inspection
(PMI) ofASONnetworksusing a PMI
tool
(download thelatest tool
from the
http://support.huawei.comwebsite) and provide the
PMI result to
Huawei HQfor filing.
2. Analyze the
PMI resultaccording tothe PMI
guide. If thereare any
problems,
rectify themimmediately.
Major To ensure
that theASON
resources
and serviceson anASON NEare in good
condition.
http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/http://support.huawei.com/
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
19/116
Huawei Optical Network Maintenance Reference -
WDM ASON 1 SOP for Maintaining NG WDM ASON Devices
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
12
1.2 ASON Device SOP Checklist
Site: ____________ Version: ___________ Maintenance Owner: __________ Time: _____________
PMI Item PMI Sub-Item Result Remarks
ASON databases Checking backup databases for
ASON NEs□ OK □ NOK □ POK
ASON resources Checking the status of control
links□ OK □ NOK □ POK
Checking virtual TE links □ OK □ NOK □ POK
Checking status of TE links □ OK □ NOK □ POK
Checking for residual
cross-connections□ OK □ NOK □ POK
ASON services Checking ASON services □ OK □ NOK □ POK
Alarms Clearing control plane alarms □ OK □ NOK □ POK
ASON events Checking for abnormal ASON
events□ OK □ NOK □ POK
Checking potential
risks based on pre-warning notices
Checking potential risks based
on pre-warning notices□ OK □ NOK □ POK
Preventive maintenance
inspection
Preventive maintenance
inspection□ OK □ NOK □ POK
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
20/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
13
2 Quick Recovery Guide for NG WDMASON Services
2.1 OverviewThis chapter provides guidelines for recovering ASON services from service interruption ornode faults on an ASON network by assuming that the ASON network has been in operation,
aiming to guide the network maintenance personnel through the service recovery and faultdiagnosis processes.
This chapter assumes that the network maintenance personnel are skilled in fault diagnosis for
traditional WDM services on transport equipment and therefore focuses on the differences for
maintaining the ASON-capable WDM transport equipment and the traditional WDM transportequipment.
2.2 ASON Service Recovery Process
2.2.1 Intended Audience
This chapter is intended for the following personnel:
Customer personnel: members of the operator's maintenance team. They are responsiblefor routine maintenance of the ASON and handling of common faults.
Customer service personnel: members of Huawei's GTS team. They provide technical
support for the customer, and assist the customer in handling network faults. R&D personnel: members of Huawei's R&D team. They assist the customer service
personnel in handling network faults.
2.2.2 Standard Operations for Routine Maintenance
To ensure high survivability of the ASON, find the potential faults and problems occurring onthe ASON as early as possible, and promptly recover ASON services from faults, the network
maintenance personnel must maintain the ASON in a routine manner according to Routine
Maintenance Operation Guide. Particularly, they must at least take the following actions:
1. Perform preventive maintenance regularly using the SmartKit Inspector.
2. Back up the databases regularly.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
21/116
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
22/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
15
Domain Rerouting Occurs When No Rerouting Occurs When
OCH optical
layer
MUT_LOS or BD_STATUS
alarms are generated on FIU boards.
LOS alarms are generated only on SC2
boards.
MUT_LOS alarms are generated only onOA boards.
LOS, SSF, DEG, or EXC alarms aregenerated only on OTU boards.
The service is locked.
No other trails are available (the fiber is broken) for the affected service.
TE link check fails (for example, the TE
link is degraded or broken).
There are discrete cross-connections.
The control link is unreachable and a CL
DOWN alarm is reported for the route.
Note: After a service is rerouted, it will not be rerouted again in case of a channel alarm(an alarm on an OA or OTU board).
2.2.5 ASON Alarms
Among all ASON alarms, the following alarms have direct impact on ASON services:
1. Service interruption alarms: CPW_OCH_SER_INT, and CPW_ODUk_SER_INT
2. Link down alarms: CPW_OCH_TEL_DOWN and CPW_ODUk_TEL_DOWN
An OTU board reports an R_LOS, OTUk_SSF, DEG, or EXC alarm or an FIU board reports aMUT_LOS or BD_STATUS alarm when one or multiple services are abnormal.
The following ASON alarms have no direct impact on ASON services:
3. Link control alarms: CPC_CC_DOWN, CPC_OSPF_CL_DOWN, andCPW_OMS_TEL_DEG
These ASON alarms are generated when an SC2 board reports an R_LOS alarm or theRES overhead of an OTU board is unavailable. When the alarms are generated,verification of control links or TE links will fail.
4. Resource alarm: CPW_XXX_TEL_PATHMISThe alarm is usually generated when there are discrete cross-connections. In other words,
the alarm is generated when a cross-connection is configured only at one end for a fiberlink between two sites or the cross-connection at one end is in use while the
cross-connection at the other end is not in use.
5. The root causes for other ASON alarms (for example, CPW_CLNT_SER_NOTOR,CPW_OCH_SER_SLADEG, CPC_NODE_ID_CONFLICT, andCPC_RSVP_NB_DOWN) are not related to traditional alarms, nor have direct
relationship with interruption of ASON services. For details about the causes andmethods for handling them, refer to the alarm reference manual.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
23/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
16
2.2.6 General Principles for Recovering ASON Services
The symptoms and causes for interruption of different ASON services vary greatly. To reduce
the downtime and promptly restore an interrupted service, the following steps are generallyused:
1. Locate the interrupted service, and optimize the service onto the preset restoration trail orrevert the service back to the original trail (or switch the service to the protection trail)
after confirming the trail is available.
2. If the preceding step fails, deactivate the service. Then create the service as a traditionalservice.
3. Check for traditional alarms that have caused the service interruption, and clear them.
The preceding steps are three essential actions for promptly recovering ASON services.
For the detailed service recovery procedure, see the next section.
To determine whether a trail is available, users can check whether an end-to-end trail is
available based on the network topology, fiber interruption symptoms, ASON topology
(TE link and control link), and service source or sink node.
NOTE
While restoring an interrupted service, confirm the following information:
Check whether any configurations are changed before or after the service is interrupted.
These configurations include protocol parameter settings (OSPF, LMP, and RSVP parameters) that
affect the functions of the control plane, port attribute settings (port loopback, FEC/AFEC mode, and port rate mode) that are related to the service, and service configurations (for example, service trail
configurations on the client side).
Check whether there is any service-affecting major alarm after the service is interrupted.
For example, check whether there is a new hardware damage alarm (HARD_BAD), fiber breakalarm (R_LOS and MUT_LOS), traditional service interruption alarm (LOF, and AIS), or ASON
service interruption (XXX_SER_INT).
2.2.7 Key Information for Recovering ASON Services
After an ASON service is interrupted, troubleshooting personnel must obtain the key
information about the service fault, because the key information is very important for preliminary fault diagnosis and prompt service recovery. The key information includes:
1. When the ASON service is interrupted?
Method: The troubleshooting personnel can determine the time either from the customerfeedback or by checking the alarm occurring time.
2. Whether a fiber is broken on the link for the interrupted service?
Method: In the main topology or signal flow diagram of the NMS, check whether a fiberis broken. If the fiber is red, then the fiber has been broken.
3. What alarms (ASON or traditional alarm) are reported for the interrupted service?
Method: Check whether there are any new major alarms.
4. Does the fault occur in an ASON or traditional service?
Method: Check for service alarms directly in the WDM ASON Trail Management and
WDM Trail Management windows of the NMS. The trail for the interrupted service is
red.
5. If multiple services are interrupted, do they all carry client services? Which services arethe most important? Are all interrupted services in the same direction?
Method: Directly confirm this information with the customer.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
24/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
17
6. Which services are interrupted (source/sink nodes and boards)? What are the service protection levels? Which trails does the service traverse? Do the interrupted services pass
any regenerators?
Method: View the information directly in the WDM ASON Trail Management window.
Click Original trail and Current trail in the lower left part of the window, and take thescreenshots for the required information.
7. Has rerouting been triggered for the interrupted service? Has the rerouting succeeded orfailed?
Method: Directly click Event to view the detailed event information (alternatively, selectthe specific event screenshot).
8. Has any network configurations been modified, has any services been added or deleted,or has service cutover been performed before the fault occurs?
Method: Directly confirm this information with the customer or maintenance personnel.
2.2.8 General Flowchart for ASON Service Recovery
Figure 2-1 General flowchart for ASON service recovery
2.3 AQuick Recovery Process for ASON Services
2.3.1 Quick Recovery Process for ASON Services (Optical-LayerService)
Step 1 Locate the trail for interrupted services.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
25/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
18
1. According to the customer feedback, determine when service interruption occurs, howmany services are interrupted in addition to the source and sink nodes and fiber status
(for example, fiber break) of the network.
2. Choose Configuration > WDM ASON > WDM Trail Management to display the
WDM ASON trail management window. Identify the interrupted ASON service trailaccording to the service source or sink node information and ASON service alarms. If
the CPW_OCH_SER_INT alarm is generated, clear the alarm to recover the services.
3. If no ASON service interruption alarm is reported, click in the lowerright of the Browse Current Alarms window. After alarm synchronization is complete,check whether an ASON service interruption alarm is reported. If there is such an alarm,clear the alarm to recover the services.
4. If no service interruption alarm is reported after alarm synchronization is performed,
filter the interrupted services according to their source or sink nodes. Check whether
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
26/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
19
there are inactive services. If there are inactive services, check whether they are thefaulty services. If yes, activate the services to recover them.
5. If no ASON service interruption alarm has ever been reported, check for traditionalalarms that are generated on the source or sink NE during the service interruption time period as specified by the customer. Then determine the affected boards and
corresponding ASON service trails according to the alarm information.
NOTE
For optical-layer services on WDM devices, only BD_STATUS or MUT_LOS alarms on FIU boards can
trigger rerouting. In other words, alarms on OTU boards cannot trigger rerouting.
Step 2 Restore the interrupted services.
Route computation for optical-layer services is very complex because it must consider manyfactors such as optical parameter constraints, regenerators, and optical power. Therefore,Huawei uses preset restoration trails to recover ASON services at the optical layer of NGWDM equipment. A maximum of two preset restoration trails can be configured for each
silver service. When a silver service at the optical layer is interrupted, the best choice to
quickly recover the service is to optimize the service onto one of the preset restoration trails,or revert the service back to its original trail.
The following describes how to quickly recover an interrupted service:
1. In the WDM Trail Management window, select the service and click .Then save the service information (for example, the service attributes, current trail, and
original trail) as an excel file. The information will be subsequently used as reference forfault diagnosis and service recovery.
2. In the WDM Trail Management window, select the service. Check whether presetrestorations are configured for the service. If they are configured, take the screenshotsfor the two preset restoration trails, because this information can be used as reference for
reconfiguring preset restoration trails for the service in future fault diagnosis. In addition,
check whether the current trail of the service is the same as one of the preset restorationtrails. Ensure that the preset restoration trails are explicit trails during service
optimization.
The following figure shows a screenshot for one of the preset restoration trails.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
27/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
20
3. If no restoration trail is configured for the service, optimize the trail for the service.During trail optimization, try to select a trail that has common nodes as the original trailand specify it as the explicit trail for the service. Then check whether the service is
recovered. If a failure message is displayed or if a success message is displayed but the
service is not recovered, go to the next step.
4. If the service fails to be optimized or reverted back to the original trail, check forconfiguration errors (such as rerouting lockout and incorrect optical parameter settings,
link verification failure/unavailability of control links not triggered by alarms). Then
assess the current network topology, fiber status (for example, fiber breaks), and servicesource or sink node. If idle trails are available for the service, deactivate the service and
then try to recover the service as a traditional service by creating a WDM trail for it. Ifthe service still fails to be recovered, go to the next step.
5. Query the traditional alarms corresponding to all ASON service trails. If there are sometraditional alarms (for example, R_LOS, MUT_LOS, VOADATA_MIS, andOPA_FAILED), clear the traditional alarms according to the traditional alarm handling
process. Then check whether the service is recovered. If not, go to the next step.
6. Contact Huawei for troubleshooting.
NOTE
For WDM ASON services, disable the function of computing optical parameters if the optical
parameter check fails when you are optimizing the service. Then optimize the service again. To disable the optical parameters, right-click the service in the ASON trail management window, and
choose Disable Optical Parameters from the shortcut menu.
----End
2.3.2 Quick Recovery Process for ASON Services (OTNElectrical-Layer Service)
Step 1 Identify the interrupted services.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
28/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
21
1. According to the customer feedback, determine when service interruption occurs, howmany services are interrupted in addition to the source and sink nodes and fiber status
(for example, fiber break) of the network.
2. Choose Configuration > WDM ASON > WDM Trail Management to display the
WDM ASON trail management window. Identify the interrupted ASON service trailaccording to the service source or sink node information and ASON service alarms. If a
CPW_ODU3_SER_INT, CPW_ODU2_SER_INT, CPW_ODU1_SER_INT, or CPW_ODU0_SER_INT is generated, clear the alarm to recover the services.
3. If no ASON service interruption alarm is reported, click in the lowerright of the Browse Current Alarms window. After alarm synchronization is complete,
check whether an ASON service interruption alarm is reported. If there is such an alarm,
clear the alarm to recover the services.
4. If no service interruption alarm is reported after alarm synchronization is performed,select and right-click all services and choose Synchronize from the shortcut menu.Check whether there are inactive services. If there are inactive services, then NMS datais inconsistent with the NE data. Activate these services if required. If activating these
services fails, refer to Step 2.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
29/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
22
5. If no ASON service interruption alarm has ever been reported, ensure that the staticcross-connections are configured correctly for the client side of the NE. If alarms have
been generated on the client side, ensure that the client-side boards are free of any faults
or are online. Then, check for alarms and performance events on OTU boards. If thereare alarms, locate the affected ASON services and rectify the faults.
Step 2 Restore the interrupted services.
1. In the WDM Trail Management window, click Maintenance to sequentially refreshthe current trail, original trail, and preset restoration trails. Then select the service and
click . Then save the service information (for example, the service attributes,current trail, and original trail) as an excel file. The information will be subsequentlyused as reference for fault diagnosis and service recovery.
2. In the WDM Trail Management window, select the service. Check whether presetrestorations are configured for the service. If they are configured, take the screenshots
for the two preset restoration trails, because this information can be used as reference forreconfiguring preset restoration trails for the service in future fault diagnosis. In addition,
check whether the current trail of the service is the same as one of the preset restorationtrails. Ensure that the preset restoration trails are explicit trails during service
optimization.
The following figure shows a screenshot for one of the preset restoration trails.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
30/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
23
3. If no restoration trail is configured for the service, optimize the trail for the service.During trail optimization, try to select a trail that has common nodes as the original trail
and specify it as the explicit trail for the service. Then check whether the service is
recovered. If a failure message is displayed or if a success message is displayed but theservice is not recovered, go to the next step.
4. If the service fails to be optimized or reverted back to the original trail, check forconfiguration errors (such as rerouting lockout, link verification failure/unavailability ofcontrol links not triggered by alarms). Then assess the current network topology, fiber
status (for example, fiber breaks), and service source or sink node. If idle trails are
available for the service, deactivate the service and then try to recover the service as atraditional service by creating a trail for it. If the service still fails to be recovered, go to
the next step.
5. Query the traditional alarms corresponding to all ASON service trails. If there are sometraditional alarms (for example, R_LOF, R_LOS, LOM, and BUS_ERR), clear thetraditional alarms according to the traditional alarm handling process. Then check
whether the service is recovered. If not, go to the next step.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
31/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
24
6. Ty to deactivate/activate the ASON service. If the service is recovered after thedeactivation/activation, reconfigure the service attributes and the preset restoration trails
for the service based on the service information and screenshots saved previously.
----End
2.3.3 Quick Recovery Process for ASON Services (VC-4 Services)Locate the interrupted VC-4 ASON service according to the control plane alarms. Determinewhether the alarms are an internal network alarm (CP_SER_INT) or an external networkalarm (CP_SRV_INT_OUT).
1. For an external network alarm, locate and handle the service faults on the client side. Forexample, check whether the service receiving boards are faulty or offline and whether
errors have been generated in client signals.
2. For an internal network alarm, handle the alarm as instructed in section 2.3.2 "QuickRecovery Process for ASON Services (OTN Electrical-Layer Service)".
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
32/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
25
If no control plane alarm indicating service interruption is reported, check for correspondingtraditional alarms according to the service interruption time, and clear them to recover the
services.
2.4 Troubleshooting Processes for Typical ASON ServiceFaults
This section provides the troubleshooting processes for typical ASON service faults (for
example, fiber breaks, device software or hardware faults that cause ASON serviceinterruption) so that maintenance personnel can promptly recover the services by isolating the
faults according to the troubleshooting processes.
2.4.1 Troubleshooting Process for Route Computation Failures
Fault Description
An ASON service (OCH, ODUK, or SDH) is interrupted, and the following alarms arereported for the end-to-end ASON service trail and the port on the WDM side:
End-to-end ASON service alarms:
CPW_OCH_SER_INT, CPW_ODUk_SER_INT, and CP_SER_INT
Port alarms on the WDM side of affected boards:
R_LOS, OTUk_LOF, OTUk_SSF, OTUk_LOF, OTUk_LOM, OUTk_DEG, ODUk_PM_SSF,
ODUk_PM_OCI, ODUk_PM_DEG, IN_PWR_HIGH, IN_PWR_LOW, R_LOF, AU_AIS,
BD_STATUS, MS_AISEnd-to-end ASON service event: route computation failure event
Workarounds
According to the fault occurrence time and the customer feedback, identify the interrupted
services and promptly recover the services by, for example, optimizing the services orreverting them to their original trails. In events of a route computation failure or optical
parameter check failure, preferentially check whether there are sufficient network resources. Normally, many factors can cause a shortage of network resources. They include:
configuration errors, fiber breaks, residual cross-connections, device hardware faults, node
faults, and software bugs. Currently, rerouting protection is provided for optical-layer services
using preset restoration trails. If a service is interrupted and cannot be automatically recovered because of a route computation failure, manually recover them using the following methods:
1. Find the current ASON service trail (including the preset restoration trail) of the service,and deactivate the service.
2. Create a static service over a traditional WDM trail and recover the service usingtraditional commissioning methods. For an optical-layer service, try to use the original
trail or configure two preset restoration trails.
NOTE
If the preceding methods cannot recover the service, isolate the fault according to the symptoms (for
example, fiber breaks, residual cross-connections, hardware faults, node faults, and software bugs) andthen recover the service. For details about the troubleshooting procedure, refer to XXX Fault Location
and Handling Procedure.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
33/116
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
34/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
27
2.4.2 Troubleshooting Process for Rerouting Failures
Fault Description
An ASON service (OCH, ODUK, or SDH) service is interrupted. For details about the servicealarms of the end-to-end ASON service trail and WDM port, see section 2.4.1
"Troubleshooting Process for Route Computation Failures."
End-to-end ASON service event: rerouting failure event
Workarounds For "Creation timeout": If timeout is caused by traffic congestion or a communication
failure at the source or sink node, the service cannot be recovered by just optimizing the
service to another trail. Instead, you have to deactivate/activate the service, or re-createan end-to-end traditional service.
For "RSVP egress or ingress port is down", explicitly specify different egress ports tooptimize the service onto another trail so as to quickly recover the service.
The system returns the error prompt "Failure of label allocation". If the faulty node is the
source or sink node of the ASON service, attempt to recover the interrupted service byoptimizing the tunable wavelength for the optical layer. For the non-tunable NEconfigurations, identify the resources that conflict with the cross-connections or
wavelengths of the current service and delete the conflicting resources to recover theinterrupted service.
Troubleshooting Procedure
Step 1 Review the ASON service events that are generated during the time period when the fault
occurs, and check whether the "Creation timeout", "RSVP egress or ingress port is down" or"Label allocation failure" message is displayed.
Step 2 If the "Creation timeout" message is displayed, check whether network nodes successfullycommunicate with each other. The nodes fail in control plane communication if the
CPC_OSPF_AUTH_ERR, CPC_OSPF_NB_DOWN, CPC_RSVP_AUTH_ERR, or
CPC_RSVP_NB_DOWN alarm is generated on the control plane. To restore the control planecommunication, clear the alarm. If the network nodes communicate successfully with each
other, check whether traffic congestion has occurred at the source node, sink node, or anintermediate node of the service using the following methods:
1. Run the nbb-set-debug-mode:open command to the node to enable the debug functionof the node.
2. Run the mpls-query-rsvp-tcevent:0.0.0.0,0.0.0.0,0,1,2 command five times at intervalsof 3 seconds.
If the same triplet information is displayed in the command output every time, then traffic
congestion has occurred at the node.
If traffic congestion has occurred at the source or sink node, deactivate/activate the service or
perform a soft reset on the node to rectify the fault. If traffic congestion occurs at anintermediate node, recover the service by optimizing it to an available trail, then perform a
warm reset on the node to rectify the fault. At this point, the "creation timeout" fault is
rectified.
Step 3 If the "RSVP egress or ingress interface Down" error message is displayed, first determine the
faulty node according to the rerouting failure event. Then check whether the interface control
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
35/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
28
block of the control plane is in normal state and whether the RSVP interface information iscomplete using the following methods:
1. Run the mpls-get-if:all command to query the interface information. Check whetherthere is an interface corresponding to the interrupted service and whether the interface
information is correct (DEFINED=0x7, ADMIN=UP, OPER=UP; the remoteinformation is not null).
2. If the interface information is incomplete, runthe :mpls-debug-command:get_lim_if,0,0,0,0,0; command to recover the interface
information or perform a warm reset on the SCC board of the NE.
At this point, the "RSVP egress or ingress interface Down" fault is rectified.
Step 4 If the "Failure of label allocation" error message is displayed, check whether there areconflicting resources on the network. Handle the fault using the following methods:
1. Check whether there are any reserved resources or residual cross-connections on the preset restoration trails or specified explicit trails.
2. Check whether there are any resource inconsistency alarms (for example,CPW_XXX_TEL_PATHMIS).
If there are some reserved resources or residual cross-connections, manually release the
reserved resources or delete the residual cross-connections.
At this point, the label allocation failure is resolved.
----End
2.4.3 Troubleshooting Process for Unreachable NEs
Fault Description
An ASON service (OCH, ODUK, or SDH) is interrupted, and the corresponding NE is
dimmed in the main topology of the NMS) and an NE unreachable alarm is reported for the
NE. As a result, users cannot operate and configure the NE. For details about the possible
service alarms generated by the end-to-end ASON service trail and port on the WDM side, seesection 2.4.1 "Troubleshooting Process for Route Computation Failures."
Workarounds
After the ASON service is interrupted and the NE is unreachable by the NMS, identify the
cause. If the NE is unreachable by the NMS because the gateway NE is faulty, check whether
the secondary gateway is available or reconfigure another gateway NE for the unreachable NE.Then try to recover the service using the Navigator if the NE can be managed by the
Navigator. The recovery steps are as follows:
Step 1 Query alarms on the add/drop boards and the source/sink nodes of the affected services.
Step 2 According to the source or sink node of the interrupted service, run a command to query anddetermine the corresponding affected ASON service (determine the affected ASON service
according to the add/drop service board and ASON service alarm).
Step 3 Recover the interrupted service quickly be optimizing the service (or reverting the serviceover to the original trail) manually.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
36/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
29
NOTE Run the following command to query service alarms:
:alm-get-curdata-ext:0,ALL,0
Run the following commands to query ASON services:
:mpls-get-och-smlsp:src (optical layer)
:mpls-get-otn-smlsp:src,all (ODUk electrical layer)
:mpls-get-smlsp:src (SDH)
Run the following commands to optimize ASON services:
:mpls-optim-och-lsp (optical layer), including the wavelength tuning command:
mpls-adjust-och-lsp
:mpls-optim-otn-lsp (ODUk electrical layer)
:mpls-optim-lsp (SDH)
mpls-restore-och-lsp (manually revert optical layer ASON services), mpls-restore-och-origrt (revert network-wide optical layer ASON services)
mpls-restore-otn-lsp (manually revert ODUk services), mpls-restore-otn-origrt (revert
network-wide ODUk services)
mpls-restore-lsp (manually revert SDH services), mpls-restore-original-rt (revert network-wideSDH services)
Recover the service quickly by optimizing the service or reverting the service over to the original
trail.
Step 4 If neither the NMS nor the Navigator is available but trails are available for the service, youcan disable the SC2 and OA laser of the reachable node on an available trail to trigger service
switching, or perform a cold reset on the FIU and remove the line fiber to trigger servicererouting, finally restoring the interrupted service.
----End
Troubleshooting Procedure
An NE may become unreachable by the NMS for many reasons: IP address conflict, subnetmask errors, OSC/ESC communication fault between NEs, SCC hardware or software faultsof the NE, ECC storm, and faults of the customer's outband DCN. You can simply
troubleshoot this fault by determining whether the NE becomes unreachable by the NMS
because of the fault of the gateway NE. If yes, replace the faulty gateway NE. For other typesof faults, refer to the DCN Recovery Guide.
NOTE
To check for network communication faults, run the following commands:
:cm-get-iproute (query the routing table):cm-set-ip (modify the IP address)
:cm-set-submask (modify the subnet mask)
:cm-get-gateway (modify the gateway NE)
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
37/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
30
2.4.4 Troubleshooting Process for Interruption of aSingle-wavelength length ASON Service
Fault DescriptionAn ASON service (OCH, ODUK, or SDH) service is interrupted and alarms are reported. For
details about the service alarms of the end-to-end ASON service trail and WDM port, see
section 2.4.1 "Troubleshooting Process for Route Computation Failures."
In the ASON service management window, only a single ASON service is interrupted. You
can determine whether the service initiates the rerouting process according to thecorresponding performance event (rerouting success or failure event) and service alarm
(ASON service interruption alarm, "not over the original trail" alarm, and rerouting lockoutalarm). The possible symptom of service interruption is that the rerouting process is not
initiated or the ASON service is still interrupted even after the rerouting process is initiated.
Workarounds
For details about the workarounds, see section 2.4.1 "Troubleshooting Process for Route
Computation Failures."
NOTE
If the interrupted single-wavelength ASON service does not initiate the rerouting process, it is possible
that no trail meets the requirement or a component (for example, board or fiber) is faulty (the currentASON service only supports the rerouting switchover triggered by a line-class fault rather than achannel-class fault).
Troubleshooting Procedure
Step 1 According to the fault information (for example, fault time, faulty node, and current alarm) provided by the customer, determine whether a single ASON service or multiple ASONservices are interrupted through the MUT_LOS alarm and the R_LOS alarm of multiple
boards. If a single ASON service is interrupted, proceed to the next step.
Step 2 If the ASON service has the protection capability (traditional protection plus ASONassociated service), check whether the service is locked to an abnormal channel forcibly. Ifyes, unlock the service and check whether the service is recovered. If a normal channel is
available, switch the service toward the normal channel forcibly. Alternatively, check whetherthe switching conditions are configured correctly (SD). If not, correct the switching condition
configurations and check whether the service is recovered. If the service is still not recovered,
proceed to the next step.
Step 3 Check the interrupted ASON service. If rerouting is locked for the ASON service, unlockrerouting. The ASON service may still fail to initiate the rerouting process (it is possible that a
channel alarm causes the service interruption; if the service has an available trail, optimize theservice manually), or the ASON service is still interrupted even after the ASON serviceinitiates the rerouting process (by default, an optical-layer alarm does not trigger service
rerouting). In this case, proceed to the next step.
Step 4 According to the interrupted service information, check whether the service trail must traversea repeater, and check whether the current trail has traversed a repeater. If the current trail does
not traverse a repeater, you can specify a repeater to optimize the service for restoring theservice. If the service trail does not need to traverse a repeater or the current trail has traverseda repeater, but the ASON service is still interrupted, proceed to the next step.
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
38/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
31
Step 5 Determine whether a service of the same wavelength causes any crosstalk. You can disablethe OTU at the source or sink node of the interrupted service or disable the laser of the trunk
board to determine whether another service board generates any same-wavelength crosstalksection by section. Alternatively, ask the customer's maintenance personnel to check whether
the wavelength of a board is adjusted at the OTU of the source or sink node, on a trunk board,
or in an intermediate site of the service, or whether any service board is inserted. According tothe check results, adjust the conflicting wavelength to recover the interrupted service. If theASON service is still interrupted, or there is no conflicting wavelength, proceed to the nextstep.
Step 6 Check whether the OTU configurations at the source node are consistent with those at the sinknode and whether the trunk board configurations (for example, wavelength, rate, FEC, andTDC) between two ends are consistent with each other. If some configurations are
inconsistent, correct the configurations immediately. If the configurations are all consistent
but the ASON service is still interrupted, proceed to the next step.
Step 7 Check whether a power adjustment alarm is generated for the service trail, for example, anOPA adjustment failure alarm (OPA_FAIL_INDI). If yes, check the corresponding
configuration information (including the WSS board, optical amplifier, and preset insertionloss) of the OPA reference section. If there are some configuration errors, correct the
configurations as instructed by the System Commissioning Guide. If no adjustment failure
alarm is generated, or if the corresponding configurations are correct but the ASON service isstill interrupted, proceed to the next step.
Step 8 Determine the current trail for the interrupted service. Check the power of the OTU (includingthe trunk board) at the source or sink node of the single-wavelength service. Then query thecurrent and historical performance of the OTU at the receiving or transmitting end. If the
current network is configured with an MCA board, use the MCA board to scan the
single-wavelength power in the service trail and check whether any power is abnormal in theservice trail. If the current network is not configured with any MCA board, check whether the
receiving power of the corresponding port is normal (performing a hardware loopback) on site.If a faulty fiber or board is identified, replace the faulty fiber or board. Then, the ASON
service is recovered.
NOTE
When checking the faulty trail, note the following alarms and configuration information:
Multiplexer or demultiplexer board alarm and optical amplifier board alarm: RLOS, MUT-LOS,
BD_STATUS, MODULE_ADJUST_FAIL, MOD_COM_FAIL, WAVEDATA_MIS, IN_PWR_LOW,IN_PWR_HIGH, OUT_PWR_HIGH, and OUT_PWR_LOW
Multiplexer or demultiplexer board and optical amplifier board configurations: insertion loss,
attenuation, rated power, gain, and dispersion compensation.
Rectify a board fault (if available) as instructed by the Board Replacement Guide, and rectify a fiber
fault (if available) as instructed by the Fiber Repair Guide.
----End
2.4.5 Troubleshooting Process for Interruption ofMulti-Wavelength Services
Fault Description
ASON services (OCH, ODUK, or SDH) are interrupted and alarms are reported. For detailsabout the service alarms of the end-to-end ASON service trail and WDM port, see section
2.4.1 "Troubleshooting Process for Route Computation Failures."
-
8/9/2019 Huawei Optical Network Maintenance Reference-WDM ASON-20140826-C
39/116
Huawei Optical Network Maintenance Reference -
WDM ASON 2 Quick Recovery Guide for NG WDM ASON Services
Issue 02 (2014-08-26) Huawei Proprietary and Confidential
Copyright © Huawei Technologies Co., Ltd
32
In the ASON service management window, multiple ASON services are interrupted. You candetermine whether the service initiates the rerouting process according to the corresponding
performance event (rerouting success or failure event) and service alarm (ASON service
interruption alarm, "not over the original trail" alarm, and rerouting lockout alarm). The possible symptom of service interruption is that the rerouting process is not initiated or the
ASON service is still interrupted even after the rerouting process is initiated.
Workarounds
For details about the workarounds, see section 2.4.1 "Troubleshooting Process for Route
Computation Failures."
Troubleshooting Procedure
Step 1 According to the fault information (for example, fault time, faulty node, and current alarm) provided by the customer, determine whether a single ASON service or multiple ASON
services are interrupted by checking the related alarms, for example, the MUT_LOS alarmand the R_LOS, IN_PWR_HIGH, or IN_PWR_LOW alarm of multiple OTUs. If multipleASON services are interrupted, proceed to the next step.
Step 2 Check each interrupted ASON service. If rerouting is locked for the ASON service, unlockrerouting. if the ASON services initiate the rerouting process and are recovered, the fiber onthe line side is faulty. Therefore, rectify the line fault. If the ASON services still fail to initiate
the rerouting process or the ASON services are still interrupted even after the ASON servicesinitiate the rerouting process, proceed to the next step.
Step 3 If the ASON services are still interrupted even after the rerouting or switching process isinitiate