ts-rnc-sw-038-i13

Technical Support Note TS-RNC-SW-038

RNC PROBLEM REPORT INSTRUCTIONS Radio Control lers

WCDMA RNC

RN3.0, RN4.0, RN5.0

This document contains following type of information

Informative X Preventive Corrective Urgent Security

Information is classified as Internal Public X Customer Specific

Nokia Siemens Networks Company Confidential 1 (32)

Table of Contents 1. Compatibility / Dependencies to other products....................................................................... 4 2. Keywords ................................................................................................................................. 4 3. Summary.................................................................................................................................. 4 4. Detailed description.................................................................................................................. 4 5. Solution / Instructions............................................................................................................... 7

5.1 Using the log collection macro rnclogcol.exe ........................................................................... 7 5.2 Basic instructions for displaying logs and for monitoring messages ........................................ 9 5.3 Commands for displaying computer logs and system logs from DMX units ............................ 9 5.4 Commands for message monitoring in DMX units................................................................... 9

6. Data to be collected in all abnormal situations....................................................................... 11 6.1 SW and HW version information ............................................................................................ 11 6.2 Alarm history .......................................................................................................................... 11 6.3 Working state of units............................................................................................................. 11 6.4 MML command log................................................................................................................. 12

7. Case specific instructions....................................................................................................... 13 7.1 System restart ........................................................................................................................ 13 7.2 Spontaneous restart of a DMX unit (OMU, RSMU, RRMU, ICSU, GTPU) ............................ 14 7.3 Spontaneous restart of a CHORUS unit (MXU, A2SP, NIP1, NIS1, SFU and DMPG) ............................................................................................................................................. 14

8. Restart of DSP/DMPG ........................................................................................................... 16 8.1 RNC active alarms and alarm history..................................................................................... 16 8.2 Plug-in Unit HW information................................................................................................... 16 8.3 All Function Unit states and information................................................................................. 16 8.4 DSP service statistics information.......................................................................................... 16 8.5 Collect computer log in destination DMPG ............................................................................ 16 8.6 Collect restarted DSP blackbox information........................................................................... 16 8.7 Collect sysdump information for suspected faulty DMPG unit ............................................... 17 8.8 Collect bblog information of suspected faulty DMPG unit ...................................................... 17 8.9 DMPG communication processor blackbox information ........................................................ 17 8.10 Collect DSP service related information................................................................................. 18 8.11 Collect PDRACT channel related information ........................................................................ 18 8.12 Collect NPGE, NPGEP, NPS1 and NPS1P units related information.................................... 19 8.13 Monitor DMX message in suspected faulty DMPG................................................................ 21 8.14 Monitor DSP message and DMX message to/from suspected faulty DSP............................ 21 8.15 Check SW version of one specific program block.................................................................. 21 8.16 Collect diagnostics report history ........................................................................................... 21

9. BTS initialization case............................................................................................................ 22 10. Call case ................................................................................................................................ 25 11. Synchronization case............................................................................................................. 27 12. ESA 12/24 LAN switch case .................................................................................................. 28 13. OMS case .............................................................................................................................. 29

13.1 General case.......................................................................................................................... 29 13.2 Measurement case................................................................................................................. 29

14. Message monitoring from OMU with families for different problem situations ....................... 31 15. Note ....................................................................................................................................... 31 16. References............................................................................................................................. 31


Contact:

Contact your local Nokia Siemens Networks support

Summary of changes:

25-02-2008 1.1 A new template and OMS log instructions added.

09-04-2008 1.2 Rnclogcol macro improvement. Amount of MML sessions are limited.

30-05-2008 1.3 Rnclogcol macro improvement and chapter 7.3 updated.

25-06-2008 1.4 Rnclogcol macro improvement. RNC name and time stamp added to log directory name. Chapter 9 and chapter 10 ddesk command added.

23.09.2008 2.0 Chapter 7.3 updated 29.01.2009 3.0 RN4.0 validity added 25.02.2009 4.0 Chapters 13.1 and 13.2 updated. RN21 info

removed. Added chapter Collect NPGE and NPGEP units related information

06.03.2009 5.0 Updated new macro versions 26.03.2009 6.0 Chapters 5.1, 5.3, 7.2, 8.12 and 13.1 updated 09.06.2009 7.0 Chapter 13 updated and zcollectlogs.sh added 02.11.2009 8.0 Chapters 9 and 10 updated, RN2.2 removed,

RN5.0 added 01.12.2009 9.0 Chapter 8.12 title updated 20.01.2010 10 Chapter 13.1 updated 22.04.2010 11 Chapter 14 updated, a new rnclogcol.exe and

default_config.ini added, other *.ini files removed.

29.04.2010 12 New rnclogcol.exe. 07.05.2010 13 New rnclogcol.exe.


Purpose

This document contains generic information about products. These can be instructions that explain problem situations in the field, instructions on how to prevent or how to recover from problem situations, announcements about changes or preliminary information as requirements for new features or releases.

1. COMPATIBILITY / DEPENDENCIES TO OTHER PRODUCTS

No affect

2. KEYWORDS

Problem report, Electra, RNC outage, System restart, Unit restart.

3. SUMMARY

This document contains basic instructions for making a problem report in such a way that all the necessary information is included in it. Filling in the problem report carefully and attaching required information to it makes the problem investigation process faster.

4. DETAILED DESCRIPTION

This document contains instructions for collecting basic data from the network element (RNC) for fault analyzing purposes. This data shall always be collected if some abnormal situation has occurred in the system, and Nokia Siemens Networks (NSN) will be informed of it. An abnormal situation may be, for example, a decreased traffic handling capacity, or a spontaneous restart of a computer unit. In the worst case the abnormal situation may lead to a system outage.

The data to be collected is essential for the further analysis of the problem for trying to find out the root cause of the fault. The data shall be collected as soon as possible after the abnormal situation has taken place. This is important because the information stored about the problem (e.g. the black box of a certain unit, or alarm history) may get overwritten in the process of time.

The data to be collected, as well as the commands to be used for collecting them, have been described in the instructions below. The data shall be stored in descriptively named log files, which shall then be zipped into to file and delivered to NSN as an attachment of the problem report / outage report.

The log collection macro (rnclogcol.exe) and ini files are available via NOLS. The macro makes possible to collect automatically the required logs for the following situations: NE spontaneous restart, unit spontaneous restart, RNC configuration, call success problems and OAM (BTS) problems. See chapter Using the Log Collection Macro rnclogcol.exe for more details.

If only few of logs are needed take manually the relevant settings for every particular case are explained later in this document.

In addition to collecting the data, fill in the problem report by taking into account the following issues:


Title of the problem report should be a very short description of the fault

situation

Description of the problem in the problem report should be an exact and a clear description of the fault situation. Note that only one fault shall be reported in one problem report and at least the following information should be provided in addition to the actual problem description:

Situation in the beginning, e.g. the first symptoms of the failure

Operations made, which possibly caused the failure

Situation after the failure

Recovery actions made

Name and version of the possibly installed new software modules

Make sure that necessary attachments are included to the problem report to avoid unnecessary information requests

In a multivendor environment, collect detailed information of other products to the Description field of the problem report

Write down the SW Release used in the NE.

Write down the SW build of the SW Release and the implemented CD level in RNC. This information can be obtained with the MML command ZWQO:CR;

For filling in the severity of the problem, see the definition of the different severities in Table 1, NSNs Problem Classification:

NSN Problem Class

Definition of problem report severity as defined in TL 9000 Quality Management System, Measurement Handbook, Release 3

Examples

A-Critical Only total or major outages that are not avoidable with a workaround solution.

Critical (Emergency duty contacted) problems severely affect service, capacity/traffic, billing, and maintenance capabilities and require immediate corrective action, regardless of time of day or day of the week as viewed by a customer upon discussion with the collaborator.

System restart, all links down Simultaneous restarts of active

computer units More than 50% traffic handling capacity

out of use Subscriber related RNC functionality is

not working RNC cannot be accessed or monitored

from NetAct B-Major The fault affects traffic randomly or problem leads only to degradation of network performance or

Major problems cause conditions that seriously affect system operation, maintenance and administration, etc. and require immediate attention as viewed by the customer upon discussion with the collaborator. The urgency is less than in critical situations because of a lesser immediate or impending effect

Single restart of computer units Problems with back-up Configuration changes (RNW, HW and

SW) are not working Problems seriously affecting end user

service, but avoidable with workaround solution

Capacity/quality related functionality is


the fault makes it difficult for the customer to operate the RNC.

on system performance, customers, and the customers operation and revenue.

not working Performance measurement or alarm

management is not working Activation of a new feature fails A single performance measurement is

not working completely Configuration changes (RNW, HW and

SW) are not working completely Subscriber related functions are not

working completely Alarm management of objects (BTS,

functional units) is not working completely

Capacity/quality related functionality is not working completely

Major errors in documentation, for example, an alarm or description is missing from documentation

Vital documents are missing from the documentation library

C-Minor Minor fault not affecting operation or service quality.

Other problems that a customer does not view as critical or major are considered minor. Minor problems do not significantly impair the functioning of the system and do not significantly affect service to customers. These problems are tolerable during system use. Engineering complaints are classified as minor unless otherwise negotiated between the customer and supplier.

Failures not seriously affecting traffic Errors in MML syntax Minor errors in MML/Statistic output Minor errors in documentation

Table 1. NSNs Problem Classification


5. SOLUTION / INSTRUCTIONS

5.1 Using the log collection macro rnclogcol.exe

Most of the RNC configuration data and logs for the call cases can be collected with the rnclogcol macro. Download the rnclogcol macro from NOLS and extract rnclogcol.zip file into the PC folder named e.g. C:\Temp\Rnclog\. Remote monitoring can be started in either WO-EX or SP-EX on OMU. Parameter omu_state define which OMU is used. It is good idea to use SP-EX omu, in order to have WO-EX omu service terminals is available for other use. Own commands can be created in [own_commands] section. For example unit = MML is is for specific MML commands. The commands are created into the text file. Path and a name of text file are defined as follows: file = C:\Users\commands.txt. Own commands can be activated by get_own_commands parameter set to 1. Unit = ICSU means commands for ICSU functional units. The commands are again created into the text file. Path and a name of text file are defined as follows: file = C:\Users\commands_2.txt. Also other RNC functional units can be use on own command list. Example own commands set up [own_commands] unit = MML file = C:\Temp\rnclog\commands.txt Example content of commands.txt file ZWQV:OMU,0; ZWQV:OMU,1;

Define function start order priority: the lower the value, the higher the priority. However value 0 means that the function is not in use. Default configuration file name is default_config.ini

1. Create IP address file with *.txt extension and save it to C:\temp\rnclogcol\ folder. It is possible to define name for your Network Element in NE IP-address file. This is

optional, so you can still use your old IP-address files.

Format: 10.11.12.13 RNC_NAME #comment This will add RNC_NAME into log directory name.

2. Run rnclogcol.exe on command prompt

Syntax rnclogcol.exe -c Default_config.ini -i

Example: c:\temp\rnclogcol\rnclogcol.exe -c Default_config.ini -i rnc.txt


Directory name will contain the current PC time stamp. E.g. 10.11.12.13_logs_20080623_1543 Using new switch "-t" and parameter 0, it is possible to fall back to previous functionality. This way time stamp will not be added to directory name Example: c:\temp\rnclogcol\rnclogcol.exe -c Default_config.ini -i rnc.txt t 0 Directory name would be e.g. 10.11.12.13_logs

If only few logs are needed to be collected see the following chapters for more details and collect logs manually. Rnclogcol *.ini files can be also edited to correspond the need of required logs. Config *.ini is not configured by default to monitor different kind of call scenarios. If logs are needed from call case activate message monitoring by changing the get_monitorings line value from 0 to 1 and give monitoring time in seconds at *.ini file.


5.2 Basic instructions for displaying logs and for monitoring messages

Displaying computer logs and system logs from Chorus units.

The chorus units are: MXU, NIP1, NIS1, DMPG and SFU

ZDDE:,:clog sa; chorus unit log

5.3 Commands for displaying computer logs and system logs from DMX units

The DMX units are: OMU, RSMU, RRMU, ICSU, and GTPU

ZDDE:,:ZGDC; computer log in short format

ZDDE:,:ZGSC; computer log in long format

ZDDE:,:ZSLE; system log of other system errors

ZDDE:,:ZSLP; system log of frozen programs

Print out the POXLIB kernel log from ICSU unit. If case is related SCSI or disks take logs from OMU.

ZDDS:ICSU or OMU,;

ZLP:1,POM

Z1KL

5.4 Commands for message monitoring in DMX units

Check how much memory can be reserved for monitoring buffer: ZDDE:,:ZSB;

In the printout the latter number on the last line tells the amount of the free memory: TOTAL FREE

BYTE CNT BYTE CNT MEMORY USED FOR BUFFERS: 1C8E9000 08D7BEC0

You may RESERVE MAXIMUM HALF of the free memory for message monitoring purposes. The bigger the reserved buffer is the longer it naturally takes to print out its whole contents. It is lot faster if you can use LAN connection to print out the contents of the buffer.

Example: If the FREE BYTE CNT in ZSB printout is 08D7BEC0 (hex), you may reserve a buffer of size 08D7BEC0/2= 46BDF60 =74,178MB (ZOEBR)

Start monitoring for each unit: 1. ZDDS:OMU; 2. ZOEQ: Reserve buffer


3. ZOEBR::302CAAA 4. ZO 5 EC:: Start monitoring from each unit 6. ZOEM:: Stop monitoring from each unit 7. ZOES: Collect monitoring from each unit. Use this (ZOEGF) command if there is expected heavy data from monitoring. After running this command, copy the result file from disk on shadows directory with FTP. After copying the file delete it from disk to save disk space. 8. ZOEGF::W0-/.BIN


6. DATA TO BE COLLECTED IN ALL ABNORMAL SITUATIONS

The following data shall be collected in all abnormal fault situations:

6.1 SW and HW version information

Display the version of the SW build, and the version of all software modules that are run in the computer unit, in which the problem exist. Collect this information with the following MML commands: ZWQO:CR; ZWQV:,;

These commands are applicable to all DMX units (OMU, RSMU, RRMU, ICSU, GTPU), as well as to all Chorus units (= other computer units than the ones listed above).

In case of a Chorus unit, display also the version information of the components included into the boot image with the MML command: ZDDE:,:cat /image/sys_bank/versions.txt;

Display the interchangeability information of the plug-in unit that is attached to the computer unit, in which the problem exists. This information is needed at least in case of faults, which have led to a spontaneous restart of a computer unit. It can be obtained by first finding out the HMS address of the plug-in unit with the MML command ZWFI:P::::,;

After this, enter the HMS address (displayed in the first row of the execution printout) as parameters to the WFL MML command. As an example, if the HMS address displayed in the execution printout of the WFI command is CHMS: 01 SHMS: 2 PPA: 03, then the command for finding out the interchangeability information of the plug-in unit would be ZWFL:P:1,2,3;

6.2 Alarm history

Display the alarm history so that it shows all alarm events from the time period, which starts one hour before the occurrence of the problem situation, and ends one hour after the problem situation was over. You should display the alarm history with the MML command ZAHP:::,,,;

For example, if the problem situation emerged on 2005-11-28 at 00:10 and ceased on 2005-11-28 at 03:20, enter the command ZAHP:::2005-11-27,23-10-00,2005-11-28,04-20-00;

6.3 Working state of units

Display the state and info of the units configured into the system with the MML command: ZUSI:::FULL;


6.4 MML command log

Display the MML command log with the MML command ZIGO:,,;

The start date should correspond to the day preceding the problem occurrence, and the end date should be the day when the problem occurred. For example, if the problem situation emerged on 2005-11-28 at 00:10 and ceased on 2005-11-28 at 03:20, enter the ZIGO:2005-11-27,,2005-11-28;

Note that the userid should have enough access rights to output the MML command log of all users. Also the parameters of ZIGO command should be such that all information of all commands of all users is output.


7. CASE SPECIFIC INSTRUCTIONS In addition to collecting the information mentioned in the chapter above, collect the following information according to the problem case in question:

7.1 System restart

System restart history

If there was a system restart caused by the problem, display the system restart history by sending the following message from the service terminal connected to the lower service terminal connector in the front panel of the OMU. The message shall be sent both in the working OMU and in the spare OMU. ZOS:FF,*,3,,,,,6034,,,FC;

The lower service terminal connector shall be used because the output is displayed only via it.

If displaying of the system restart information is supported in the SW build in question, information on four latest system restarts is displayed on the service terminal in the following format: ======================= Record number 0x0002 ======================

Date and time : 2005-09-16 13:40:45.47 Object unit : SFU-1 Working state : WO-EX Recovery task : fault by alarm Fault observer : 0000 0002 0000 00 Alarm(s) : 1227 FFFF FFFF Fault class : fault_class_t_disturbance_c ======================= End of record number 0x0002 ===============

On the basis of the date and time shown in the first data row you can identify the information, which concerns the system restart that is of interest. The unit displayed as the object unit is the one that is the origin of the system restart. Note that this information may reside in the system restart history of the current spare OMU if there was a switchover of OMU made in connection with the system restart.

Black box(es) of the computer units of interest (store to a file named like omu_0_bbox.txt):

You should display the black box of both OMUs and the black box (DMX unit) / system dump and logs (Chorus unit) of the object unit shown in the system restart history. If the system restart history is not supported in the SW build in question, try to figure out (e.g. on the basis of the alarm history) the object unit by your own and display the black box / system dump & logs of that unit as instructed in the section Spontaneous restart of a DMX unit or Spontaneous restart of a Chorus unit, respectively.

Note that there may be more than one system restart occurring in a row in a certain fault situation. In that case you also should attach to the outage report all the stored black box information of the DMX units of interest (i.e., of OMUs and of system restart object units) from the W0-/var/crash/ directory on the disk. This directory contains max. 12 latest formed black boxes for each DMX unit. The black box files of a certain


unit, as well as their control file, can be identified on the basis of the physical address of the unit, which is used as a part of the name of these files. For example, as the physical address of the OMU-1 is 0001 (obtained from the output of the USI MML command), the back box files of the OMU-1 are named as 0001_00.bbox 0001_11.bbox, and the control file is named as 0001.seq.

7.2 Spontaneous restart of a DMX unit (OMU, RSMU, RRMU, ICSU, GTPU)

Black Box(es) of the restarted unit (store to a file named like omu_0_bbox.txt):

Display the black box with the following MML commands:

ZDDE:,:ZLP:1,BOX,Z1U,ZL:1;

ZDDE:,:ZLP:1,BOX,Z1U,ZL:1;

Note that there may be more than one unit restart occurring in a row in a certain fault situation. In that case you should attach to the problem report all the stored black box information of the DMX unit in question from the W0-/var/crash/ directory on the disk. This directory contains max. 12 latest formed black boxes for each DMX unit. The black box files of a certain unit, as well as their control file, can be identified on the basis of the physical address of the unit, which is used as a part of the name of these files. For example, as the physical address of the OMU-1 is 0001 (obtained from the output of the USI MML command), the back box files of the OMU-1 are named as 0001_00.bbox 0001_11.bbox, and the control file is named as 0001.seq.

Printout of the MML command:

ZDVL::CONT:::::,;

For example: ZDVL::CONT:::::OMU,0;

Print out the POXLIB kernel log from ICSU unit.

ZDDS:ICSU,;

ZLP:1,POM

Z1KL

7.3 Spontaneous restart of a CHORUS unit (MXU, A2SP, NIP1, NIS1, SFU and DMPG)

"Tip! Sysdump and bblog data are stored in RAM memory of a Chorus unit. The data is lost if the computer unit is powered down."

System dump of the restarted unit:

Display system dump and the stored computer logs and OS logs from the restarted unit with the following commands

ZDDE:,:loadext sysdump;


ZDDE:,:sysdump W;

ZDDE:,:loadext bblog;

ZDDE:,:bblog c a ; where = 210

ZDDE:,:bblog o ; where = 210

The parameter value y (where y = 210) in the commands above defines the order number of the log. With the default parameter value 2, log data written the previous unit restart is shown, and so on. Please make sure that you define the value y correctly in order to display the logs that have been written just before the unit restart(s) of interest.

Printout of the MML command:

ZDVL::CONT:::::,;

For example: ZDVL::CONT:::::DMPG,34;


8. RESTART OF DSP/DMPG

Tip! Sysdump and bblog data are stored in RAM memory of a Chorus unit. The data is lost if the computer unit is powered down.

8.1 RNC active alarms and alarm history

RNC active alarm:

ZAAP;

RNC active alarm for one specific unit:

ZAAP:,;

RNC alarm history from start date (default one is current date):

ZAHP:::[start date]:;

RNC alarm history for one specific unit:

ZAHP:,;

8.2 Plug-in Unit HW information

ZWFL:P;

and

ZWDI;

8.3 All Function Unit states and information

ZUSI:::FULL;

8.4 DSP service statistics information

ZWPS:U:UNIT=,INDEX=;

8.5 Collect computer log in destination DMPG

ZDDE:,:clog sa;

8.6 Collect restarted DSP blackbox information

1. Start remote session to the DMPG where the restarted DSP belongs to

ZDDS:DMPG,;

2. Load dspbb service terminal extension into DMPG ram disk


dxfload -n CSXDSPBB -d -u 0 -o /bin/dspbb.gz

3. Dump DSP blackbox by specifying restarted dsp index (0 7)

dspbb c

8.7 Collect sysdump information for suspected faulty DMPG unit

1. Start a remote session to the suspected DMPG

ZDDS:DMPG,;

2. Load sysdump service terminal extension into DMPG ram disk

loadext sysdump

3. Print all system dump information from all available restarts (sysdump can store at max ten latest restarts)

sysdump W

8.8 Collect bblog information of suspected faulty DMPG unit

1. Start a remote session to the suspected DMPG

ZDDS:DMPG,;

2. Load bblog service terminal extension into DMPG ram disk

loadext bblog

3. Dump computer logs where file id is between 2 to 10

bblog c a

4. Dump operating system logs where file id is between 2 to 10

bblog o

8.9 DMPG communication processor blackbox information

1. Start a remote session to the suspected faulty DMPG

ZDDS:DMPG,;

2. Load dprdump service terminal extension into DMPG ram disk

loadext -n CSXDPRDU /bin/dprdump.gz

3. Dump all saved memory regions from the system dump, where nb indicates the restart number 1 - 10.


dprdump d -4 r

8.10 Collect DSP service related information


ZDDS:DMPG,;

2. Load rmdext service terminal extension into DMPG ram disk

loadext rmdext

3. Get current DSP state and other information

rmdext d 1

4. Get current DSP service related information

rmdext d 2

5. Get current DSP service instance information

rmdext d 3

6. Get current connection list

rmdext d 5

7. Get current channel list

rmdext d 6

8. Get RMDPRB event log

rmdext l

9. Dump RMDPRB log under directory /DSP

cat /DSP/rmd_log.txt

10. Dump DSP fatal error log under directory /DSP

cat /DSP/dsp_fatal_errors.txt

11. Dump RMDPRB severe error log under directory /DSP

cat /DSP/rmd_serv.txt

8.11 Collect PDRACT channel related information


ZDDS:DMPG,;


2. Dump all client information

pdrext cli

3. Dump all connection status

pdrext con

4. Dump detail information from specific connection

pdrext con

5. Dump channel status

pdrext dsp

8.12 Collect NPGE, NPGEP, NPS1 and NPS1P units related information

From APP in any traffic problem: jane rspgstat jane kuostat jane rsppstat all -z jane hoststat jane fppstat 0 jane fppstat 1 jane bonstat 2 0 jane rmregd 0.0.0 30 jane smon jane hwstatus -d From M/D-FPGA and SD2G5 for all traffic problems: loadext -n MDXEXTXX mdxext loadext -n SD2EXTXX sd2ext

mdxext -stat mdxext -info sd2ext -lstat sd2ext -pstat sd2ext -istat sd2ext -bpmap If the problem is related to NPGE(P), the following input is needed: jane macreg g 1 jane macreg g 2


jane bonstat 1 0x2000 jane bonstat 1 0x2001 Adding information about IP CAC related B/W for NPGx PIU with kdesk command when investigating IP based transport issues. Eg if_id|rt_id|used__bw|commit_bw|route__bw|rtp_num|gtp_num|fp_num

ZDDE:NPGEP,:"loadext kdesk";

kdesk -r -t

kdesk -r -i If the problem is related to NPS1(P), the following input is needed: loadext pq2atm pq2atm -u Information about ATM connections: jane pratmconn jane pratmconn Information about NCIDS: jane kuostat -c8 jane prncid -v If there are many NCIDs you can limit the amount of printed NCIDs by specifying mphy, vpi and/or vci: jane prncid -v Also if you know the specific NCID where the problem is then it is good to print all the details from the NCID with: jane prncid -a CID can be seen from the previous jane prncid command. In case of dynamic calls the CID values might change quite rapidly, so this is not so useful in that case.


8.13 Monitor DMX message in suspected faulty DMPG


ZDDS:DMPG,;

2. Load dmxmon service terminal extension into DMPG ram disk

loadext dmxmon;

3. Monitoring DMX message for specific family IDs

dmxmon [f fam0 fam1 fam9];

8.14 Monitor DSP message and DMX message to/from suspected faulty DSP

1. Start a remote session to the DMPG which suspected faulty DSP belongs to

ZDDS:DMPG,;

2. Monitor DSP message and DMX message in both directions for specific DSP

p2dext monitor [p ]

8.15 Check SW version of one specific program block

ZWQV:DMPG,:;

8.16 Collect diagnostics report history

ZUDH;


9. BTS INITIALIZATION CASE

Collect the following information in BTS initialization cases, for example if WBTS cells remain in blocked state, or if there are problems in changing / editing parameters via RNW object browser. Message monitoring from OMU with families REK (050B), REZ (04FA), RAK (04EE), VEX (04EF) and ENP (0553).

SR:(OFAM=050B,04FA,04EE,04EF,0553)AND(NOT(NUM=0001,0002))

BTS initialization ICSU monitoring conditions and families: RRC (04FD), NBA (04FE), and BRM (04FB).

SR:(OFAM=04FE,04FD,04FB)AND(OPRO140)))AND(NOT((NUM>9000)AND(NUM

A2SU Unit

A2SPs SUBRACK

0 0..3 1 (RNAC)

1 4..7 3 (RNAC)

2 8..11 4 (RNAC)

3 12..15 1 (RNBC)

4 16..19 2 (RNBC)

5 20..23 3 (RNBC)

6 24..27 4 (RNBC)

After finding out A2SP handling the Iub AAL2 path take ddesk logs from affected A2SP service terminal:

ZDDE:A2SP,:loadext ddesk;

ZDDE:A2SP,:ddesk a pq_alg_ncid p aal2xc;

ZDDE:A2SP,:ddesk a pq_alg_ncid p ncid_all;

ZDDE:A2SP,:ddesk a pq_alg_ncid p ncid_hsdpa;

ZDDE:A2SP,:ddesk p msg_counters;

ZDDE:A2SP,:ddesk a pq_alg_ncid p traffic;

ZDDE:A2SP,:ddesk -a pq_alg_ncid p ncid;

can be found from the output of command "ddesk -a pq_alg_ncid -p ncid_ob" in VPI column


10. CALL CASE

Collect the following information by macro if there are problems related to calls, for example unsuccessful registrations, unsuccessful calls, unsuccessful handovers, unacceptable KPI values, or call scenarios not working as expected.

Message monitoring from ICSUs with families NRM (04B7), RRC (04FD), NBA (04FE), BRM (04FB), HA3 (507), IUR (508), IUV (509) and RBR (54E). IUR is needed in Inter-RNC handover cases only.

To reduce the size of the monitoring log, following monitoring condition can be used: SR:(OFAM=4B7,4FD,4FE,4FB,507,508,509,54E)AND(NOT(NUM=0001,0002,0D5BA,0C5DF,0D7A6,0D7A8,08037,08038,09904,09835,09836,D9DA,0CEC9,0CECA))

Example, calls monitoring conditions:

SR:((OFAM=04FD,509)AND(NOT(NUM=8037,8038,0D7A6,0D5BA,9904,9835,9836,0AB2E,0D5B8)))OR(NUM=0C5DF,0E47B)

Start the monitoring. Produce the fault. Stop the monitoring and print it out. Save log file with name where units name and number are mentioned.

Attach the Nethawk trace file(s) with configuration file included (*.gfc) or in text format (*.txt) if they exist from the fault case. If it is used for Nethawk M5 analyzer take a ( *.rec ) log and Export Online Configuration ( *.wcf ) log.

Other needed info from MXU unit:

Check with MML command ZRRI and ZLJI the affected AAL2 User plane info (Interface, VPI and VCI)

Check the WO-EX RSMU: ZUSI:RSMU;

Printout RM2PRB error logs from WO-EX RSMU: ZDDE:,:ZLP:3,LGU,Z3RIE:*;

Give to the command the affected AAL2 User plane info from WO-EX RSMU:

ZDDE:,:Z3IT:,,;

Check from printout the A2SP number and in MML check which A2SU unit is handling the A2SP: ZUSI:A2SU,::FULL;


A2SU Unit

A2SPs SUBRACK

0 0..3 1 (RNAC)

1 4..7 3 (RNAC)

2 8..11 4 (RNAC)

3 12..15 1 (RNBC)

4 16..19 2 (RNBC)

5 20..23 3 (RNBC)

6 24..27 4 (RNBC)

After finding out A2SP handling the Iub AAL2 path take ddesk logs from affected A2SP service terminal:

ZDDE:A2SP,:loadext ddesk;

ZDDE:A2SP,:ddesk a pq_alg_ncid p aal2xc;

ZDDE:A2SP,:ddesk a pq_alg_ncid p ncid_all;

ZDDE:A2SP,:ddesk a pq_alg_ncid p ncid_hsdpa;

ZDDE:A2SP,:ddesk p msg_counters;

ZDDE:A2SP,:ddesk a pq_alg_ncid p traffic;

ZDDE:A2SP,:ddesk -a pq_alg_ncid p ncid;


11. SYNCHRONIZATION CASE

Collect additional logs in the following way if there is some problem with RNC synchronization. Store the collected information to a file named e.g. as synchronization_problem_case.txt.

SYPROC (3DF) message monitoring from WO-OMU when the problem exists (15 minutes period). If the problem is seen in MML command output, message monitoring must be ongoing during MML command execution.

MML command ZDYI few times and also all other MML command outputs where the problem is seen.

Following service terminal commands in WO-OMU: ZL:F; ZLP:F,FUT; ZFTN:A1L; ZFTN:A2P; ZFTN:A2T; ZFTN:GS2;

Active alarms ZAAP and alarm history ZAHP

OMU computer logs ZGSC (both OMUs).


12. ESA 12/24 LAN SWITCH CASE

In case of problems related to ESA 12/24 LAN switch , collect the following information and store them to a file named e.g. as ESA_problem_case.txt. For ESA24: 1) Take a console or telnet connection to ESA24 LAN switch. When ESA24 card has 3.5.0 or older software, following commands can not perform. Those commands are available since 3.14.7 software. More details in Technical Note 86. 2) Give following commands: ESA24>enable ESA24#show tech-support ESA24#show log buffer ESA24#show configuration-history For ESA12: Take a console or telnet connection to ESA12 LAN switch and display all possible settings under different menus.


13. OMS CASE

13.1 General case

If the problem concerns OMS, collect the following logs and transfer logs to local PC. Open Parameters Tools on Application Launcher and change dwFlags to 3 for logs trace. fsFragmentId=OMS

omsfragmentId=System omsFragmentId=TraceConfig

omsFragmentId=Any omsFragmentId=dwFlags

dwFlags parameter value 0 means trace not in use, 3 means trace is in use. Buffer size (dwLogSizeinkB) is as default 15360 kB and can be change if needed. Re-procedure a case and run log collect script zcollectlogs as root user (su -). The script is already available RN3.0 CD3.0 onwards in OMS. The script includes TN38 attachments and can be copied before CD3.0 to OMS and to give operation rights by chmod +x zcollectlogs.sh and run it. Output log will create into a directory where the script is run.

zcollectlogs o .tgz --full Example

# zcollectlogs o rnclogs.tgz full Windows environment: Log in to OMS as Nemuadmin user using puTTY tool. Change to the root user by entering su command. Copy files first into the /home/Nemuadmin directory and then transfer files to local PC by WinSCP tool as Nemuadmin user. cp //.tgz /home/Nemuadmin/ E.g. cp /root/rnclogs.tgz /home/Nemuadmin

Linux environment: Log in to OMS as Nemuadmin user. Transfer files to local PC with the following command. scp //.tgz @:/

E.g. scp /root/rnclogs.tgz @:/

13.2 Measurement case

Logs from OMU: ZIFO:OMU:GENE00:;


OMUs computer log ZGSC; Collect following files from OMU disk by FTP. /RUNNING/ASWDIR/FTPSERVR.XML and additionally /RUNNING/ASWDIR/FTPSERVR.OLD if that exists

Collect also zcollectlogs as instructed in chapter 13.1. and provide data from Netact or raw data to show which measurements are missing and when.


14. MESSAGE MONITORING FROM OMU WITH FAMILIES FOR DIFFERENT PROBLEM SITUATIONS

In all cases include following families: REK (050B), REZ (04FA), RAK (04EE) and VEX (04EF): SR:(OFAM=50B,4FA, 4EE, 4EF)

RNW measurement collection: PMI (04ED): SR:(OFAM=4ED)

Transport & HW measurement collection: AMN (0391): SR:(OFAM=0391)

Data saving to disk service: SLY (04C7) and VID (0024): SR:(OFAM=4C7, 0024)

Data transfer communication with OMS (via EXW): SLY (04C7): SR:(OFAM=4C7)

EMT-interface implementation: EXW (0433): SR:(OFAM=433)

Start the monitoring. Produce the fault. Stop the monitoring and print it out. Save log file with name where units name and number are mentioned.

If the fault has something to do with COCO, message monitoring from OMU with families MNM (03C6), ARM (02F7 VEX (04EF), RBR (054E): SR:(OFAM=03C6, 02F7,04EF,054E).

15. NOTE

16. REFERENCES



Disclaimer

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Siemens Networks customers only for the purposes of the agreement under which the document is submitted, and no part of it may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Siemens Networks. The documentation has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation.

The information or statements given in this documentation concerning the suitability, capacity, or performance of the mentioned hardware or software products are given as is and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Nokia Siemens Networks and the customer. However, Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which may not be covered by the document.

Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL NOKIA SIEMENS NETWORKS BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA, THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT.

This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws.

The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia Corporation. Siemens is a registered trademark of Siemens AG.

Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only.

Copyright Nokia Siemens Networks 2010. All rights reserved.

1. COMPATIBILITY / DEPENDENCIES TO OTHER PRODUCTS2. KEYWORDS3. SUMMARY4. DETAILED DESCRIPTION5. SOLUTION / INSTRUCTIONS5.1 Using the log collection macro rnclogcol.exe5.2 Basic instructions for displaying logs and for monitoring messages5.3 Commands for displaying computer logs and system logs from DMX units5.4 Commands for message monitoring in DMX units

6. DATA TO BE COLLECTED IN ALL ABNORMAL SITUATIONS6.1 SW and HW version information 6.2 Alarm history6.3 Working state of units6.4 MML command log

7. CASE SPECIFIC INSTRUCTIONS7.1 System restart7.2 Spontaneous restart of a DMX unit (OMU, RSMU, RRMU, ICSU, GTPU)7.3 Spontaneous restart of a CHORUS unit (MXU, A2SP, NIP1, NIS1, SFU and DMPG)

8. RESTART OF DSP/DMPG8.1 RNC active alarms and alarm history8.2 Plug-in Unit HW information8.3 All Function Unit states and information8.4 DSP service statistics information8.5 Collect computer log in destination DMPG8.6 Collect restarted DSP blackbox information8.7 Collect sysdump information for suspected faulty DMPG unit8.8 Collect bblog information of suspected faulty DMPG unit8.9 DMPG communication processor blackbox information8.10 Collect DSP service related information8.11 Collect PDRACT channel related information8.12 Collect NPGE, NPGEP, NPS1 and NPS1P units related information8.13 Monitor DMX message in suspected faulty DMPG8.14 Monitor DSP message and DMX message to/from suspected faulty DSP8.15 Check SW version of one specific program block8.16 Collect diagnostics report history

9. BTS INITIALIZATION CASE10. CALL CASE11. SYNCHRONIZATION CASE12. ESA 12/24 LAN SWITCH CASE13. OMS CASE13.1 General case13.2 Measurement case

14. MESSAGE MONITORING FROM OMU WITH FAMILIES FOR DIFFERENT PROBLEM SITUATIONS15. NOTE16. REFERENCES

ts-rnc-sw-038-i13

Documents

unit hw information

dmx units

spontaneous restart

system restart

dmx unit omu

system logs

basic instructions

solution instructions