shelfmanager esw update guide to 06166 and with mml to 06169.doc

ATCA Shelf Manager eSW update guide to 06166 manually and then to 06169 with MML

Author: Hannu Sainio

Draft

Version 1.11-0

This document requires that 06163 levels of Shelf Manager eSW are installed to ATCA HW.

For internal use only

ATCA Shelf Manager eSW UPDATE GUIDE 2 (16)


Change History

Issue Date Handled by Comments1.0-0 25.10.2010 Hannu Sainio First version1.1-0 26.10.2010 Hannu Sainio Second version1.2-0 10.11.2010 Hannu Sainio Some small issues fixed1.3-0 24.11.2010 Hannu Sainio Update to 061691.4-0 9.12.2010 Hannu Sainio Second shelf and one troubleshooting case added1.5-0 20.12.2010 Hannu Sainio More notes about checking update 06166 version

added to avoid later problems with fru data update and ShelfManager stability.

1.6-0 27.12.2010 Hannu Sainio Kernel and U-boot update fixed to update to 06166 instructions page 16, point 7.

1.7-0 29.12.2010 Hannu Sainio Added improvements from TS. Two new troubleshooting cases added.

1.8-0 30.10.2010 Hannu Sainio Version and frudata printouts added1.9-0 3.1.2011 Hannu Sainio Frudata versions added.1.10-0 12.1.2011 Hannu Sainio Troubleshooting additions. One addition to IIMSTE

check.1.11-0 27.1.2011 Hannu Sainio Troubleshooting additions. One record could be too

short.



TABLE OF CONTENTS:

1. Check versions and update to 06166 if necessary.........................................................................41.1 Check U-boot, Linux kernel and RFS versions.........................................................................41.2 Frudata check...........................................................................................................................51.3 Back up frudata before updating..............................................................................................51.4 Update to 06166 version if not up to date.................................................................................6

2. Check HPDL activation...................................................................................................................63. Set Minimum Fan Level to 8...........................................................................................................74. Update ShelfManager eSW with MML to 06169.............................................................................8

4.1 Check and update Shelf Manager update files.........................................................................8For 2 nd shelf (Only needed in case 2 shelfs are in use):...................................................................84.2 Update Shelf Manager with MML commands...........................................................................8

5. Troubleshooting............................................................................................................................135.1 HPDL activation......................................................................................................................135.2 Connection problem between Shelf Manager and OMU........................................................135.3 IHWMAN fails at switchover and Shelf Manager is in reset loop...........................................145.4 IP of Shelf Manager is lost at switchover................................................................................15



Instruction for Shelf Manager eSW update first to 06166 and then to 06169 level.

1. CHECK VERSIONS AND UPDATE TO 06166 IF NECESSARY.

Shelf Manager must have 06166 versions installed or newer to enable updating with MML commands.Check is performed in points 1.1 and 1.2.

NOTE: If versions are older than 06163 then this document does not cover support for such update. It will require another procedure before update is possible. Please contact author in such case.

1.1 Check U-boot, Linux kernel and RFS versions.

Those should be 63998-06166.

version

# version

Current BootImage Name: 63998-06166 U-Boot 1.1.4Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Firmware (uncompressed)Data Size: 185316 Bytes = 180.97 kB = 0.18 MBLoad Address: 0x9FC00000Entry Point: 0x00000000

Current KernelImage Name: 63998-06166 kernel Linux-2.4.26Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Kernel Image (gzip compressed)Data Size: 882342 Bytes = 861.66 kB = 0.84 MBLoad Address: 0x80100000Entry Point: 0x802D4040

Current RFSImage Name: 63998-06166 RFS 2.6.4.2Created: Fri Nov 6 15:34:56 2009Image Type: MIPS Linux RAMDisk Image (gzip compressed)Data Size: 3878850 Bytes = 3787.94 kB = 3.70 MBLoad Address: 0x00000000Entry Point: 0x00000000

Alternate BootImage Name: 63998-06166 U-Boot 1.1.4Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Firmware (uncompressed)Data Size: 185316 Bytes = 180.97 kB = 0.18 MBLoad Address: 0x9FC00000Entry Point: 0x00000000

Alternate LinuxImage Name: 63998-06166 kernel Linux-2.4.26Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Kernel Image (gzip compressed)Data Size: 882342 Bytes = 861.66 kB = 0.84 MBLoad Address: 0x80100000Entry Point: 0x802D4040



Alternate RFSImage Name: 63998-06166 RFS 2.6.4.2Created: Fri Nov 6 15:34:56 2009Image Type: MIPS Linux RAMDisk Image (gzip compressed)Data Size: 3878850 Bytes = 3787.94 kB = 3.70 MBLoad Address: 0x00000000Entry Point: 0x00000000

1.2 Frudata check

Frudata check shall be done for Shelf and ACB Carrier boards. Old systems might have ACB-IV carrier board.

If versions are older than 06163 levels then update is not possible even with manual instruction. In that case another procedure is needed.

Shelf FRU Area Storage File for ADX -> 11596050HABIN.binACB-IV Carrier -> 21596180HABIN.binACB-V Carrier -> 21596309HABIN.bin

Different FRU data versions:

06159: 1159650DABIN.bin06159: 21596180BCBIN.bin

06162: 11596050DBBIN.bin06162: 21596180BCBIN.bin06162: 21596309ABBIN.bin

06163:11596050HABIN.bin06163: 21596180HABIN.bin06163: 21596309HABIN.bin

06165: 11596050HBBIN_adx.bin06165: 21596180HABIN.bin06165: 21596309HABIN.bin

HCBIN for shelf frudata and HBBIN for shelfmanagers frudata should be found (06166 levels). If versions are older then manual update is needed (point 1.3 & 1.4).

You can display the content of frudatas with clia fruinfo

clia fruinfo 20 254 | grep Programmer <- frudata of the shelfFRU Programmer File ID = 11596050HCBIN.bin

clia fruinfo 10 0 | grep Programmer <- frudata of the left side shelfmanagerFRU Programmer File ID = 21596309HBBIN.bin

clia fruinfo 12 0 | grep Programmer <- frudata of the right side shelfmanagerFRU Programmer File ID = 21596309HBBIN.bin

1.3 Back up frudata before updating

Backup Frudata before uptading.



Note! Before updating FRUDATA you should always have a backup of it in a safe place! In case if something goes wrong during the upgrade, you can restore the previous version. FRU Information from the specified FRU can be read and stored into a file with command frudatar. Take the backups and move them to OMUs disk with ftp put command.

# clia frudatar 20 254 FRU20_254.bin# clia frudatar 10 0 FRU10_0.bin# clia frudatar 12 0 FRU12_0.bin

- Take console connection to active shelf manager- Take ftp session towards Omu using the messages bus address of omu; ftp 10.200.200.1- Use binary mode in the file transfer, run bin -command- Transfer the files to OMU's HDD, run put <file name> -command- Verify that the files are on the omu, with for example running ZMX:W0-/ -command on active OMU service terminal

1.4 Update to 06166 version if not up to date

Install the 06166 version now (HCBIN for shelf frudata and HBBIN for shelfmanagers frudata) if versions are not on the 06166 level. Update zip for 06166 can be found via ShareNet Link.

ATCA eSW update granny guide from page 12 onwards. Update the parts which are not on the 06166 level.

2. CHECK HPDL ACTIVATION.

NOTE: HPDL must be active before proceeding with updating Shelf Manager. If not active then fru data may get corrupted.

Check both Shelf Managers with version and getenv rc2 commands.(Chassis product ID can be seen only on the active side)

version

...Pigeon Point Shelf Manager Command Line Interpreter

Pigeon Point Shelf Manager ver. 2.7.0Pigeon Point and the stylized lighthouse logo are trademarks of Pigeon Point Systems.Copyright (c) 2002-2010 Pigeon Point SystemsAll rights reservedBuild date/time: Jun 16 2010 09:13:22Carrier: HPDL/NSNCarrier subtype: 0; subversion: 0Cooling management: DefaultCarrier Product ID: 21596309Chassis Product ID: 11596050_mfl05 (minimum fan level = 5)

# getenv rc2rc2=/etc/rc.nsn-hpdl#

If not up to date check troubleshooting 5.1. Do not proceed with update if HPDL is not active. Contact Hannu Sainio if issue was not solved.

https://sharenet-ims.inside.nokiasiemensnetworks.com/Open/D424848331



3. SET MINIMUM FAN LEVEL TO 8

This is done only to active Shelf Manager.

Check fans level:

# clia fans

Pigeon Point Shelf Manager Command Line Interpreter

20: FRU # 3Current Level: 8 Minimum Speed Level: 5, Maximum Speed Level: 15 Dynamic minimum fan level: 520: FRU # 4Current Level: 8 Minimum Speed Level: 5, Maximum Speed Level: 15 Dynamic minimum fan level: 520: FRU # 5Current Level: 8 Minimum Speed Level: 5, Maximum Speed Level: 15 Dynamic minimum fan level: 5

Set Minimum fan level to 8.

cd /var/nvdata/

ls -lrt

clia frudataw –d 20 254 /var/nvdata/11596050ABDATmfl08.bin.gz

After installing the hpdl file you must execute a simultaneous reboot of both ASMGR-A (running via local Command line interface) or power cycle the chassis

Check after reboot

clia fans



4. UPDATE SHELFMANAGER ESW WITH MML TO 06169

4.1 Check and update Shelf Manager update files

You need to check that Shelf Manager update files are named right before updating.This is done with IIMSTE tool.

ZLP:1,IIM

ZL:1

Z1RORG:1,41,1,0,FF,3,1:100:88

sw_fw_name: asmgr-a_sc_frudsw_fw_ver: 06166sw_fw_build: DDMMYYYYsw_fw_upd: 20101021sw_fw_chk:

Completion code: 00Command Completed Normally.

Z1RORG:1,41,1,0,FF,3,2:100:88

sw_fw_name: asmgr-a_sc_frudsw_fw_ver: 06166sw_fw_build: DDMMYYYYsw_fw_upd: 20101021sw_fw_chk:

Completion code: 00Command Completed Normally.

File names should end “frud” and sw_fw_ver “06166”. Id there is something else like “shmc” you need fix it with these two commands.

Z1ROWG:1,41,1,0,FF,3,1:100:88:asmgr-a_sc_frud:06166:DDMMYYYY:20102010:12345678


For 2 nd shelf (Only needed in case 2 shelfs are in use):



4.2 Update Shelf Manager with MML commands

1. Check IPMI versions

ZWDD:UT=SHMU,:;

EXECUTION STARTED

2 VALID PLUG-IN UNITS FOUND



EXECUTION STARTED AT 2010-10-25 14:10:16 50% DONE, PLEASE WAIT 0 H, 9 MIN AND 59 SEC100% DONE, PLEASE WAIT 0 H, 0 MIN AND 0 SEC

FUNCTIONAL PLUG-IN UNIT VERSION UNIT TYPE INDEX CURRENT -> AFTER UPDATE ----------- ------------- -------------------------------- SHMU-0 ASMGR_A-0 asmgr-a_sc_shmc_06166 -> asmgr-a_sc_shmc_06169 BLABMNT_1038 SHMU-1 ASMGR_A-0 asmgr-a_sc_shmc_06166 -> asmgr-a_sc_shmc_06169 BLABMNT_1038

2. Check the SHMU status and make SHMU-0 spare if needed

ZUSI:SHMU;ZUSC:SHMU,0:SP;

3. Update spare SHMU

ZWDF:UT=SHMU,UI=0,:::;

4. Give new password to updated SHMU via serial connection

shmm500 login: root

Choose a new password.

Changing password for rootEnter the new password (minimum of 5, maximum of 8 characters)Please use a combination of upper and lower case letters and numbers.Enter new password: shmm4nsn /* Enter here a password */Re-enter new password: shmm4nsn /* Confirm here the entered password */ Password changed.

5. Switchover

ZUSC:SHMU,1:SP;

6. Update new spare SHMU

ZWDF:UT=SHMU,UI=1,:::;

7. Check from both ShelfManagers that U-boot, Linux Kernel and RFS are updated to 63998-06169.

version

Current BootImage Name: 63998-06169 U-Boot 1.1.4Created: Wed Jun 23 12:39:18 2010Image Type: MIPS Linux Firmware (uncompressed)Data Size: 185316 Bytes = 180.97 kB = 0.18 MBLoad Address: 0x9FC00000Entry Point: 0x00000000

Current KernelImage Name: 63998-06169 kernel Linux-2.4.26Created: Wed Jun 23 12:39:18 2010Image Type: MIPS Linux Kernel Image (gzip compressed)Data Size: 882884 Bytes = 862.19 kB = 0.84 MB



Load Address: 0x80100000Entry Point: 0x802D6040

Current RFSImage Name: 63998-06169 RFS 2.7.0Created: Wed Jun 23 12:39:26 2010Image Type: MIPS Linux RAMDisk Image (gzip compressed)Data Size: 5022046 Bytes = 4904.34 kB = 4.79 MBLoad Address: 0x00000000Entry Point: 0x00000000

Alternate BootImage Name: 63998-06166 U-Boot 1.1.4Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Firmware (uncompressed)Data Size: 185316 Bytes = 180.97 kB = 0.18 MBLoad Address: 0x9FC00000Entry Point: 0x00000000

Alternate LinuxImage Name: 63998-06166 kernel Linux-2.4.26Created: Fri Nov 6 15:34:49 2009Image Type: MIPS Linux Kernel Image (gzip compressed)Data Size: 882342 Bytes = 861.66 kB = 0.84 MBLoad Address: 0x80100000Entry Point: 0x802D4040

Alternate RFSImage Name: 63998-06166 RFS 2.6.4.2Created: Fri Nov 6 15:34:56 2009Image Type: MIPS Linux RAMDisk Image (gzip compressed)Data Size: 3878850 Bytes = 3787.94 kB = 3.70 MBLoad Address: 0x00000000Entry Point: 0x00000000

Pigeon Point Shelf Manager Command Line Interpreter

Pigeon Point Shelf Manager ver. 2.7.0Pigeon Point and the stylized lighthouse logo are trademarks of Pigeon Point Systems.Copyright (c) 2002-2010 Pigeon Point SystemsAll rights reservedBuild date/time: Jun 16 2010 09:13:22Carrier: HPDL/NSNCarrier subtype: 0; subversion: 0Cooling management: DefaultCarrier Product ID: 21596309Chassis Product ID: 11596050_mfl08

8. Check FRU package version

WDP:UT=SHMU,::;

EXECUTION STARTED






COMMAND EXECUTED

9. Update FRU data to active SHMU.

ZWDQ:UT=SHMU,UI=0::;

LOADING PROGRAM VERSION 11.10-0

EXECUTION STARTED


CONFIRM COMMAND EXECUTION: Y/N ? Y

EXECUTION STARTED AT 2010-10-21 15:28:00100% DONE, PLEASE WAIT 0 H, 0 MIN AND 0 SEC

FUNCTIONAL PLUG-IN UNITUNIT TYPE INDEX RESULT----------- ------------- -------SHMU-0 ASMGR_A-0 FRU BIN UPDATED, SIMULT SHMG REBOOT NEEDED

Reboot both ShelfManagers

Check that fru data has changed 11596050HCBIN->11596050HDBIN

SHMU 0

# clia fruinfo -x 0x20 0xfe | grep ProgrammerFRU Programmer File ID = 11596050HDBIN.bin

SHMU 1

# clia fruinfo -x 0x20 0xfe | grep Programmer# FRU Programmer File ID = 11596050HDBIN.bin

10. Check versions with MML. IPMI should be 06169 and FRU package 06168.

ZWDD:UT=SHMU;



EXECUTION STARTED




COMMAND EXECUTED

ZWDP:UT=SHMU,;

EXECUTION STARTED



FUNCTIONAL PLUG-IN UNIT VERSION UNIT TYPE INDEX CURRENT -> AFTER UPDATE ----------- ------------- -------------------------------- SHMU-0 ASMGR_A-0 asmgr-a_sc_frud_06168 -> asmgr-a_sc_frud_06168 BLABMNT_1028 SHMU-1 ASMGR_A-0 asmgr-a_sc_frud_06168 -> asmgr-a_sc_frud_06168 BLABMNT_1028

11. Check that Shelf Managers work and switchover works

With serial connection:

clia shmstatus –v

clia session

clia fru 10

clia fru 12

clia fru 20

With MML:

ZUSI:SHMU;

ZAHO:SHMU;

ZUSC:SHMU,<wo-unit>:SP;



5. TROUBLESHOOTING

5.1 HPDL activation

If HPDL is not activated it shall be activated before any updates.

HPDL activation should be done with command below to both Shelfmanagers.

setenv rc2 /etc/rc.nsn-hpdl

After that simultaneus reboot of ShelfManagers.

If this leads to situation where one Shelf Manager has HPDL active and other does not.In this case pull out the one that was activated. Then try activation to the one where activation failed.If it works put the other ShelfManager back in.

If this does not work then go back to original version: setenv rc2 /etc/rc.nsn and simultaneus reboot of ShelfManagers.

Do not leave Shelf Managers so that one is using HPDL and other one is not. They will not communicate with eachother in this case.

5.2 Connection problem between Shelf Manager and OMU

This may be seen so that IIMSTE commands do not work or ZWDD command gives an error.

In this case check following:

clia shmstatus –v

clia session

clia fru 10

clia fru 12

clia fru 20

ifconfig

Fixing:

1. Fix status of fru’s.

2. Check Shelf Manager ip addresses. If not ok do reset with command

“clia dhcp restart”

If this does not work then set addresses with ifconfig.

3. If fru’s are active and ip addreses are OK and “clia session” does not show any session then restart OMU’s.

Below is right IP Config:

Left:ifconfig eth0 10.200.200.10 netmask 255.255.255.0ifconfig eth1 10.200.201.10 netmask 255.255.255.0



Rightifconfig eth0 10.200.201.11 netmask 255.255.255.0ifconfig eth1 10.200.200.11 netmask 255.255.255.0

Active if left

ifconfig eth0:0 10.200.202.3 netmask 255.255.255.248ifconfig eth1:0 10.200.202.11 netmask 255.255.255.248

if right

ifconfig eth0:0 10.200.202.11 netmask 255.255.255.248ifconfig eth1:0 10.200.202.3 netmask 255.255.255.248

4. OMU switchover

5. If still problem exists then take these printouts from both ShelfManagers and send those to author. We will check if one record is too short. That can be patched but printouts are needed to know what to patch there.

clia fruinfo -x 0x10 0clia fruinfo -x 0x12 0

clia frudata 0x10 0clia frudata 0x12 0

5.3 IHWMAN fails at switchover and Shelf Manager is in reset loop

In this case switchover after update fails and Shelf Manager is in reset loop.

/*** STATE TRANSITION WO-EX -> SP ***//*** NOT CONTROLLED ***/

/*** ERROR INFO FILLED BY OBSERVER ***//*** DX ERROR: 10141 ***//*** PASSIVE UNIT STATE INCORRECT ***/

/*** OBSERVED BY PROGRAM BLOCK : RYSSIX, 58EH ***//*** FAILED UNIT : OMU-0 ***//*** FAILED PROGRAM BLOCK : IHWMAN, 6D5H ***/

COMMAND EXECUTION ABORTED

Indentify that ShelfManager boot ends like this:

<*> 14:47:53.630 [217] Pigeon Point Shelf Manager ver. 2.6.4.2. Built on Oct 29 2009 14:16:53<*> 14:47:53.695 [217] *** Lock log print buffer at 1006c9d0 ***<*> 14:47:53.697 [217] *** Pthread lock log print buffer at 10071220 ***<I> 14:47:53.708 [217] Reading configuration file: /etc/shelfman.conf

shmm500 login: Failed to initialize the carrier!

Perform following steps to recover:

STEP_1



Interrupt the boot-up process by pressing the space-bar on console of the ShMM:

STEP_2

From u-boot, issue the following commands:

setenv rc2 /etc/rc.nsn

saveenv

Boot

STEP_3

You should install the correct FRU ID file for shelf Manager and reboot again.

Take a new file from 06169 zip e.g. 21596309HBBIN.BIN in case of ACB-V and 21596180HBBIN.BIN in case of ACB-IV.

This is done by Active shelf Manager via clia, issue the following commands:

If left ShelfManager:

chmod +x nsn_merge_fru_infofrudatar 10 0 old_fru_data10.bin ./nsn_merge_fru_info 21596309HBBIN.bin old_fru_data10.bin result_fru_data10.bin

clia frudataw 10 0 result_fru_data10.bin

If right ShelfManager

chmod +x nsn_merge_fru_infofrudatar 12 0 old_fru_data12.bin./nsn_merge_fru_info 21596309HBBIN.bin old_fru_data12.bin result_fru_data12.bin

clia frudataw 12 0 result_fru_data12.bin

NSN merge tool

06169 zip

Do not update with MML. It will corrupt FRU data.

STEP_4

Then you need to set the rc2 variable to the HPDL value.

By Shelf Manager console issue the following command:

# setenv rc2 /etc/rc.nsn-hpdl

and after reboot again.

5.4 IP of Shelf Manager is lost at switchover

After an upgrade of Backup shelf manager (4.2. step 5) , it is observed that on backup Shelf Manager the IP_Address is missing , and then when the switchover is performed , the new backup shelf manager haven't the IP_address while the new active has the IP_address. The IP_address have to inserted manually on new backup shelf manager and then the upgrade itself can be performed.

https://sharenet-ims.inside.nokiasiemensnetworks.com/Overview/D426335564

https://sharenet-ims.inside.nokiasiemensnetworks.com/Overview/D425093563



For fixing check Trobleshooting 5.2. point 3.

shelfmanager esw update guide to 06166 and with mml to 06169.doc

Documents