nextel emc manual

Nextel EBTS Monitoring Center Training Manual: Level One

EBTS Monitoring CenterTraining Manual

Skill Level One

Provided For

EMC Training Manual.doc04/11/23

1


Document Name: EBTS Monitoring Center Training Manual, Level One

Document Revision History: Created on December 13, 2000

Document Owner: Nextel Communications EBTS Monitoring Center, Oakland, California

This reference guide is intended for Nextel internal use only. The information in this document is subject to change without notice. The Nextel EBTS Monitoring Center Supervisor, Nathan Clement, with Technical Writer Julia Gilden has authored this document. Any questions or comments regarding this material should be directed to Nathan Clement, Nextel EBTS Monitoring Center, Oakland, CA, at (510) 208-8867, or by sending email to [email protected].


2


List of Effective ChangesDate Name Page # Change

2/1/01 Todd Lefemine 7,56 Changed the SMS to read 240 and 2way to 500 character

2/14/01 Todd Lefemine 455142

Added AM callout to PB account repAdded Training ChecklistAdded PNW Multi Power outage Visio Graph

3/09/01 Ramon Colon 51 3

Modified Training Checklist, Added List of Effective Changes.

3/13/01 Ramon Colon 44 Updated the NorCal Multiple Power Outage Procedure.

3/15/01 Todd Lefemine 22 Updated the notification of OMC/OSS group process for Measles

4/30/01 Gail Allen 9-16 Added EBTS Power information

7/18/01 Todd Lefemine 39 Added PNW T-1 Centest IP's and Login's

7/25/01 Ramon Colon 62 Added NTM Guidelines

8/01/01 Ramon Colon Added the Rocky Mountain Weekend T-1 handling Procedure.

Added the Rocky Mountain Multiple Power Outage Procedure.

8/03/01 Gail Allen Multiple Format changes

8/03/01 Ramon Colon 61 Added the Nextel Partners List to the NTM Procedure.

8/07/01 Ramon Colon 59 Added the NTSG Procedures.


3


TABLE OF CONTENTS

1. TABLE OF CONTENTS..................................................................................................................... 4

2. OVERVIEW......................................................................................................................................... 6

3. NETWORK ELEMENTS.....................................................................................................................7

4. POWER SYSTEMS............................................................................................................................ 9

5. FREQUENTLY USED ACRONYMS.................................................................................................10

6. EMC TECHNICAL RESOURCES.....................................................................................................18

7. ACCESSING AND USING FAULT MONITORING TOOLS.............................................................20

LOGGING ON AND OFF THE XTERM............................................................................................................20LOGGING ON AND OFF THE OMC.............................................................................................................21LAUNCHING THE MMI FROM LAPTOP.........................................................................................................22SETTING UP THE OMC FOR FAULT MONITORING.......................................................................................23USING THE OMC TO MONITOR FOR FAULTS..............................................................................................25

Using the OMC to Get (Query) BR Power..........................................................................................25Using the OMC to Reset BR...............................................................................................................26

LAUNCHING, LOGGING ON AND OFF NET EXPERT.......................................................................................20SETTING UP NET EXPERT FOR MONITORING.............................................................................................28LAUNCHING, LOGGING ON AND OFF MEASLES...........................................................................................29SETTING UP MEASLES FOR MONITORING..................................................................................................30

8. ROUND-THE-CLOCK PROCEDURES............................................................................................31

EMC SHIFT CHANGE RESPONSIBILITIES....................................................................................................31EBTS MONITORING CENTER (EMC) DAILY REPORT..................................................................................32NTM OPS TICKET PREPARATION.............................................................................................................64SITE VISIT LOG........................................................................................................................................ 66CONTINUOUS SITE MONITORING...............................................................................................................33TECHNICIAN ID AND SITE INFO DATABASE RESOURCES.............................................................................30

9. TESTING T-1 LINES......................................................................................................................... 34

ACRONYMS USED TO TEST T-1 LINES.......................................................................................................34T-1 TESTING RESOURCES........................................................................................................................ 35USING CENTEST TO TEST T-1 LINES.........................................................................................................36ISOLATING AND TROUBLESHOOTING A T-1 OUTAGE FOR AN EBTS.............................................................38EBTS/T-1 FAULT REPORTING DIAGRAM...................................................................................................40EBTS/T-1 FAULT DIAGNOSIS DIAGRAM.....................................................................................................41

10. PROCESSING ALARMS..................................................................................................................42

TYPES OF ALARMS................................................................................................................................... 40Environmental Alarms........................................................................................................................ 40High Temperature Alarm: NorCal.......................................................................................................40High Temperature Alarm: PNW..........................................................................................................44Power Outage Alarm – Single Site NorCal.........................................................................................45Power Outage Alarm – Multiple Sites NorCal.....................................................................................47Base Radio (BR) Alarms.................................................................................................................... 50

11. NOTIFICATION AND ESCALATION PROCEDURES.....................................................................52

NEXTEL FIELD TECH NOTIFICATION PROCEDURE.......................................................................................52Business Hours Notification................................................................................................................52


4


After Business Hours Notification.......................................................................................................52TELCO TROUBLE TICKET CONTENT AND NOTIFICATION PROCEDURES.........................................................53CONTACT INFORMATION........................................................................................................................... 54OSS TROUBLE TICKET PROCEDURE.….................................................................................52BUSINESS HOUR ALARM NOTIFICATION.....................................................................................................57AFTER BUSINESS HOURS ALARM NOTIFICATION........................................................................................53

NTSG Notification Procedures………...…………………………………………………59

12. NTM GUIDELINES ……………………………………………………………………………………………62

13. REVIEW TEST AND SIGNOFF........................................................................................................67

APPENDIX................................................................................................................................................ 69

A. IDEN NETWORK ELEMENTS SLIDE SHOW........................................................................................70B. NCLR SCREEN LAYOUT TUTORIAL...................................................................................................80C. MEASLES SCREEN SAMPLE AND OVERVIEW SUMMARY.....................................................................82

Measles II Executive Summary..........................................................................................................82Measles II User Guide........................................................................................................................ 83

D. NET EXPERT SCREEN SAMPLE AND OVERVIEW SUMMARY................................................................89E. VENDOR CODE SUMMARY FOR ENVIRONMENTAL ALARMS.................................................................90F. NOR-CAL MORNING REPORT.........................................................................................................91G. PNW DAILY REPORT...................................................................................................................... 92H. DENVER MORNING REPORT............................................................................................................83


5


OVERVIEWThe purpose of this training manual is threefold. It identifies existing tools and procedures used by the EMC (EBTS Monitoring Center); it also provides a training aid for new hires and a reference guide for new and existing employees. The manual is a living document and exists in both hard and soft format. Each topic covered in the manual includes a definition and purpose, appropriate tools, instructions, and review of key points.

The EMC was launched in February 1999. The EMC was created for three reasons:

1) To centralize knowledge by providing and developing SME (Subject Matter Experts) as Area resources2) To improve the customer’s Nextel experience by closely monitoring the EBTS (Enhanced Base Transceiver

System) Networks and taking immediate action to resolve all issues3) To provide root cause analysis. The EMC is a sister organization to NNOC (National Network Operations

Center) and reports to the Director of the NNOC.


6


NETWORK ELEMENTSBSC-CP Base Site Controller – Call Processor

Connects to multiple EBTS sitesPerforms handovers for subscribers involved in interconnect calls between EBTS sites under its controlRoutes call set up information from the MSC to the SCDR, as well as from the EBTS to the XCDR

BSC-XCDR Base Site Controller/TranscorderProvides the Transcoding Function between Motorola VSELP compressed audio and 64kbps PCM used in the PSTN.

EBTS Enhanced Base Transceiver SystemProvides the radio communication link between the network and the Mobile SubscriberConsists of BR-Base Radio; ACG-Access Control Gateway; and AntennaCan be configured Omni-directional or sectored

DACCS Digital Access Cross Connect SwitchSplits the T-1’s to sitesRoutes proper channel to BSC for interconnect callsRoutes proper channels to IGX/DAP for dispatch calls

IGX Integrated Gigabit ExchangeResponsible for switching and routing of dispatch voice packets (Frame Relay)

APD Advanced Packet DuplicatorVoice packets are duplicated for simultaneous broadcast over a number of cells with members of the same talkgroup. Used in both Private and Group Dispatch group calls.

DAP Dispatch Application ProcessorProcessing entity responsible for the overall coordination and control of Dispatch Networks.Handles all call authentication and registrations.Keeps billing records for dispatch calls.

MDG Mobile Data GatewayCentral packet data switch pointObtains operational information from the DAP

NAS Network Access ServerConverts channelized T-1’s to Unchannelized fractional T-1’s

X.25 PS X.25 Packet SwitchConnects the OMC to the network entities

OMC Operations and Maintenance CenterPerforms Configuration, Fault, and Performance ManagementMonitors the system performance of the networkProvides event and alarm handling functions

MSC Mobile Switching CenterProvides the interface between the PSTN and the mobile networkControls the interconnect call set up and routing proceduresHelps to administer interconnect handover proceduresCollects billing information for interconnect callsCollects traffic statistics on the switch itself


7


HLR Home Location RegisterIs the master database for all subscriber interconnect service informationAllows subscribers to roam between MSCContains the location of the current VLR for each subscriberSupports SS7 system connectivity to network STP’s (Signal Transfer Point Switches)Contains the Authentication Keys for all MS’s

IWF Interworking FunctionMain purpose is to convert standard modem protocols to a format supported by the iDEN networkCircuit switch data for the hearing impaired

SMS-SC Short Message Service – Service CenterProvides for the delivery of short messages up to 280 characters to a MS (500 Two-Way Messaging).Handles voice mail indications

VLR Visited location RegisterDatabase that contains most recent subscriber information and location of subscribersContains most of the same information that the HLR contains for the active MS unitsAdds/Deletes MS’s as units roam into/out of the coverage area

PSTN Public Switched Telephone NetworkProvides T-1 access to land lines (Local Exchange Carrier and Long Distance Carrier)Connects MSO to EBTS

MS Mobile Subscriber Multi-service, integrated dispatch interconnect, roaming, Message Mail and data communications

VMS Voice Mail ServerStores voice messages for later retrieval or forwarding.


8


POWER SYSTEMS

i. Purpose

This documentation should serve:

- As a tool to help interpret and define alarms seen from EBTS sites in the field.- A guide to troubleshooting and identifying possible false alarms.- A guide to assist EMC technicians with understanding what Field Technicians

are experiencing as an aide to simplify the troubleshooting process between Field Technicians and EMC technicians.

When notifying technicians and facilities groups, keep in mind that procedures and responsibilities for both EMC and field technicians differ from market to market. Be sure to refer to the EMC training manual for the correct procedures for the market you are currently watching if you are uncertain.

1. Cell Sites

a. Types of Shelters

Nextel typically employs use of full-size shelters for the majority of our sites. The shelters vary in size from 10’ x 12’ up to 15’ x 20’ in width and length. These shelters are pre-fabricated by several companies with Communication Buildings of America (CBA) is the most common manufacturer of our shelters. They are made primarily of an outer shell of fiberglass with a wire-mesh shielding running through the walls of the shelter. This is part of the grounding network. While grounding is of importance for dissipation of transient power discharges (otherwise known as lightning strikes), it is critical for keeping the noise floor low at a site. A little noise is tolerable in analog systems, but can play havoc with TDMA packets in digital systems.

Since stand-alone shelters have excellent grounding properties, one can close the door on a shelter once inside and have absolutely no coverage for their portable communications if the site is disabled. Technicians often need to leave the door open on the site and keep their phone and pager near the door while performing ATP’s or any other site maintenance that involves disabling the cell.

As municipalities become more restrictive with zoning and landscaping regulations, Nextel (which is classified as an ESMR, not a Cellular provider) and other Cellular providers are having to forego using full-sized shelters and install telco-sized shelters. Other providers use radio base station equipment manufactured by Ericsson and Lucent. The gear is smaller, lighter and can yield call handling capacity comparable to those often found in any of our larger shelters. Ericsson and Lucent base radios fit on a single or dual card slot backplane, and often weigh less than 8 pounds. Equipment such as this, fits easily into the


9


telco-shelters.

A single EBTS Base Radio assembly, fully populated (meaning all cards, receiver, exciter etc.) weighs approximately 80 pounds. EBTS radios create quite a bit of heat. Additional heat is generated from hybrid combining (2:1, 3:1, 4:1, 6:1 etc). An HVAC failure on a full-sized shelter can cause the site to heat up to temperatures nearing or over 160F in about an hour all depending on the number of radios at the site. In a telco type shelter this can happen in a matter of 15 minutes or less due to the lack of physical space in the shelter for heat dissipation. Due to these scenarios, response time for high-temperature alarms is critical. EBTS equipment will become damaged at higher temperatures, either causing it to fail immediately or causing several failures spread out over the next month or two in the same site. ISC’s are designed to shut themselves down at 125F to prevent damage while running at such high temperatures.

Picture of ISC’s in site. Top is primary, bottom is secondary.

b. Basic Shelter Layouts

Every attempt to place EBTS in a logical layout is made for each site. A few sample layouts are shown here.

<insert equipment layout drawings>

A typical three sectored site will have RF rack #1 located adjacent to the control rack (control racks are never labeled as rack #1). With very few exceptions, the first sector on the site is always RF rack #1, with the primary rack for sector #2 being labeled as RF rack #3 and the third being RF rack #5. Expansion racks for each of the three sectors in a site are almost always even numbered racks. Controller racks can be located either near the


10


entrance end of the shelter with RF racks following toward the back of the shelter. Conversely, the Controller can be Located in the far end of the shelter with RF racks beginning next to it and increasing in numbering as they near the shelter door. The standard that is followed in all sites will usually involve proper spacing and Location of the racks allowing access to the rear of all the racks with ample room for the site technician to perform any work. All cabling for the equipment in the racks is located in the rear. The only access on an RF rack that can be found on the front are visual indicator lights for the BR’s as well as 9-pin D-shell serial connectors on the front of each radio for command-level access.

Front of BR showing 9-pin access port on controller card.

Telco shelters have a severe lack of access space to the rear of the radios. All radios and controller racks are mounted with the equipment front panels facing outward and mounted along the lengthy sides of the shelter. Doors on the telco shelters close with only a few inches clearance between the inside face of the doors and the equipment faces. Because of this, telco shelters are difficult to maintain equipment in during inclement weather since water can easily enter the shelter and land directly on the BR’s or controllers. While tarps and other shelter equipment is often located inside Telco shelters, it is difficult to put together and ineffective in all but the mildest of rain. It’s often better for Field Technicians to wait until the weather clears before accessing the shelter rather than compromise the shelter and possibly expose the remainder of working equipment to the elements potentially causing further damage.


11


2. Power Distribution System Layout

Unless otherwise specified, all Nextel sites are supplied with AC power between 208-220VAC. This is either 1 or 3-phase power, depending on the availability of supply types in the area and power load requirements in the site. The EBTS equipment in all Nextel sites is configured to run on –48VDC (this is standard Telco voltage, which is advantageous when interfacing with Telco equipment).

Three Phase transfer switch (Generator to AC Power)

The configuration for all the power distribution systems remains the same for all sites.

1. AC power enters the shelter. If the site Location is capable of handling generator power, then the power is routed through a transfer switch. The transfer switch is the point where AC power is chosen from either the Local power utility provider (such as PG&E), or from a generator source.

2. AC power is routed from the transfer switch to the circuit breaker panel. This panel provides as the distribution point for wall outlets, lighting, HVAC and any other miscellaneous items in the shelter that are not powered from the rectifier and battery plant.

3. AC power is then routed to the rectifier plant. While we use several vendors for Rectifier plants (PCP and Reltec are the most common). In the end, however, all the plants perform in basically the same manner with little change in function. Rectifiers convert AC power into high-current DC voltage (-48VDC). Several rectifiers run in parallel, simultaneously providing DC voltage for the EBTS equipment and charging one or more battery strings in the site.

4. Additional equipment is attached to the rectifier plant. However, this normally EMC Training Manual.doc

04/11/2312


consists of the Smart CSU at the most. Rarely is there any additional equipment in a site pulling power from the rectifier plant.

These battery strings provide backup power for the EBTS equipment only. Batteries float constantly on the same circuit as the EBTS gear, which eliminates any need for switching to take place, should AC power at the site fail. Depending on the number of batteries in a given site versus the number of BRs in service, the amount of battery backup time can vary greatly. In most cases, a given site will stay on the air for no less than 1 hour after losing AC power. A few sites will actually last for up to 6 hours or more. While there is no true average time and site loading versus the number of batteries seldom is calculated using any exact science, a few simple rules should be observed when dealing with power loss issues:

1. Always dispatch appropriate resources for power failures whenever found. Verification of the outage through the Local power utility provider (Local to the area / market you are working) and callout of the tech for that site need to be acted upon as soon as possible.

2. Regardless of BR to battery string ratios at any given site, most sites will stay on air for at least 3 hours once AC power has been lost. However, acting upon a power outage immediately remains a critical issue. HVAC equipment power is not supplied by the rectifier / battery plant. The site may be running, but temperatures can quickly reach up to or in excess of 160F in about an hour if the site houses a large number of BRs or is in a confined space such as a Rohn or telco type enclosure.

4. Alarm Types (NCAL Specific)

In most cases, alarms from power plants have the same meaning from plant to plant regardless of plant design or manufacturer. However, depending on the overall power plant configuration of a plant the severity of an alarm can be quite different. Here are some examples:

1. PDB Rectifier Minor [246]: One rectifier has failed. Triggers for this alarm include either a loss of DC power output from the rectifier into the distribution buss or the breaker / power switch on the front of the rectifier has been moved to the off position. Again, this signifies such a condition for only a single rectifier in a given site.

2. PDB Rectifier Major [247]: More than one rectifier has failed. Triggers include any number of failed rectifiers numbering two or greater in the plant. Being fairly robust devices, the odds of more than one rectifier failing at a time are quite slim.

3. PDB AC Power Failure [242]: The Power Distribution Buss has detected a lack of AC power to the rectifiers. Provided the alarms on the PDB are configured correctly, this will be seen in conjunction with PDB Rectifier Major.

4. Surge Arrestor [223]: This indicates a line-voltage surge or low voltage condition across one or more of the surge arrestor installed in the AC power supply side. Most surge arrestor are designed to allow AC power to pass unregulated after a


13


failure. Even though this allows the site to stay on the air, this is still a condition that requires attention by facilities as soon as possible. A failed surge arrestor offers no protection against transient line voltage for equipment in the site. A surge could cause serious damage to rectifiers and other power distribution equipment.

Additional alarms concerning power loss are available depending upon site configuration and accessories. In no particular order:

1. Utility Power Loss [206]: This is a detector placed on the incoming utility company provided AC power feed to the shelter. The sole function of this alarm is for the indication of AC power loss from the provider. On sites equipped with permanent generators, it is not uncommon for a Utility Power Loss alarm to occur while having the other PDB related alarms bounce from Major / Critical states and back to Cleared states within a few seconds after receiving the Utility Power Loss alarm. Notably, the Utility Power Loss alarm will not clear after all PDB alarms clear if the site has a fixed generator Located onsite. A breakdown of what happens at fixed generator sites upon a power loss event will be explained in this section.

2. Generator Running [209]: Indication that a fixed generator at a site has started. 3. Generator Minor (fuel) [207]: Indication that a fixed generator is low on fuel supply.

This requires immediate attention to the site tech / facilities. Pay special attention to this alarm as it usually appears then clears immediately after a generator starts.

4. Generator Mechanical [208]: Indication that a fixed generator is experiencing a mechanical failure or is suffering from a mechanical problem of some sort. Unfortunately, this alarm is vague at best but requires a response from facilities and or the site technician as soon as possible.

5. AC power loss at a fixed Generator site

When AC power loss is detected at a fixed generator site, the following occurs.

1. Automatic AC power transfer switch detects the power loss. This changes a logic state in the generator controller array causing the generator ignition to trigger.

2. Utility AC Power Loss [206] (if so equipped), Rectifier Major [247] and AC power loss [242] alarms are sent to the OMC.

3. The generator starts, triggering a Generator running alarm [209].4. Once the generator is up to speed and outputting an acceptable voltage level

to supply the site with, the automatic transfer switch in the site changes the AC supply from the utility power side to the generator supply side.

5. PDB Rectifier Major and PDB AC Power Loss alarms clear. If installed, Utility AC power loss [206] will remain as an indicator that Utility power is still not available.

These alarms do not always enter the OMC at the same time causing them to appear in random order in many cases.


14


6. Rectifier Major and Minor Alarms

Twin Pack Plus / II rectifier plant. This shows two racks of rectifiers, four each. Batteries to right.

The number of rectifiers in a given site varies greatly. On the average, a site will contain no fewer than four rectifiers in its’ plant. However, there are some considerations to think about when Rectifier Major alarms take place.

- How many rectifiers does the site contain? If the site contains only four rectifiers when a Major alarm is received the site now has two (or fewer!) rectifiers to handle both the current load from the EBTS gear and charging of the batteries. This is now a critical situation that requires immediate attention of facilities and / or the site technician for that site. In many cases, this is enough to push the loading of the rectifiers to over 100% of their rated duty. If this is the case, it is only a matter of time before low voltage alarms are received and the site drops off the air.

- There are some sites that contain fewer than four rectifiers. Consider a Rectifier Major alarm on a site created in this situation. One rectifier is now all that remains to run the entire site and charge the battery string(s).

- I personally have been in sites that contain only two rectifiers. Rectifier Major in a site such as this = no power to charge the batteries = the site will soon be down. Watch for low-voltage alarms.


15


- Even though some of the information is out-of-date, it would probably be a good idea to check the Site-Trak Database in the power services data entry area. You can at least get an idea of what kind of power load is in the site and have a rough idea of what kind of response is warranted once you receive alarms from the site.

Rectifiers are hot - swappable, and only require the float and equalization voltages to be set upon installation. However, the stock of rectifiers site technicians currently have on hand is often limited or depleted. Normally the facilities group needs to address this issue.


16


FREQUENTLY USED ACRONYMS ACG Access Control Gateway

APD Advanced Packet Duplicator

BR Base Radio

DAP Dispatch Application Processor

EBTS Enhanced Base Transceiver System

FMT Fault Management Tool

HLR Home Location Register

IGX Integrated Gigabit Exchange (Part of the MPS)

ISC Integrated Site Controller (Part of EBTS)

IWF Inter-Working Function

MOBIS Motorola adaptation on the GSM-A.bis protocol

MSC Mobile Switching Center

MSO Mobile Switching Office

NAS Network Access Server

NIU Network Interface Unit

OMC Operations and Maintenance Center

PCM Pulse Code Modulation

PSTN Public Switched Telephone Network

RSL Radio Site Link

SMS Short Message Service

SNMP Simple Network Management Protocol

TDMA Time Division Multiple Access

VLR Visitor Location Register


17


EMC TECHNICAL RESOURCESOperation Maintenance Console (OMC)Fault Management ToolMotorola configuration management tool for iDEN network.Launched from Xterm pull down menu or Windows Reflections application.To repair, call OMC Group or MSO.

MeaslesFault Management ToolWeb based tool launched from browser.Launch from Xterm pull down menu or Windows Netscape CommunicatorMeasles is the software application that displays alarmsServers reside in each Local market; national server resides in McLean, VA.To repair Measles, call OMC Group.Use Measles primarily for Pacific Northwest

Net Expert (NetX)Fault Management ToolA software application that displays alarms. A stand-alone product, Unix based. Launch from Xterm pull down menu bar.Regional server resides in Los Angeles; national server resides in McLean, VA.Use NetX primarily for NorCal and as a secondary resource for PNW.To repair, call OMC Group.

NTM(Clarify)NTM/Clarify application/softwareNetwork Trouble Management SystemUsed to fill out Ops ticket, and Site Visit Log.Site Visit Log tracks people walking into cell sites.Ops Ticket is used to identify and track reported alarms for EBTS and other network elements.Launch from Notebook Computers.To repair, call Help Desk @ *123, IT group.

ReflectionsSoftware application used to Telnet, FTP, provide terminal emulation and to simulate a Unix environment for running Unix –based applications in Windows environment.Launch from Windows.To repair, call *123, IT group.

Centest Application and piece of equipment. Equipment resides at each MSO.Used for remote T1 testingLaunched by Telnet from Windows using Reflections application, or from Xterm Remote Access pull down menu.

Nextel Circuit Layout Record, also called Danny’s Database\\Wtcntas01\wwwroot\Database\NDB_cur.mdbIf this Network drive is not mapped to your notebook let your supervisor know. Access database used to obtain T-1 information.

NorCal Technician Access Database, aka Tom Smith web sitehttp://www-oak.nextel.com/home/ephelps/ index.html Web-based databaseProvides site and on-call technician assignment, and site Location in Northern California


18

http://www.oak-nextel.com/home.ephelps/index.html


PNW Technician Access Database http://home.cybergs.com:802/fieldman/main.htmlWeb-based databaseProvides site and on-call technician assignment, and site Location in PNW

Site Track Databaseg:\eng\facildb\siteform.mdb/wrkgrpg:\eng\facildb\system.mdb/user OpsAccess databaseProvides Power Outage Information and site Location


19

http://home.cybergs.com:802/fieldman/main.html


ACCESSING AND USING FAULT MONITORING TOOLSLogging on and off the XtermXterm is the remote terminal for the Local Unix server. Assume the Xterm is powered on, and you are sitting at a monitoring workstation.To log on to the Xterm:

Step Screen Process Procedure1. From the upper left hand corner menu of the

XDMCP screen,click on either Xserv 1, or Xserv 2.

2. A prompt welcomes you to the session. Enter user name and click OKScroll through Options, Session, Common Desktop Environment.

3. Be sure Common Desktop Environment isselected; if it isn’t

Select it now.

4. As Solaris launches, watch in upper left handcorner for Common Desktop Environmentconfirmation.

Enter password and click OK.

This completes the Xterm log in process. To launch the Nextel OC menu, left click and hold mouse on the monitor background.

To log off:

Verify all applications have been properly closed on all desktops. Choose Exit and click OK at log out confirmation.


20


Logging On and Off the OMCOMC is the secondary tool for monitoring NorCal and PNW sites.The OMC software interfaces with the network elements. OMC equipment manages site configuration, fault, and performance; monitors the network’s system performance, and provides functions to handle events and alarms. To launch the OMC software from your desktop:

Step Monitor Screen Purpose Procedure1. Xterm Monitor Screen To launch OMC application Click and hold down cursor on

monitor background.2. Nextel OC Menu Window

opensTo launch Man-Machine Interface (MMI) and select market to monitor

Scroll down to OMC access, then scroll right and select market by releasing mouse.

3. Up to 10 workstations can log on to a market OMC at the same time. If one is available, you will automatically be logged on.

Observe screen for announcement of log-in availability. When you are logged on, click OK.

4. OMC Console Screen opens.

Click OK to close the “About the OMC” window.

To prepare the OMC application for monitoring, several tools must be opened. Across the top of the Console window, you’ll see several menu choices.

The most common tools used by EMC technicians are:1) Fault Management, used to open the Site Listing viewer2) Event Management, used to open site monitoring tool windows3) Event Log, which provides a chronological history of events by site.

Note: Several tool windows can be opened and cascaded across the monitor screen for easy access.To log off:

1. From the OMC Viewer screen, select File, then Exit.2. From the OMC Console, select File, then Exit.


21


Launching the MMI From LaptopUse the following market-specific IP addresses to log on:

Sacramento sacmmi 192.168.104.201 Oakland oakmmi2 192.168.130.204 Santa Clara scmmi 192.168.103.207

Step Monitor Screen Purpose Procedure1. Laptop Monitor Screen To set wizard for MMI application Using mouse, select in order:

- Start- Programs- Reflection- Wizards- X Client Wizard

2. Wizard Launch Window To launch Man-Machine Interface (MMI) and select market to monitor

- Type IP address for market OMC- Select “Sun host” in “type” field- Type username “OC_LOGIN” - Type password “nextel1” (DO NOT SHARE THIS INFO)- Click “Show login process”

2. OMC X Client WindowXxxmmi:OC_LOGIN> prompt

To complete log-in set-up - Change name to “xxxmmi” at 192.168.xxx.xxx- Click on Finish.

3. X Client window closes. Reflection folder shows shortcut to MMI.

To log in - Double click on the short cut (should automatically log you in; remember the 10 workstation log-in limit).

- At xxxmmi:OC_LOGIN> prompt, type “mmi&”

4. OMC Console Screen opens.


22


Setting Up the OMC for Fault Monitoring Step OMC Screen Purpose Procedure

1. OMC Console Screen To open Fault Monitoring tool window

Select from menu across the top the tools you want to use by clicking on each tool. Each one must be selected individually.

2. OMC Console Screen To open Event Monitoring windows

Click on Fault Management on the OMC Console menu.

3. OMC Console Screen Sub-Menu

To launch Solstice EM2.0.1 Viewer

Select System Status. Double click on Select Sites, which opens a complete listings of sites in the market.

4. OMC Market Listing Screen

To arrange sites in an ordered sequence

Move cursor to View, select Layout, which opens the Viewer Layout window.

5. OMC Viewer Layout Window

To customize labels De-select “Save position changes to MIS”Select the number of characters per label according to the market size and your preference.

6. OMC Viewer Layout Window

To customize labels Select the number of columns according to market size and preference.

Note: Every label represents an EBTS site antenna, and all EBTS site numbers are displayed in all tools.Label colors have the following values for the OMC tool:

Green Fully operational Yellow One or more elements is impaired Red Out of service Blue Indeterminate Black Label is selected by user


23


Step OMC Screen Purpose Procedure7. OMC Console Screen To open Site Monitoring tool

windowsClick on XXX and scroll to Open Event Display, and select.

8. OMC Event Display ScreenEvents are listed by alarm type and by site. Choices include:- *ALL-MINUS-STATS- All Events- *BR Events- Download Events- *Environmental Events- PLMN Events- *X.25 FAILURES

To choose the windows for the events you want to monitor

Select one at a time and press OK. Re-open Event Display window after every choice. Several windows can be open, and can be minimized on desktop.

9. OMC Event Window Screens

Windows can be customized to present pertinent information only. To format windows

Hold down cursor on Format on top pull down menu and scroll to Wide. Select. De-select parameters you don't wish to view (ID, state, operator). Each item must be chosen individually.

* These alarm events should always be opened.The Event Management windows are now ready to display any site-selected events detected by the OMC

Note: Minimized windows will appear on your desktop as alarm clocks, which will alert the technician when there is an event occurrence in that window. Caution: Closing event windows causes current history to be erased. When a window is closed, the information in it must be retrieved from the Event Log.


24


Using the OMC to Monitor for FaultsUse OMC filtered Event Logs to track X.25, BR, Environmental, etc. activity.Note: In most cases, the OMC is the backup Fault Monitoring Tool. However, for the following types of faults, the OMC is the primary tool.

Using the OMC to Get (Query) BR PowerStep OMC Screen Purpose Procedure

1. OMC Console Screen To get power at a BR site Select Miscellaneous from top pull down menu, scroll down to NE (Network Element) Mgmt and select, which opens the Network Element Window.

2. OMC Network Element Window

To select site Scroll down the list of site numbers and names in the Network Element window, and select.


To continue process Click on Device, hold mouse down and scroll down to Power and Channel, and select.

4. OMC Network Element Window, Power and Channel sub-window

To specify which BR Under “Value to Send,” enter the Base Radio number. Get this number from the NetX alarm or Site Status Display. Click on “Send.”

5. OMC Network Element Window System Message area

To confirm message processing

Note the power level of the selected BR, e.g. “4 power level BR.”


25


Using the OMC to Reset BR Step OMC Screen Purpose Procedure

1. OMC Console Screen To reset power at a BR site Select Miscellaneous from top pull down menu, scroll down to NE (Network Element) Mgmt and select, which opens the Network Element Window.


To select site Scroll down the list of site numbers and names in the Network Element window, and select.


To continue process Select Command from top pull down menu, scroll down to Reset and Select, which opens XXX

4. OMC Network Element Window, XXX sub-window

To specify which BR Under BR “Cabinet/Position,” enter the Base Radio number. Get this number from the NetX alarm or Site Status Display. Reset mode is pre-selected at “0”. Click on “Send.”

5. OMC Network Element Window System Message area

To confirm messageprocessing,

Note message; “Reset request for EBT received.” Observe screen for status changes. If the BR is down, you will see the message, “BR reset request to EBTS failed.” Call field technician to investigate further.


26


Launching, Logging on and off Net ExpertNet Expert, also called NetX, is a fault-monitoring tool built by OSI.Net Expert is the primary monitoring tool for ALL sites.

First, choose which market you will be monitoring. Categories are divided among technicians on a first-come, first-choice basis. Available monitoring logins are posted on the closet door near the exit. Markets are broken into the following categories:

Northern California Log in 1-5 Pacific North West Log in 7-8 Denver

Next, assuming you have logged on to the Xterm:Step Net Expert Screen Purpose Procedure

1. From Xterm monitor background

To launch Net Expert Click and hold down cursor on Xterm background until OC menu appears.

2. Nextel OC Menu Window

To select server Scroll down to Net Expert Access, then right to whichever server you chose when you logged on to Xterm (1 or 2), and select that server now.

3. Net Expert Client StartUp Window

To open Net Expert Enter server name: westnms(Western Region Network Monitoring Server), and confirm by typing Y for yes.

4. Net Expert Client Manager Window

To log in, At the prompt

Enter the oc login and market monitoring number you have selected for this session, e.g. “oclogin3” (where 3 is a NorCal market).Enter the password: westnetXFollowed by the same server number as used above, e.g. “westnetX3”

Net Expert is now launched.

Note: If Net Expert fails to launch, try to re-launch using whichever server you didn’t choose the first time and notify the OMC Group Manager (Richard Ishikawa).

Note: Occasionally, you will get a warning window indicating the log in you have chosen is already in use. In this event, click OK at the warning, then click OK to boot off the other user.


27


Setting Up Net Expert for MonitoringAssuming you have logged on to Net Expert:

Step Net Expert Screen Purpose Procedure1. Net Expert Client Manager

WindowTo launch the Alert Summary window

Select Alert/Display.

2. Alert Summary and Alert Display Windows launched

If Alert Display Window failsto launch

Move cursor to Alert Summary box, hold cursor down on the Alert Window, scroll to New Window and release cursor.

Sort by Severity – Newest Create Time (recommended).Net Expert is now ready to monitor.While Net Expert application is in use, the Client Manager software will move to the background, or you can minimize its window.

To log off:

3. Click on Log Out.4. Choose OK at the warning and close all windows.5. Choose Quit at the Log On window.

Label colors have the following values for the OMC tool: Red Critical Gold Major Light Grn. Minor Blue Informational Lavender Summary (not used)


28


Launching, Logging On and Off MeaslesMeasles is a Web-based monitoring tool, which is launched from Netscape.

To launch Measles from a laptop, use the following IP address:http://167.20.103.209:82 for NorCalhttp://167.20.22.170:82 for PNW

To launch Measles from your desktop, assuming you have logged on to the Xterm:Step Measles Screen Purpose Procedure

1. From Xterm monitor background,

To launch Measles Click and hold down cursor on Xterm background until OC menu appears.

2. Nextel OC Menu Window

To select Measles application Scroll down to NSC Access, then right and choose NorCal Measles or PNW Measles.

3. Measles II Launch Pad Screen

To proceed to application, Choose Measles Link on the left side of the screen. Screen will flash, then will open “Measles Launch Page.”

4. Measles II Launch Page Screen

To confirm success Look for gray “Browser PASSED” box in upper left corner.

5. Measles II Screen Monitoring tool ready.Measles is now launched.

Note: If the application fails to launch or update, notify the OMC/OSS group.1. Left mouse click on the desktop of the Xterm to bring up the Nextel OC Menu or go to

http://oakosi.nextel.com/ops from a web browser and go to step 3.2. Select the Operations Web Page3. Select the Support link (upper right corner of window)4. Determine the on-call person and call out information for the OMC/OSS group.

To log out:6. Go to the File pull down menu, as on any web site.7. Choose exit or close


29

http://oakosi.nextel.com/ops

http://167.20.22.170.82/

http://167.20.103.209.82/


Setting Up Measles for MonitoringAssuming you have logged on to Measles:

Step Measles Screen Purpose Procedure1. Measles II Launch Page

ScreenTo select market to monitor,On the gray bar “National Network,”

Click on the yellow button on the RIGHT side of your screen. This will open all the markets monitored by Measles.

2. Measles II Launch Page Screen

To complete your selection,On the gray bar of the market you wish to select,

Click on the yellow button on the RIGHT side of your screen. Be sure market is the same as your log-in choice.

3. Measles II Launch Page Status Report Window

Subcategories of your chosen market are now open for monitoring. To proceed,

Click on the PLUS SIGNS on the yellow buttons on the LEFT side of the subcategories listed.

Current status of all sites showing alarms are now listed.

Label colors have the following values for Measles: Red Critical, e.g. X.25 failure Yellow Major Lt Green Environmental Notification Dk Green Informational, e.g. site entry Lt Blue Undetermined Blue Environmental Trouble


30


ROUND-THE-CLOCK PROCEDURESEMC Shift Change Responsibilities

Oncoming Technician Procedure Check 8802 Voice Mail Box Check White Board Launch Monitoring tools Identify the market(s) you will be covering Review all tickets Ask previous shift for updates on outstanding tickets Ask for verbal hand-off, important issues Read e-mails

Off-going Technician Procedure Check 8802 Voice Mail Box Verify all White Board information is valid Verify all tickets are properly updated Identify oncoming technician who will replace you Provide a verbal hand-off of the events that happened during your shift Assign tickets to an individual in oncoming shift and verify that tickets are in an ‘new’, ‘work

in progress,’ monitor or


31


EBTS Monitoring Center Daily Report The Local Market Daily Report is sent out every day at 7 a.m. for Northern California (NorCal), 7:30 a.m. for Rocky Mountain Area (RMA), and 8:00 a.m. for Pacific Northwest (PNW). Note: Since preparation for the daily reports can take some time, begin requesting the Mobile Switching Office (MSO) reports from each MSO at least 1 hour prior to sending the report out. Each site issue included in the Local Market Daily Report should have a ticket number and a current status. The Bay Area EMC is responsible for NorCal and PNW issues. If report is more than 10 minutes late, notify Nathan via email with an explanation.Note: See Appendix for samples of NorCal, PWN, and RMA Daily Reports.

Report Preparation Procedure Use the Daily Report Matrix to create a consolidated report from MSO for each Local

market. Include all current BR issues, site issues, environmental issues, and MSO updates. List issues in order by MSO beginning with NCAL #1 (Oakland), NCAL #2 (Santa Clara),

NCAL #3 (Sacramento), NCAL #4 (Sacramento), NCAL #5 (Dublin) and compile into one report.

For PNW: (PNW1, PNW2, PNW3) and compile into one report. RMA (Denver, Rock2, Salt Lake City) When it is time to put the report out, dial into voicemail at ext. 8600. The mailbox number is 8802. When prompted for password, enter 12345. Press 2 to record message and # when your message is complete. Press 1 to Replay your message. Press 9 to send message. You are then prompted to enter the mailbox number or group list. Press 1 for NorCal, 76 for Denver and 88 for PNW (The current distribution list for the

Mountain Pacific Daily Reports). Press # to send message.

Note: Before sending the voicemail for each report, play it back and review for grammatical errors, bodily noises, stutters, accuracy etc. Do this even when you think everything went well. If you make an error in recording your message, hit 7 to erase, and then 2 to begin again.


32


Continuous Site MonitoringNote: EMC Technician group is responsible for monitoring all sites in all market areas at all times.

Monitoring Procedure Read and initial Alert Board. Check E-mail (respond as needed). Check v-mail (respond as needed. Document discrepancies in current tools, open Ops tickets with the appropriate groups, and

notify Supervisor. Identify failures or holes in established procedures and notify Supervisor.

Technician ID and Site Info Database ResourcesNorCal Technician Information (also called Tom Smith web site)http://www-oak.nextel.com/home/ephelps/If database has incorrect information, notify [email protected] and copy Gail and Nathan. If unable to find technician responsible for a site, notify Richard Ruiz or Richard Varela.

If the site is down or does not load, dispatch or call Eric Phelps.

Provides site technician assignment, site on call technician, site address and directions and links to different MSO in the Northern California region.

The site is downloaded every day at 7:35 am; the site is not available during download time, approximately 10 to 15 minutes. If the site is not available any other time, contact Eric Phelps at (916) 997-8913, or dispatch him at 1161 (preferred).

PNW Technician Information http://home.cybergs.com:802/fieldman/main.htmlLog in: EMCPassword: Nextel 1Note: Log in and password are case sensitive.If database has incorrect information, notify Jim O’Brien and Robert Erickson, and copy Gail and Nathan.If unable to find technician responsible for a site, notify Robert Erickson, then Jim O’Brien.If the site is down or does not load, contact Robert Erickson.

Provides site technician assignment, site on call technician, site address and directions and links to different MSO in the Pacific Northwest region..

Site Track database (also called facilities database )eng\facildb\siteform.mdb/wrkgrpg:\eng\facildb\system.mdb/user Ops

If database has incorrect information, notify Ed Hall and Mike.Miller and copy Gail and Nathan. If unable to find technician responsible for a site, notify Ed Hall, then Mike Miller.

This database is mostly used to get Power Outage Information such as Provider, Account Numbers, and Meter Number. It also provides the site address.

Note: See Chapter 11, Notification and Escalation Procedures; Contact Information for phone numbers.


33


mailto:[email protected]

http://www-oak.nextel.com/home/ephelps/


TESTING T-1 LINESAcronyms Used to Test T-1 Lines PCM Pulse Code Modulation PCM is the most common industry standard for digital Voice Encoding. PCM

samples the analog signal 8,000 times a second for each of the 8 bits; this generates the 64kbps DS0.

Intrusive Intrusive testing interrupts the normal path of the circuit and is customer impacting.

Non-intrusive Non-intrusive testing does not interrupt the normal path of the circuit; it is not customer impacting. It is passive monitoring.

AMI Alternate Mark Inversion AMI is the line coding normally used in T-1 voice applications. This signaling format is represents the binary value of one by a square wave (pulse), alternating between positive and negative value for each consecutive mark; the binary value of zero is represented by a straight line (no pulse).

CRC Cyclic Redundancy Check CRC is a mathematical formula that computes a six bit code based off the number of marks and spaces in a bEMCk of data. The receiving end utilizes the same formula to calculate the 6-bit code and compares the results. This can detect errors with a 98.4% accuracy.

TDM Time Division Multiplexing TDM is the process of combing/decombining multiple, separate data sources into one data stream. TDM divides the T-1 link into 24 discrete 64kbs timeslots also known as DS0s.

ESF Extended Super Frame

B8ZS Binary Eight Zero Suppression B8ZS is the line coding normally used in T-1 data applications. The most significant between B8ZS and AMI is B8ZS deliberately inserts a pair of Bipolar Violations when it receives 8 consecutive zeros, instead of having a straight line (no pulse).

SF-D4 D4 Super Frame

BPV Bipolar Violation BPV occurs when two or more consecutive ones (marks) occur with the same polarity, both positive or both negative.

NIU Network Interface Unit

CSU Customer Service Unit. This unit is the demarcation point between Telco and the customer (Nextel)


34


T-1 Testing Resources NCLR Circuit Layout(also called Danny’s Database) E:\Intecon\Danny|Network Design\nb_new.mdeSteps to get to NCLR:1. Sign in, using shortcut2. Answer “OK” to the “Sign out after you are done” request3. Select Network Systems Reports4. Select Network Assignment Report5. Then select NCLR Assignment Report (Site/Trunk)6. Enter the site number under Site/Trunk ID7. Print the NCLR

Network Circuit Layout Record

T-1 Information retrieval tool.Gives information needed to monitorMaps the routing of T-1 lines.

If you have any access problems or site management issues, contact Danny Lee at (925) 279-2300. For issues with the data itself, contact Margaret Grantham at Ext. 2428.

Reflections Telnet Software Application

Centest Set-UpUse Reflections to create shortcut.Open Reflections, enter MSO, IP Address, Log In and PasswordOakland 167.20.103.135. Log in: oakoc. PW: oc2000Sacramento 171.17.26.141. Log in: oakoc. PW: oc2000Santa Clara 167.20.24.116. Log in: oakoc. PW: oc2000Dublin 172.17.16.40. Log in: oakoc PW: oc2000 Launch Wizard by choosing Start, Programs, Reflection, Wizards, Connection Wizard Make the following Selections Terminal Session Over a network Type one IP Address for desired Centest in host fieldVT HostContinue with the following choicesUsernamePassword (DO NOT PASS OUT OR SHARE THIS INFO)Check the “show login” processFinishName the file and make sure the shortcut appears on your desktop.

Testing equipment used to monitor T-1 lines.

Test set with T-1 capacity.

PNW Centest Information *Use the above to make reflections sessions or use Xserv telnet command.centest1 K10 10.204.66.104 port 23 login: ncal password: edmund1 centest2 P21 172.17.30.120 port 23 login: ncal password: edmund2 centest3 P31 172.17.30.119 port 23 login: ncal password: edmund3

centest4 SW0 172.17.30.52 port 23 login: ncal password: edmund4 Portland centest OR0 172.17.30.57 post 23 login: ncal password: edmund5 Eugene centest EUO 172.17.30.62 port 23 login: ncal password: edmund6

Note: For further reference, see NCLR Slide Shows in Appendix.


35


Using Centest to Test T-1 LinesAssume you are logged on to Centest:

Step Centest Screen Purpose Procedure1. Sun Copyright Disclaimer

ScreenTo access Centest, Type any character to go to the DS1

Main Menu screen2. Main DS1 Menu Screen To initiate a T-1 line test,

At MAIN>Type ds1 for the type of circuit to be tested.Press ENTER.

3. DS1 Test Menu Screen To begin a monitoring session:At DS1>

Type monitor.Press space.

4. DS1 Test Monitor Screen ToAt DS1>monitor

Type in the desired systeme.g. sc1_5500Press space.

5. DS1 Digroup Number ScreenToAt DS1>monitor scl 5500

Enter the SDF or T3T1 info.e.g. for SDF type 45-13e.g. for T3T1 type T3T1-45-13and press ENTER.

6. DS1 Digroup Confirmation Screen

To execute your choice, Type Y and press ENTER.Or press ENTER to select defaults for the DS1 monitor.

7. Full T-1 Summary Category Screen

Note results. RX PAT =received pattern RX FRM =received framing (ESF for live data) E.Time = test time SPLIT TYPE =N/A for monitor

Type letter at screen bottom for details L = logic S = Signal B = BPV T = Time F = Frame I =Line Status G = G.821 H = Hot Key Q = Quit

8. Full T1 Summary Category Screen

Note results.e.g. RX PAT 1111 (all ones or A1’s) RX FRM Unframed Sync w/Alarms

Type Q to quit this screen. You are returned to the DS1 Monitor Screen automatically.

Or go to Step 10 to intrusively test circuit.

9. DS1 Test Monitor Screen To end the session, At DS1_MON>

Type disconnect to terminate a test, press space. Type release, press space. Type yes, press return, and, at the prompt, enter Y to finish.

10. Full T-1 Summary Category Screen

Note results.e.g. Rx Freq = 154100

Type quit to return to DS1 Monitor Screen.

11. DS1 Test Monitor Screen To test for (E Split??),At DS1_MON>

Type ds1_split and press ENTER.

12. DS1 Split: Screen Note results.e.g. Test Name = TS1 1 Direction = E Split Code = Local Loopback

Type Y to continue test with selected parameters.

Step Centest Screen Purpose Procedure13. DS1 Loopback: Screen Confirm next test at screen bottom.

e.g. DS1_TST>loobpack up FAC2.At DS1__TST>

Type loopback and press ENTER.


36


14. DS1 Test Menu Screen To proceed with test, At DS1 Test Menu

Select the loopcode direction, or hit ENTER for default: Up.

15. DS1 Loopback: Screen To proceed with test,AT DS1_TST>

Confirm parameters and type Y to execute test.

16. DS1 Test Menu Screen At bottom of screen, note test you requested is in progress.e.g. FAC2 Loop Up SEL: In progress.Note test complete.e.g. FAC2 Loop Up SEL: Confirmed.To select another test,At prompt

Type ds1_measure and press enter.

17. DS1 Test Monitor Screen To proceed with test,At DS1_TST>

Enter parameters for measure test, e.g. ds1_measure 3 in 24.

18. DS1 Measure: Screen Confirm test parameters are correct.

Type Y to execute.


Note results of test.For Looprun,e.g. RX FREQ = 1544212.


Note results of testfor Measure,e.g. Freq + 1544235.

21. Full T1 Line Status Category Screen

Note results of all tests,e.g. measure and looprunNote T-1 line fault found.e.g. B8ZS Detected Yes at bottom of screen.

Return to DS1 Test Menu Screen (HOW?)

22. DS1Test Menu Screen For next series of tests, At DS1_TST>

Type loopback.

23. DS1 Test Menu Screen To test loop in opposite direction from previous test,At DS1_TST>

Type Loopback down and press ENTER>.

24. DS1 Test Parameters MenuTo choose a loop code, Type choice, e.g. loopback Fac2press ENTER for default choice CSU.

25. DS1 Loopback: Screen To confirm test parameters, Type Y to continue.


Note outcome of test for parameters, e.g., For measure: Signal present.


37


Isolating and Troubleshooting a T-1 Outage for an EBTS (NorCal)A T-1 failure is recognized by an X.25 failure event, which literally means the X.25 path between the EBTS and the OMC has failed, usually indicating a T-1 failure. However, it is also possible that the second DSO on the T-1 (also known as SNMP) may fail, or for the composite link from the CP to the OMC (also known as the EOML) to fail, representing a lack of visibility but no customer impact.

Fault Monitoring Procedure Use the Fault Management Tool to identify an X.25 failure, or note that icon on the Site

Status Display has turned red or blue to identify a T-1 failure. Open a NTM case. Use Danny’s Database/NCLR Database to identify the Location of the T-1. Monitor (non-intrusive) the T-1 with Centest to see if it is down or taking errors. Verify site status, if necessary, by checking the subrates of the sites in the XCDR to see if

there is any traffic on them. (Subrate should be in ‘Active’ state with traffic, ‘idle’ without, and occasionally and only for ½

a second ‘active reserve’ when the call is in the process of being set up. Even one Subrate in ‘active’ represents the site is not down. ‘Blocked’ or ‘Active reserve’ that stay ‘Active reserve’ represents a XCDR issue that needs to be referred to the MSO).

Notify TSG by desk extension 2401 of Site outages. Assuming T-1 has been verified in a failed (OOS/CGA) or impaired state, (Excessive,

constant errors BPV/CRC/Slip/Errored Seconds) Go intrusive and loop the NIU and test. If unable to loop the NIU, log ticket out to carrier. NEED TO ATTACH OR IDENTIFY CARRIER IDENTIFICATION PROCESS. Follow up with carrier and escalate with carrier if they are not providing adequate and

prompt service. If test to NIU is good, loop up CSU and test. If test to CSU is good or if it fails, (Assuming Site does not mysteriously clear and start

working) contact the Field technician. Update NTM case.


38


Isolating and Troubleshooting a T-1 Outage for an EBTS (PNW)A T-1 failure is recognized by an X.25 failure event, which literally means the X.25 path between the EBTS and the OMC has failed, usually indicating a T-1 failure. However, it is also possible that the second DSO on the T-1 (also known as SNMP) may fail, or for the composite link from the CP to the OMC (also known as the EOML) to fail, representing a lack of visibility but no customer impact.

Fault Monitoring Procedure Use the Fault Management Tool to identify an X.25 failure, or note that icon on the Site

Status Display has turned red or blue to identify a T-1 failure. Open a NTM case Use TOED Database/NCLR Database to identify the Location of the T-1. Also, in the Field

Ops web page are the vendor circuit id, and other pertinent site information. http://home.cybergs.com:802/fieldman/main.html

Verify site status, if necessary, by checking the subrates of the sites in the XCDR to see if there is any traffic on them.

(Subrate should be in ‘Active’ state with traffic, ‘idle’ without, and occasionally and only for ½ a second ‘active reserve’ when the call is in the process of being set up. Even one Subrate in ‘active’ represents the site is not down. ‘Blocked’ or ‘Active reserve’ that stay ‘Active reserve’ represents a XCDR issue that needs to be referred to the MSO).

Notify TSG by desk extension 2401 of Site outage. Assuming T-1 has been verified in a failed (OOS/CGA) or impaired state, go intrusive and

loop the NIU and test. If unable to loop the NIU, log ticket out to carrier. NEED TO ATTACH OR IDENTIFY CARRIER IDENTIFICATION PROCESS. Follow up with carrier and escalate with carrier if they are not providing adequate and

prompt service. If test to NIU is good, loop up CSU and test. If test to CSU is good or if it fails, (Assuming Site does not mysteriously clear and start

working) contact the Field technician. If the site clears, verify the site status, if necessary, by checking the subrates of the sites in

the XCDR to see if there is any traffic on them. Update NTM case.


39



EBTS/T-1 Fault Reporting Diagram The core functional responsibilities of the EBTS Monitoring Center are illustrated below. The center serves a network surveillance role for maintaining the Nextel’s operations network systems integrity. The diagram below sets forth the EMC commitment level by activity, owner, and other elements of the fault response process.


40


EBTS/T-1 Fault Diagnosis Diagram


41


Processing AlarmsTypes of Alarms

Type of alarm Identified by Action to Take Resource to UseEnvironmentalAll 200+ series of alarms

Fault Monitoring Tool Notify field or facilities technicians – See Environmental Notification Procedures below.

Alarm Codes FlowchartAlarm Code Definitions

High Temp Alarm subset of environmental

Fault Monitoring Tool Notify field or facilities technicians – See Environmental Notification Procedures below.

High Temp procedure and Alarm Code Flowchart (NorCal)Alarm Code Flowchart (PNW)

Power Outage Alarms subset of environmental AC power outage/ Utility power loss alarm PA Power Rollback alarm Tower Top Amp alarm

Fault Monitoring Tool Notify field or facilities technicians - See Procedure Below.

Alarm Code Definitions and Power Outage Procedure (NorCal)Alarm Code Definitions (PNW)

Base Radio Site Failure

Fault Monitoring Tool Notify on call field technician – See Reporting Base Radio Outages below.

Alarm Code Flow Chart

T-1 Failure (X.25) Fault Monitoring Tool Investigate and troubleshoot T1.

Use NorCal NCLR Database, and Centest

Note: Create new NTM tickets for all incoming alarms. Verify through previous tickets that this alarm is new, and is not already being addressed.


42


Environmental AlarmsNote: These are all of the 200+ series of alarms (see NCAL and PNW Alarm Codes and Severity in Appendix). Power and Temperature alarms are subsets to Environmentals, and have their own, specific procedures (see Environmental Alarm Actions in Appendix). See Environmental Alarm Actions Flow Charts for specific alarm procedures. Fault Monitoring Procedure Use a Fault Management Tool, depending on several factors to identify an environmental alarm. Identify whether the alarm is during or after business hours. Identify the alarm’s criticality level. Identify whether the alarm is a field or facilities issue. Field notification is site specific; use the NorCal web site/Tom Smith Web site. Notify Facilities via desk extension/v-mail. Open NTM Ticket.

High Temperature Alarm: NorCalWhen any Fault Management Tool receives a high temperature alarm, the following steps should be taken:

Fault Monitoring Procedure Wait 15 minutes of alarm create time to ensure the alarm is not caused by hot air from HVAC

during startup. Open NTM ticket. Call Ext. 8802. Enter password when prompted. Press 2 for “Record Message”. Record message with alarm details. Site # Site name Time MSO contact NTM ticket number Press # to end message. Press 5 for sending options. Press 1 to send as urgent. Press 9 for the group options. Press 20 for the group list. Press # to send the message. Update NTM ticket as technician provides updates.


43


High Temperature Alarm: PNWWhen a high temperature alarm is registered on any Fault Management Tool, the following steps should be taken:Fault Monitoring Procedure Wait 15 minutes of alarm create time to ensure the alarm is not caused by hot air from HVAC

during startup. Open NTM ticket. Notify the appropriate technician and follow escalation procedure. Update NTM ticket as technician provides updates.


44


Power Outage Alarm Note: Field Technicians are first level support for emergency generator deployment. If for some reason the assigned or On Call Field Technician can not deploy the generator, the responsibility is shifted to Facilities to respond. The Facilities Team is responsible for all scheduled power outages.To get the latest updates on power outages To make an initial power outage report to PG&E, call 1-800-743-5000. To locate Nextel field technicians, refer to Tom Smith’s Database:

http://www.oaknextel.com/home.ephelps/index.htmlTo identify site Location, use Site Track database (also called facilities database):

g:\eng\facildb\siteform.mdb/wrkgrpg:\eng\facildb\system.mdb/user Ops. Fault Monitoring Procedure Use the Fault Management Tool to identify a Power Outage alarm. Alarm will appear on the normal viewer. If alarm shows one of the following, dispatch a Field Tech during normal business hours; if alarm

shows two of the following, treat as a valid power outage: Code 206 utility power loss Code 223 surge arrestor Code 242 PDB A/C power circuit failure Code 247 PDB Rectifier Major Look on OC/MSO White Board/Information Center and check latest email from Ed Hall to

determine whether temporary generator is powering site. If yes, notify Facilities, continue to monitor, and request hourly updates from Facilities or

appropriate contractor. Refer to Facilities’ database for the site’s power provider contact number and the site’s power

meter number. If database does not contain information, contact Mike Miller or appropriate Facilities

representative immediately. Identify site’s power provider. Report outage to power provider, provide meter number and request

an ETR. Notify Field Technician of power outage and power provider’s ERT. Include all alarm information.

If Field Services deems it necessary to deploy a generator, provide Field Technician with Location and status of available generators.

Notify Field Technician when and if power alarm clears. Follow up hourly with power provider to request updates until commercial power is restored. Provide Field Technician with all status changes. Update NTM ticket.

NorCal single site power outage DiagramEMC Training Manual.doc

04/11/2345

http://www.oak-nextel.com/home.ephelps/index.html



46


Power Outage Alarm – Multiple Sites NorCalNote: A multiple power outage is defined as more than 5 outages seen on a single OMC in a 1 hour timeframe.

Fault Monitoring Procedure1. Notify the technicians associated with the sites.2. Send a voice Mail to distribution list 33 marked urgent.3. The voice mail will include:

Site names/site numbers and the OMC Give conference bridge information (510) 208-8544, pass code 3704# or (510) 208-8540 Pass

Code # 06044. If neither number is available, call (888) 5 NEXTEL and advise McLean NOC of bridge

requirements.5. Advise participants to avoid using speakerphone and to mute when possible to reduce background

noise. 6. Update NTM ticket.


47


Multiple Power Outage Process (For All Other Markets)This process will be utilized to ensure timely response by Field Services Team members to multiple power outage situations. This process will ensure sufficient resources are engaged as well as an efficient action plan for generator deployment is utilized. Follow the procedure outlined in the NCAL Multiple Power Outage Process for Conference Bridge setting. NOTE: Outage interval is 5 power outages in 1 hour on the same OMC. .


48


Power Outage Alarm – Multiple Sites (All Other Markets) Note: A multiple power outage is defined as more than three outages seen on a single OMC in a two-hour timeframe.

Fault Monitoring Procedure1. Notify the technicians associated with the sites.2. Contact the Field Operations Manager for the market, if no answer leave a voice mail and mark as

urgent.3. The voice mail will include:

Site names/site numbers and the OMC Give conference bridge information (510) 208-8544, pass code 3704# or (510) 208-8540 Pass

Code # 06044. If neither number is available, call (888) 5 NEXTEL and advise McLean NOC of bridge

requirements.5. Advise participants to avoid using speakerphone and to mute when possible to reduce background

noise. 6. Update NTM ticket.


49


Base Radio (BR) Alarms-NorCal

Note: The assigned technician and the on call technician prioritize the workload to determine if help is needed to restore sites to full functionality. The goal is to return each BR (base radio) to service within a two-hour period. It is up to the technician to contact the field team leadership if assistance is required to achieve these goals.NorCal Site Technician contact information is on the Tom Smith web site:

http://www.oak-nextel.com/home.ephelps/index.html. If the site is down or does not load, contact Field Technician Eric Phelps at (916) 997-8913 or dispatch him at 1161 (preferred). Base Radio Outage: NorCalWhen a BR alarm indicates a power outage, take the following steps immediately:Fault Monitoring Procedure - NorCal Use the Fault Management Tool to identify a BR alarm. Alarm will appear on the normal viewer. Site Status will turn yellow if it is not already yellow on the OMC site icon. Follow the BR Report Procedure: - During the hours, 8:00 am to 5:00 PM Monday through Friday, contact the assigned technician. - During the hours 5:00 PM to 10:00 PM Monday through Friday, contact the on call technician. After 10:00PM Hold the ticket and notify the assigned technician at 7:00AM. Update NTM ticket.

Note: The following BR outages should be treated as a site outage and reported immediately to the assigned or on call technician.

Outages to multiple BR(s) on a single site. High VSWR alarms. These alarms indicate an antenna or path problem that could affect

multiple BR(s). Power roll back alarms. If the alarm occurs on the PCCH BR, first reset the BR then call the

appropriate technician.

In addition, Technicians will continue to report the BR outages via the Morning Report, stating that the site technicians have been notified.


50

http://www.OAK.Nextel.com/home.ephelps/index.html


Fault Monitoring Procedure – PNW/Denver

Use the Fault Management Tool to identify a BR alarm. Alarm will appear on the normal viewer. Site Status will turn yellow if it is not already yellow on the OMC site icon. Follow business hour notification as noted on the Environmental Alarm flowchart. Page 57 & 58. For contact information refer to the National Network Operations Center (NNOC) http://ossinfo.nnmc.nextel.com/noc/ Update NTM ticket.


51


NOTIFICATION AND ESCALATION PROCEDURESNextel Field Tech Notification ProcedureFollowing initial notification, follow up every 15 minutes until a response is received.

Business Hours Notification Monday – Friday 7 am to 5 pm NCAL Monday – Friday 7 am to 6 pm PNW

Initial Field Service Alarm notification should be to assigned field tech PTN (find technicians’ home phone numbers in NorCal and PNW Technician Access databases), and Dispatch (NCAL).

First escalated notification should be to assigned field tech Pager. Second escalated notification should be to assigned field tech home.

After Business Hours NotificationMonday – Friday 5 pm to 7 am NCALMonday – Friday 6 pm 7 am PNWAll day Saturday, Sunday, and Nextel holidays

Initial Field Service Alarm notification should be to oncall/duty field tech PTN (find technicians’ home phone numbers in NorCal and PNW Technician Access databases), and Dispatch (NCAL).

First escalated notification should be to assigned field tech Pager. Second escalated notification should be to assigned field tech home.


52


Telco Trouble Ticket Content and Notification ProceduresHave the following details ready for telephone company technicians who need to create internal trouble tickets when processing Nextel alarm notifications:

Circuit ID (refer to NCLR database) End user’s address (refer to site map) Test results – any of all of the following:

DS1 – channelized Red alarm, A1’s, yellow, etc. Loss of framing/frame loss No PCM CRC errors Framing errors Bipolar violations Timing slips

Intrusive test authorization, if applicable Your first and last name if authorizing technician dispatch, and ask repair tech to call with

ETA Local Contact for EMC Group (510-208-8802) EMC access hours (24/7)

Note: When requesting trouble tickets for Performance Monitor (non-intrusive), state so, and do not give any test results or the ticket will be treated as a measured trouble report instead of an informational ticket.Note: Be sure to get telco internal trouble ticket number for future reference, as well as names of company technicians assisting you.

*NorCal added procedureAt 7am call Pacific Bell Service Executive Nilene Brown at 510-986-7230 and provide her with any Pacific Bell outages that have a first AM dispatch. Leave a voice mail to include the trouble found and the Pacific Bell ticket number.


53


Contact InformationContact How to Reach Special Instructions

Nextel Field TechniciansNorCalRichard Varela

Richard Ruiz

PNWRobert Erickson

Jim O’Brien

Tom Smith Web Site916-997-8915 PTN 3426 cell ID916-997-8908 PTN8908 ID916-859-4208 Office916-369-0103 Home650-222-6210 PTN1957 cell ID510-774-6741 PTN6741 PTN ID650-368-5838 Home888-448-1942 Pager

425-251-5739 Office360-866-4173 Home206-730-1861 Iden206-977-3156 Page 1861 Private ID

503-277-5555 Office503-623-8353 Home503-209-5555 Iden503-271-0383 Page 5555 Private ID

Refer to Tom Smith Web Site for more tech contact info.

Refer to PNW Technician Database for more tech contact info.

Tech Support Group (TSG) X2401 If after hours, leave message with site name and number, outage start/stop time and NTM ticket number. If reporting a DS3, BSC-CP, or BSC-XCDR mark message as URGENT.

Facilities Voice Mail Box 20 For facility alarms, NorCal only. Mark URGENT for after hours contact.

EMCNathanGail

x8867x8873

Mark URGENT for after hours contact.


54


Contact How to Reach Special InstructionsTelco/Telephone CompaniesPacific BellInitial Report-open TTEscalation Desk 1st, 2nd, 3rd Level Escalations4th Level Escalation Nilene Brown, Service Executive

800-388-7299

800-922-7742

510-986-7230 Office800-276-8439 Pager

For first escalation, ask for the name of the first level representative and request a callback from the manager. For second level escalation, request a callback, and/or page the second level manager. If no response within 30 minutes, call Service Executive for assistance. (Use PacBell’s instructions as a model for all telco communications.)

Nevada BellAll ServicesCenter/Escalation

Larry Gibbs

800-880-0293

775-333-3162 Office775-325-5707 Pager

See Pacific Bell Special Instructions.

Qwest/SprintSprint Contact NumberFirst Level EscalationTroy Sedlock, Sr. Acct Manager Second Level EscalationBrian Bullock, Field Support ManagerThird Level Escalation Deanna Kuszmaul, Director Wholesale Markets

800-743-1885

614-336-4906 Phone614-798-6460 Fax

614-799-7552 Phone614-798-6460 Fax

614-798-6075 Phone888-956-9583 Pager614-519-0811 Cell614-798-6460 Fax

Nextel Customer No:4233-58164

Current Ticket Number

See Pacific Bell Special Instructions.

MCI/WorldcomOpen Trouble Ticket Follow up on TTOpen Trouble Ticket and Follow ups

800-444-6600800-864-6111888-858-9226

Circuits beginning with:M,Z,E,P,F (Chicago-MCI)Circuits beginning with a number or with W(Sacramento-Worldcom)

See Pacific Bell Special Instructions.Roseville Telephone 800-639-1611 See Pacific Bell Special Instructions.


55


OSS trouble ticket procedure

Go to the OSS web page at http://ossinfo.nnmc.nextel.com/new/

Click on the link to ‘Trouble Ticketing’

Click on the link to ‘Login’

Type in the following for Username: pac_loc Password: nextel1

Click Login after typing the login information

*The page will display all tickets for this login

To open a new trouble click on 'request work' in the upper right corner

Enter the appropriate information into the ticket. Severity should be Critical and the Location will depend on what the trouble is for i.e. Northern California Region for the EMC, Element is the tool causing trouble i.e. Measles/Rubella

Enter the Comments. Include as much detail as possible include the platform you are using when you experience the trouble, the nature of the trouble and how tools should normally act

*Click 'view requests' then 'Show My Open Tickets' to view tickets open under this login


56

http://ossinfo.nnmc.nextel.com/new/


Business Hour Alarm Notification

The core functional responsibilities of the EBTS Monitoring Center are illustrated below. The center serves a network surveillance role for maintaining the Nextel’s operations network systems integrity. The diagram below sets forth the EMC commitment level by activity, owner, and other elements of the fault response process.


57


After Business Hours Alarm Notification

The core functional responsibilities of the EBTS Monitoring Center are illustrated below. The center serves a network surveillance role for maintaining the Nextel’s operations network systems integrity. The diagram below sets forth the EMC commitment level by activity, owner, and other elements of the fault response process.


58


Procedure: EBTS Monitoring Center - NTSG Site Outage Notification Procedure

Issuing Department EBTS Monitoring Center Policy Type DepartmentalDocument Number MNMF2.100 Date of Issue July 17, 2001

Version Number 1.1 Revision Date New

Replaces 1.0 Contact Gail Allen510-208-8873

Introduction This procedure has been developed to provide guidelines for issuing NTSG EBTS site outage notifications when EBTS sites have failures that take an individual site out of service in a given market.

Procedure Application

This procedure is applicable to EBTS monitoring center Level I Technicians involved with the monitoring of EBTS network alarms via the network management systems/tools resident in the NNOC and for managing and resolving EBTS network troubles. NTSG EBTS Notification requirements listed in this procedure is required by

established SLAs between the NNOC and NTSG organizations.

Application of this procedure is expected to be a continuous practice, rather than on an exception basis.

All NNOC personnel must understand their responsibilities in applying this procedure.

Failure to issue NTSG EBTS site outage notifications in accordance with this procedure may result in unnecessary escalations, additional calls from customer care to operations and lengthened network and service outages and under performance relative to established SLAs.


59


EBTS Monitoring Center (EMC)/NTM Ticket Guidelines

Ops (Operations) tickets are used to track site and equipment problems from first report to resolution. The Ops Ticket application provides tools to query, create, dispatch, assign, and escalate trouble tickets, and to notify groups and users of pending trouble tickets. Ops Tickets track all incoming calls, information requests, system outages, and cycle times. A case number is assigned to each problem and remains open until the root cause is identified. Any EMC member can create a ticket, thereby initiating the process.

Every event requires a separate ticket (e.g., weeds, leaking batteries, and missing EMC may all be reported at one site; each gets its own ticket)

Verify that you are not opening a duplicate ticket. Queering by Title or EBTS ID performs this check. Please read existing tickets to verify for ticket duplication.

Fill in fields, as indicated below:

Ticket Creation Guidelines

1. EMC sees and acknowledges alarm in Net Expert (new step)

2. EMC creates ticket in NTM (guidelines below)

A) Case Title Field:

The standard naming convention for all tickets created by the EMC is as follows:MSO_ELEMENT_SITE NAME_DESCRIPTION

I.e. - NCAL1_CA0607_Oakland_PA Rollback BR1_1 NCAL2_CA0496_Fairwood B_ High Temp

MSO:

1. NCAL - Northern California2. DENV - Colorado and Utah 3. PNW - Washington and Oregon4. HAWA- Hawaii5. CARO- Carolina’s6. TENN- Tennessee7. FLOR- Florida

Element:1. Make sure to use the 6 character alphanumeric code for the site #. We use this code to query by

Title.

B) Cell Site Tab:

Fill out the Cell Site Tab whenever the site ID has been identified. This procedure also applies to the Site Visit Log; there should be an entry per site. This will take a few additional minutes on your part but is required.

C) Telco Vendor Tab:

Fill out the Tele Vendor tab for all circuit issues. Select the Add Circuit in the Telco Vendor Tab. Then, type the 6 character alphanumeric EBTS identifier and hit return or select List. This will provide you with a list of circuits the selected circuit should at the top of the list.


60


D) Ext. Vendor Tab:

Use the Ext. Vendor tab to fill in information for all vendors and site technicians

3. EMC calls appropriate Field Tech and notifies of problem

4. Field Tech calls EMC to status ticket

5. Issue resolved, EMC changes status to Closed in NTM and clears alarm in Net Expert


61


Nextel Partners

Partner Region Partner Area Partner MarketPartner HI Partner HI Partner HIPartner North All AllPartner North Partner Central PA Partner HarrisburgPartner North Partner Dakotas AllPartner North Partner Dakotas Partner NDPartner North Partner Dakotas Partner Sioux FallsPartner North Partner ID Partner IDPartner North Partner IL-IN AllPartner North Partner IL-IN Partner PeoriaPartner North Partner IL-IN Partner SpringfieldPartner North Partner IL-IN Partner Terre HautePartner North Partner IA-NE AllPartner North Partner IA-NE Partner Des MoinesPartner North Partner IA-NE Partner Eastern IAPartner North Partner IA-NE Partner OmahaPartner North Partner MN AllPartner North Partner MN Partner DuluthPartner North Partner MN Partner Eau ClairePartner North Partner MN Partner Rochester MNPartner North Partner NY-Erie AllPartner North Partner NY-Erie Partner AlbanyPartner North Partner NY-Erie Partner BuffaloPartner North Partner NY-Erie Partner EriePartner North Partner NY-Erie Partner Rochester NYPartner North Partner NY-Erie Partner SyracusePartner North Partner NY-Erie Partner VTPartner North Partner WI Partner Green BayPartner Southeast All AllPartner Southeast Partner Mid South AllPartner Southeast Partner Mid South Partner KYPartner Southeast Partner Mid South Partner Southern INPartner Southeast Partner Mid South Partner TNPartner Southeast Partner Mid South Partner VAPartner Southeast Partner Mid South Partner WVPartner Southeast Partner Southeast AllPartner Southeast Partner Southeast Partner ALPartner Southeast Partner Southeast Partner FLPartner Southeast Partner Southeast Partner GAPartner Southwest Partner Southwest AllPartner Southwest Partner Southwest Partner ARPartner Southwest Partner Southwest Partner LAPartner Southwest Partner Southwest Partner MSPartner Southwest Partner Southwest Partner TX


62


Facilities Tickets

1. EMC sees and acknowledges alarm in Net Expert (new step)

2. EMC creates ticket in NTM

3. EMC calls Facilities to notify of ticket

4. EMC dispatches ticket to the West Region Facilities queue

5. Facilities owns addressing problem, updating and closing ticket

Field Operations (FO)/ Mobile Switch Office (MSO) Tickets

1. Follow the EMC/NTM ticket guidelines Above

2. If you are actively working an issue, keep the ticket. If the ticket is not being actively worked, and you don't expect an update within your shift or the next, dispatch it to the appropriate FO/MSO/BSO queue.

3. If the ticket is and MSO related issue, assign the ticket to the appropriate MSO queue, preceded by a phone call to let the MSO know about the ticket assignment.

Authorized FO queue:

a. CA FO NorCal Northern California Field Operationsb. WA/OR FO- Washington/Oregon Field Operationsc. CO FO-Colorado Field Operationsd. UT FO-Utah Field Operationse. Partners Southwest-Partners Idaho Field Operations

Authorized MSO queue:

a. CA MSO Dublinb. CA MSO Oaklandc. CA MSO Santa Clarad. CA MSO Rancho Cordova (Sacramento MSO)e. CO MSO (Colorado MSO)f. UT BSO (Utah and Partners Idaho MSO)g. WA MSO Seattle (Washington and Oregon MSO)

4. Don’t dispatch tickets for the Hawaii market. If the ticket isn’t being worked on dispatch it to the WNOC North Queue.

Suggested Queries

These are some of the suggested shared queries that can be “iconified” to minimize ticket search time.

1. WNOC_All NCAL open tickets NOT created by #NXTLOAKLCA01 and NOT in Open-Dispatch

2. WNOC_All NCAL open tickets created by #NXTLOAKLCA01 and NOT in Open-Dispatch


63


3. WNOC_All PNW open tickets NOT created by #NXTLOAKLCA01 and NOT in Open-Dispatch

4. WNOC_All PNW open tickets created by #NXTLOAKLCA01 and NOT in Open-Dispatch

5. WNOC_NCAL Closed within the last 24 hours

6. WNOC_North Open Mountain Tickets created by #NXTLOAKLCA01

7. WNOC_North Open Mountain Tickets minus Newt

8. WNOC_North Personal Ownership of tickets not in Open-Dispatch

9. WNOC_North Contractor Ownership of tickets not in Open-Dispatch

Note: In NTM, you can create your own queries, but remember, Never “Iconify” a changeable query. For more information on how to create or use queries, refer to Chapter 9 in the NTM training guide. If you don’t have one readily available, contact you shift supervisor.

NTM (Clarify) OPS Ticket Preparation Suggestions

NTM FIELD FILL INOutage Start Time Time stamp of original outage/alarm (not the time ticket is created)

Outage Stop Time Time stamp of clear indication of event (not the time ticket is closed)

Priority Critical For N/E issues where service is denied to a large portion of customers (BSC, DAP, MSC, HLR, DACC, 20 or more EBTS, etc.).

Major/Non Customer For redundancy loss on N/E that has a potential to deny service to a large portion of customers (CPU divergence on DAP, A_B side out of sync on MSC, Loss of a plane in the DACCS).

Major Hardware or software failures that deny or impair service to a significant number of customers. (Single IGX failure, Single APD failure, DS3 outages.

Minor For loss of service or impairment to a limited number of customers. (1-4 EBTS outages, single card failure in any N/E that does not cause a higher level impact, 25% or greater failure of any EBTS , any call processing issue).

Info For all non-customer impacting issues. (Most EBTS environmental, facilities issues).

NTM FIELD FILL INStatus New Default status after creating a ticket or assigning a ticket

and the target has not accepted ownership yet.Work In Progress Ticket should remain in this state while you are actively

working the ticket (contacting the tech, waiting for a


64


response from the tech, actively working with Pac Bell, etc).Pending Customer Internal/External

Use when no active work is being done on site (Tech is 8 hours from site, waiting on parts, etc).

Solution Implemented Issue identified and corrected, root cause identified, stop time put into ticket.

Closed OC-Manager or OC-Supervisor reviewed ticket and was satisfied with the information provided.

Monitor Problem has been identified and believed corrected (T-1 up, monitoring ticket prior to taking to resolved).

Vendor Ticket Number Use all caps (SN156746)

Circuit ID Use all caps (70HCGS762010147)

Root Cause Root Cause This pull down menu is required to close a case. Select the appropriate Root Cause.

Duplicate Ticket Ticket already exists.

RF/Site Element For any problem diagnosed as an environmental that does not fit another root cause category.

T3/DS1/Telco Used only when swapping out hardware was diagnosed as the hard fix (T-1 down, swap CSU, T-1 comes up. Not, rolled to redundant ISC)

NTF/Unknown Used when no other category fits the root cause.Power Commercial power outages only (Not a rectifier failure at the

site). Also, if the site is operating on generator only.Routine Maintenance Ensure that the outage selected is equal to Planned.

Use only for outages that were previously schedule. Datafill Misconfigured of software

(Airgen changes value in load to restore).Telco Used when telco acknowledges fault and gives a hard fix.

(If you test and KNOW it is a telco problem, but telco does not acknowledge fault, state that in diary, but close ticket with root cause as Unknown).


65


NTM FIELD FILL INRoot Cause Translations Call processing issue diagnosed to a MSC translation

issue.

User Error User error. (e.g., Tech admits testing wrong T-1).

Description A brief synopsis of the observed symptoms, or suspected cause. It can include a suggestion when the update should happen (This field can be changed, so all information in this field should be duplicated in the Work Log).

Work Log For every entry, include Name and Department of people you talk to (Carl/PacBell, Joe/SacMSO, Bob/Field, etc).

Every ticket should have a plan of action, ETA, and ETR. Anyone who reads the ticket should know, not have to guess, what the next step is for problem resolution.

Include time stamp of all events in work log. (e.g., Started test at 09:12; completed test at 09:39; called Bob/Field at 03:32; paged Bob/Field at 3:49; Bob/Field returned call at 3:55, etc).

Include all steps taken to reach the current status. (e.g., High temp alarm came in at 15:15. Called Bob/Field (510-555-1212) and left Voicemail at 15:19, received user not available on Private (1212). Bob/Field called in at 15:28. )

Last entry should concisely restate fix action and root cause with the exception of when it is held in a monitor status for X number or hours, and then resolved. (e.g., ‘Next to last entry’ PacBell swapped NIU to restore at 09:32, monitor for four hours. “Last entry” No errors for last four hours, resolving ATT.)

Site Visit Log Note: The Site Visit Log tracks ALL TRAFFIC on a site.

Ensure that you have selected the correct site visit log. PNW, NCAL or Denver. Hawaii site visits are log in the PNW Site Visit Log.

When noting a site visitor logged in, get time stamp from monitoring tool. If site visitor plans to log in but isn’t there yet, ask if the visit will be one, five or ten minutes

and note this information in the log. All site entries should have accompanying text in the work log explaining the actions and

contact information. Ask these questions:

How long will you be there? What impact is your work going to have on the site? (i.e., will I see any alarms, and if so, what

type)? Add the EBTS number under the “Select EBTS”

Before you clear any 219 alarms in NetX, note the clear time and make a note in the site visit log if it is not up-to-date.


66


TRAINING CHECKLIST REVIEW TEST AND SIGNOFF

Name: _________________Date Started: __________________

Estimated Completion Date: ____________________

Type of Training Date of Training/ Date Demonstrated/ Trainee Initials Lead Tech Initials Lead Tech Initials

Network Elements ___________ ____________ ____________ Accessing and Using FMT

Logging on/off X-term ___________ ____________ ____________ Logging on/off the OMC ___________ ____________ ____________ Launching The MMI Form Laptop ___________ ____________ ____________ Setting Up the OMC for FM ___________ ____________ ____________ Using the OMC to Monitor Faults ___________ ____________ ____________ Using OMC to query BR power ___________ ____________ ____________ Using OMC to reset BR ___________ ____________ ____________ Launching, Logging and Off NetX ___________ ____________ ____________ Setting Up NetX for monitoring _________ __________ __________ Launching, Logging on/off Measles ___________ ____________ ____________ Setting Up Measles for monitoring ___________ ____________ ____________

1. Round- the Clock Procedures EMC Shift Change responsibility. ___________ ____________ ____________ EMC Daily Report ___________ ____________ ____________ NTM OPS Ticket Preparation ___________ ____________ ____________ Site Visit LOG ___________ ____________ ____________ App Tracking ___________ ___________ ____________ Continues Site Monitoring ___________ ____________ ____________ Tech ID and Site Info Database ___________ ____________ ____________

2. Testing T-1 Lines Acronyms Used to test T-1 ___________ ____________ ____________ T-1 Testing Resources ___________ ____________ ____________ Using Centest to Test T-1 ___________ ____________ ____________ Isolating and Troubleshooting A T-1 __________ ____________ ____________

6. Outage for an EBTS EBTS/T-1 Fault Reporting Diagram ___________ ____________ ____________ EBTS/T-1 Diagnosis Diagram ___________ _____________ ____________

7. Processing Alarms Environmental Alarms ___________ ____________ ____________ High Temp Alarm :NorCal ___________ ____________ ____________ High Temp Alarm: PNW ___________ ____________ ____________ Power Systems ___________ ____________ ____________ Power Outage (Single NorCal) ___________ ____________ ____________ Power Outage (Multiple NorCal) ___________ ____________ ____________ Base Radio Alarms ___________ ____________ ____________

8. Notification and Escalation Procedures Business Hours Notification ___________ ____________ ____________ After Business Hours ___________ ____________ ____________ TELCO T.T. Content and Notification ___________ ____________ ____________ Contact Information ___________ ____________ ____________


67


Business Hour Alarm Notification ___________ ____________ ____________ After Business Hours

Alarm Notification ___________ ____________ ____________

The trainee has eight weeks to complete the training manual from the date that it is presented to him/her. The trainee is responsible for meeting the Estimated Completion Date (ECD). If unable to complete this task by the ECD contact your Lead Technician or The EMC manager.

Lead Technicians will provide weekly progress reports to the EMC manager.

Trainee Signature: ________________

Lead Technician Signature: ______________

EMC Manger Signature: ______________


68


EBTS Monitoring Center

Training Manual

Reference

Appendix


69


A. iDEN Network Elements Slide Show


70



71



72



73



74



75



76



77



78



79


B.NCLR Screen Layout tutorial


80



81


C.Measles Screen Sample and Overview Summary

Measles II Executive Summary The Measles application is a real time utility for iDEN infrastructure fault management. The utility gives the user a service level, site centric view of the network and offers a suite of powerful diagnostic tools for fault investigation. The Measles user interface communicates with one or more RUBELLA server processes. The RUBELLA server process is a light weight event translation and storage system which allows the rapid access to current and historical event information. The server collects, correlates and interprets the event stream from multiple sources for presentation by the Measles user interface. A critical feature of the Measles system is the low bandwidth broker mechanism between the RUBELLA server process and the user interface. The event information is translated and compressed by a factor of 1000:1 compared to typical network management systems. This allows the full usability of the system in remote access and wireless data configurations. The Measles application integrates six powerful tools for the real time management of the network information:

Summary View - The Summary view allows the user to select from one or more iDEN Urban networks. Each Urban network is categorized by user definable regions and the real time availability of each region is displayed. Service level problems are displayed by region to give the user a quick and easy starting point for fault resolution.

Status View - The Status view gives the user a regional overview of all the sites within a network. Each site is color coded according to it's customer perceived service level and the display is updated in real time as conditions change.

Chart View - The Chart view gives the user an overview of historical state transitions within the network. The display can show an entire day's worth of information for the identification of trends and relationships in fault behavior. This display is interactive and allows the user access to the underlying event logs for diagnostic purposes.

Detail View - The Detail view allows the user to drill down on a site and view configuration and alarm information within that site. This view also allows filtering of event stream information down to a specific component or managed entity within a site.

Event View - The Event view shows the textual event information coming from the network and from historical event logs. The view shows state transitions in the service levels by color coding the events accordingly.

Reporting Tool - The Reporting tool allows the user to generate downtime reports from various parts of the network. The tool can be used to generate a list of outage events and categorize them by cause and subsystem. The reports can be summarized for identifying problem areas within the network. The reporting tool can also generate reports across different network elements and on various fault types.

Together these tools allow the prioritization and quick resolution of network fault conditions. Through it's service level charting and reporting capabilities, Measles offers the first truly proactive solution for network fault management.


82


Measles II User GuideMeasles II Access http://nnmc.nextel.com:88/docs/guide.html

http://nnmc.nextel.com:88/docs/models.html

The following user guide explains how to use the features of Measles 2.1. Layout

Main View Flow Views Console

Information Views Summary Status Detail Events

Interface Tabs Buttons Boxes Icons

Getting Started Connecting Selecting Drill Down

Tips & Tricks ??


83

http://nnmc.nextel.com:88/docs/models.html

http://nnmc.nextel.com:88/docs/guide.html


LayoutThe new Measles II window consists of five frames. There is a main display frame, a console display in the top left, and a series of three smaller display frames along the bottom.

Main View

Most of the viewing and selection would normally take place in the large display frame. This is the main viewing area and can display all of the available views. These views are selected by clicking on one of the tabs above the viewing area. All the frames are linked together, so if the main view updates, the other smaller ones will.

Flow Views

The three frames along the bottom of the Measles window display a series of smaller views. The left most frame has a selectable view and allows a user to select the stating point in the diagnostic sequence. The center frame is not selectable and will always show the drill-down information for the currently selected site. The final part of the sequence is a static event view in the right hand frame. Together these three smaller frames show the sequence from selection, drill-down to events. They keep the user informed of the various levels of activity performed in the main viewing area.

Console

The upper left-hand frame displays information about what is currently being pointed at and which site has been selected. When the mouse is clicked, the element displayed in the Pointing at section is selected and appears in the Selected section. By clicking anywhere within the console frame area, the selected item is de-selected and the current event stream is displayed in the event views.


84


Information Views

Summary View

The summary view is the user's starting point and comes up as the default view in the main viewing frame. This view allows the user to connect and select which parts of the network to display in the Measles application. The availability of the network is calculated and summarized, allowing the user to determine which parts of the system have the greatest service impairment. The summary view is always ordered by severity, with the greatest impairement towards the top of the display.

The buttons towards the far right of the list allow the user to select and de-select the items. Items that have been deselected are no longer displayed in the other views until they have been selected again. This allows customization of the network view (only view your region or remove sites with known problems). The buttons towards the far left allow the user to drill down to the site level within the list.


85


Status View

Detail View


86


Event View


87


User InterfaceThe new Measles II interface consists of various buttons for selecting, connecting and expanding items in the information displays.

Tabs

Tab De-selected Tab Pointed at Tab Selected

View shown by tab is currently not selected

View shown by tab is currently not selected, mouse over tab

View shown by tab is currently selected

Buttons

BUTTON TYPEDESCRIPTION

DRILLBUTTONS

Button inactive. No information is available to drill down

Information can be drilled, click to expand view

Information has been drilled, click to retract view.

CONNECT/SELECTBUTTONS

Button inactive. Currently connecting or disconecting this item

Ready to connect or select, click to select

Currently connected or selected, click to de-select

Getting startedThe new Measles II window starts with no server connection. In the summary view, click on the right hand connect button of the National Network box. This will load the available systems. Select the one you want by clicking the connect button on the right hand side of it's box. After a couple of seconds, the information appears....still under construction..more to come


88


D.Net Expert Screen Sample and Overview Summary

View NetExpert Manual online in G/eng/Ops Center/Training Manual/NetExpert Manual.


89


E. Vendor Code Summary for Environmental Alarms

ALARM CODES NEW STANDARDS (11/00)201 T1202 Technician On Site (New)203 Quasi Failure204 Auto Tune Combiner Major205 206 Utility Power Loss207 Generator Minor (Fuel)208 Generator Major (Mech)209 Generator Running (Mech)210 211 Fire Suppression Minor212 Fire Suppression Major213 Microwave Minor/Low Pressure214 Microwave Major215 RF Cabinet 6 Tower Top Amplifier216 RF Cabinet 5 Circuit Breaker (Sum)217 RF Cabinet 5 C/M Amplifier218 RF Cabinet 5 Power Supply219 Site Entry220 Site High Ambient Temperature221 Site Low Ambient Temperature222 Site Smoke Detector223 Site AC Surge Protector224 225 RF Cabinet 3 Circuit Breaker226 RF Cabinet 3 Combination/MC Amplifier227 RF Cabinet 3 Combination/MC Power Supply228 RF Cabinet 3 Tower Top Amplifier229 RF Cabinet 2 Circuit Breaker230 RF Cabinet 2 Combination/MC Amplifier231 RF Cabinet 2 Combination/MC Power Supply232 RF Cabinet 2 Tower Top Amplifier233 RF Cabinet 1 Circuit Breaker234 RF Cabinet 1 Combination/MC Amplifier235 RF Cabinet 1 Combination/MC Power Supply236 RF Cabinet 1 Tower Top Amplifier237 Control Cabinet Circuit Breaker238 RF Relay Cabinet 1239 RF Relay Cabinet 2240 RF Relay Cabinet 3241 RF Cabinet 4 Circuit Breaker242 AC Power Failure243 Low DC Voltage244 High DC Voltage245 Breaker Failure Alarm246 Minor Rectifier Module Failure247 Major Rectifier Failure248 Generator Remote Start249 RF Cabinet 4 C/M Amplifier250 RF Cabinet 4 Power Supply251 RF Cabinet 4 Tower Top Amplifier252 Tower Light Alarm


90


F. NOR-CAL Morning ReportGood morning, this is Your Name with the Nor-Cal Daily Report for Friday, Oct 27, 2000BR ISSUES:

MSO SITE NAME STATUS TICKET #SC CA-1113 N. Fremont 51 OOS Failed Rem. #11242

CA-411 Whipple/880 31 PA Power Rollback alarm Rem. #11251

SAC CA-1062 Pittsburg Pt 21 Receiver Antenna Branch Imbalance alarm

Rem. #11224

CA-371 Rio Linda Blvd 22 OOS Failed Rem. #11250

“The technician(s) associated with this/these site(s) are in the process of being notified.”SITE ISSUES:

MSO SITE NAME SITE ISSUE/STATUS TICKET #OAK CA-1909 Harbor Blvd. Sec. 1 will be off the air from 7am to 5pm today

for PG&E Tower Maintenance11238

CA-1057 Oakley x.25 failure 11257CA-147 Brentwood x.25 failure 11256

SC CA-131 Pinecate Secondary ISC has been removed UFN 11231CA-1051 Pacific ExchangeStandby ISC impaired 11254

SAC CA-416 County Rd 17 and I 505

Standby ISC OOS 11109

NV-682 Mountain Pass Standby ISC OOS 11252“The technician(s) associated with this/these site(s) are in the process of being notified.”ENVIRONMENTAL ISSUES:

MSO SITE NAME SITE/ISSUE/STATUSOAK CA-823 S Pleasanton Bouncing Low Temp alarm Rem. #11253

“The technician(s) associated with this/these site(s) are in the process of being notified.”MSO UPDATES:

MSO PROJECT NAME STATUS TICKET #OAK OMC 9.15 upgrade The 9.15 upgrade of all OMC’s was completed

lastnight, with no issues encountered. Health checks for packet data, the MDG’s and all DAP’s was verified as functioning properly. There was no problems encountered during this procedure.


91


G.PNW Morning ReportHello this is Your Name with the PNW Daily Report for Friday, October 27, 2000BR ISSUES:

MSO SITE NAME BR #

STATUS TICKET #

1 SEA WA143 Madison Valley 1-3 PA Power Rollback Alarm NTM 110981

“The technician associated with this site is in the process of being notified.”SITE ISSUES:

MSO SITE NAME SITE ISSUE/STATUS TICKET #

There are no site issues to report.

ENVIRONMENTAL ISSUES:MSO SITE NAME SITE/ISSUE/STATUS

TICKET #SEA WA018_MAYTOWN High Temp Alarm Tech has been notified NTM 9581

MSO UPDATES: MSO PROJECT NAME STATUS TICKET #

1 SEA Cisco Router IOS Upgrade to Version 12.0 (13)

The MSO is currently in the process of completing the Cisco Routers IOS Upgrade to Version 12.0. This activity is non-service affecting.

This concludes the PNW DailyPNW Daily Report. Thank you very much and have a great day.


92


H.Denver Morning ReportHello this is Your Name with the Denver Morning Report for Monday, August 28, 2000.BR ISSUES:

MSO SITE NAME BR #

STATUS

DNV CO132 Colfax & Buckley 56 BR-56 OOS, I tried a reset and the BR failed to respond. Tech has been notified.

DNV CO318 E Pueblo Hights 35 BR-35 OOS, I tried a reset and the BR failed to respond. Tech has been notified.

“The technicians associated with this site, has been notified.”SITE ISSUES:

MSO SITE NAME SITE ISSUE/STATUS TICKET #DNV CO150 Double Header Site is down, site tech is at the site waiting on the

Quest tech to Isolate trouble. NTM 8539Qwest CD014865

DNV CO024 Monument Site is currently down. Qwest is going tech to meet our site tech at the between 09:00AM and 10:00 AM for further troubleshooting.

NTM 8542ICG #121673

ENVIRONMENTAL ISSUES:.MSO SITE NAME SITE/ISSUE/STATUS

“No environmental issues to report”

MSO UPDATES: MSO PROJECT NAME STATUS TICKET #

"No projects to report

This concludes the DENVERDENVER Midday Report. Thank you very much and have a great day.


93

nextel emc manual

Documents