tems™ monitor master v10.4 - mantica-qm.ant...

Standard Reports

© InfoVista Sweden AB, 2000-2017. All rights reserved. of 74 21 July 2017

TEMS™ Monitor Master v10.4.4

Standard Reports

Document: MM-0070-Q

Created by: EWJP

Status: Released

Standard Reports


Copyright © InfoVista Sweden AB, 2000-2017. All rights reserved. Unauthorized copying, distribution and use of the

document or its content is strictly prohibited. All registered trademarks, quotes, products or service names are

acknowledged.

Note: In this guide the term "Monitor Master" refers to the InfoVista product "TEMS Monitor Master".

InfoVista Sweden AB

Laboratorgränd 3, 931 62

Skelleftea, Sweden

TEMS Monitor Master is part of the TEMS Portfolio, from InfoVista.

www.infovista.com

Version Overview

Version Date Author Modification

1.0 21/07/2017 EWJP Initial release

Information in this document is subject to change without notice. For the latest version please visit the TEMS

Customer Portal, https://customerportal.tems.infovista.com

Document Management

File name: standard_reports.pdf

Pages: 74

Released on: 21/07/2017

Approved by: GJ

http://www.infovista.com/

https://customerportal.tems.infovista.com/

Standard Reports


Table of Contents CHAPTER 1 - INTRODUCTION .................................................................................................................................... 5

THE STANDARD SCRIPTS ................................................................................................................................................... 5

ETSI METRICS .................................................................................................................................................................. 5

USING THIS GUIDE ............................................................................................................................................................ 6

RELATED GUIDES .............................................................................................................................................................. 6

CHAPTER 2 - INSTALLING THE STANDARD REPORTS ....................................................................................... 8

REQUIREMENTS ................................................................................................................................................................. 8

STEP 1: INSTALL THE DASHBOARDS .................................................................................................................................. 8

STEP 2: CONFIGURE ACCESS .............................................................................................................................................. 8

STEP 3: CHECK OPERATION ............................................................................................................................................. 11

CHAPTER 3 - USING DASHBOARDS ......................................................................................................................... 12

USING REPORTS: GETTING STARTED .............................................................................................................................. 12

USING REPORTS: THE RESULT SUMMARY DASHBOARD .................................................................................................. 14

USING REPORTS: TABLES ................................................................................................................................................ 15

USING REPORTS: HOVER ................................................................................................................................................. 15

USING REPORTS: THE SIDEBAR ....................................................................................................................................... 16

USING REPORTS: PARAMETERS ....................................................................................................................................... 16

USING REPORTS: THE VIEWLET MENU ............................................................................................................................ 18

USING REPORTS: VIEWLET CONTROLS ............................................................................................................................ 19

USING REPORTS: CLICK-THROUGH ................................................................................................................................. 19

USING REPORTS: PRINTING ............................................................................................................................................. 20

USING REPORTS: FILING A REPORT ................................................................................................................................. 21

USING REPORTS: RE-USING A REPORT............................................................................................................................. 21

USING REPORTS: SCHEDULING A REPORT ....................................................................................................................... 21

USING REPORTS: RADIO TRACE ...................................................................................................................................... 23

USING REPORTS: IP TRACE ............................................................................................................................................. 24

USING REPORTS: PROBLEMS? ......................................................................................................................................... 24

CHAPTER 4 - USING THE STANDARD REPORTS ................................................................................................. 25

THE RESULT SUMMARY DASHBOARD ............................................................................................................................. 26

THE FTP DOWNLOAD DASHBOARD ................................................................................................................................. 27

THE FTP UPLOAD DASHBOARD....................................................................................................................................... 31

THE HTTP DASHBOARD .................................................................................................................................................. 35

THE MMS DASHBOARD .................................................................................................................................................. 39

THE SMS DASHBOARD .................................................................................................................................................... 42

THE VOICE DASHBOARD ................................................................................................................................................. 46

THE WAP DASHBOARD ................................................................................................................................................... 51

THE KPI SUMMARY DASHBOARD .................................................................................................................................... 55

THE ETSI KPI OVERVIEW DASHBOARD .......................................................................................................................... 58

THE GRAPHICAL OVERVIEW DASHBOARD ....................................................................................................................... 60

Standard Reports


THE ALL SERVICE GRAPHS DASHBOARD ......................................................................................................................... 63

THE CUMULATIVE DURATION DASHBOARD .................................................................................................................... 64

THE FAILURE OVERVIEW DASHBOARD ............................................................................................................................ 66

THE COMPLETION FAILURE LIST DASHBOARD ................................................................................................................ 67

THE SERVICE FAILURE LIST DASHBOARD ........................................................................................................................ 69

THE SYSTEM FAILURE LIST DASHBOARD ........................................................................................................................ 70

INDEX ............................................................................................................................................................................... 72


Chapter 1 - Introduction

This is the installation and user guide for the Monitor Master Standard Reports - the reports you can use to output

results from the Standard Scripts. The installation part is intended for your system administrator, but the rest of

the guide is for the people who need to run the reports and interpret the results they contain.

This guide should be all you need to start using the Standard Reports, although you may wish to refer to the

Related Guides.

WARNING: If possible, the Standard Reports should be installed and used as supplied, without any changes.

While customers may need to adapt the Standard Reports for their own purposes, InfoVista Sweden AB will only

support the original, unedited reports.

The Standard Scripts The Monitor Master Standard Scripts are example scripts for testing and monitoring key network and service

metrics. They can be used to test Voice, Data and Messaging services.

The Standard Scripts are designed to be used as supplied; customers adapt them for their requirements by setting

runtime parameters to the appropriate values. Customers should not expect the Standard Scripts to work correctly

in all circumstances - they may need to be rewritten for a the customer's specific requirements or environment.

The Standard Scripts can be run on demand or scheduled to run at specific times and dates.

The Standard Reports are provided to present key results from these scripts. Standard Reports can also be run on

demand or scheduled to run on a daily basis.

If your Monitor Master system has the Radio Trace or IP Trace options installed, the Standard Scripts can use them

to provide detailed insight into communications at Radio and Ethernet packet level. These options are required for

the Standard Scripts to produce the full range of ETSI-compliant KPIs.

ETSI Metrics The Standard Scripts produce a rich set of metrics out-of-the-box, including metrics aligned with the ETSI

standards for QoS testing. See the SPD for each package for details of the Standard Metrics it produces.

ETSI Compliant Metrics

The following Standard Scripts include an option to generate ETSI-compliant metrics (with limitations) by using

the increased accuracy afforded by the Radio Trace and IP Trace options:

QM GPRS Attach

QM GPRS Context Activate

QM Ping

QM DNS

QM Web (HTTP)

QM WSP (WAP)

QM FTP Upload

QM FTP Download

QM Voice Availability

QM Voice Quality

QM SMS

QM MMS

Chapter 1 - Introduction


ETSI Limitations

Please note the following limitations on the production of ETSI-compliant metrics by the Standard Scripts:

Radio Trace is needed to report accurate PDP Context Activation KPIs.

The ETSI specifications apply to GSM, LTE and WCDMA networks; they do not apply to CDMA.

The probe must be capable of running IP Trace in order to generate KPIs for data tests.

IP TraceP does not produce accurate absolute timings, so any KPIs that are based on an absolute IP Trace

time and either a Radio Trace or Monitor Master application timing may be inaccurate, although

synchronization is performed in the code to minimize this effect. The KPIs affected are: HTTP Setup Time

(ETSI), FTP Download Setup Time (ETSI) and FTP Upload Setup Time (ETSI).

The Device Profile used for each test is set to the default Sony Ericsson handset profile. This may not be an

accurate representation of the most common customer handset.

Some internal application timeouts are less than the timeouts specified by the test scripts.

Bearer selection is automatic and will revert to the handset default which is normally HSDPA/3G preferred,

then 2G.

KPIs are reported as failures instead of successes. While this is as per the specification, some customers

might prefer them as success measurements.

Radio Network Unavailability is not reported by any of the Standard Scripts.

Using this Guide The Standard Reports Guide consists of four chapters:

1. Introduction - a brief introduction to the Standard Reports (and Standard Scripts) for everyone

2. Installing the Standard Reports - how to install and configure the Standard Reports, for your system

administrator

3. Using Dashboards - how to run Box Office Dashboards; a convenient introduction for new users

4. Using the Standard Reports - details of each Standard Report, including its parameters

Related Guides Standard Scripts:

Standard Scripts Guide - installing and using the Standard Scripts for testing Voice, Data, and Messaging

services

Standard Scripts for Voice SPD - summary specification of the Voice Standard Scripts, including a full list of

the metrics they report

Standard Scripts for Messaging SPD - summary specification of the Messaging Standard Scripts, including a

full list of the metrics they report

Standard Scripts for Messaging over LTE SPD - summary specification of the Messaging Standard Scripts when

used with LTE, including a full list of the metrics they report

Standard Scripts for Data SPD - summary specification of the Data Standard Scripts, including a full list of the

metrics they report

Standard Scripts for Data over LTE SPD - summary specification of the Data Standard Scripts when used with

LTE, including a full list of the metrics they report

General:

Standard Reports


Installing & Managing Monitor Master - general guidance on installing, setting up, and maintaining your

Monitor Master system

Quality Manager User Guide - scheduling, monitoring and controlling testruns

Using Box Office Reports - how to launch reports and use their controls and options

Developing Reports with Box Office - includes details of Box Office Favorites and Jobs, which can be used to

help make work easier


Chapter 2 - Installing the Standard Reports

Read this chapter if you are the Monitor Master system administrator.

This chapter describes how to install and set up the Standard Reports on your Monitor Master system. You should

only need to do this once. It consists of three simple steps:

Step 1 - Install the dashboards

Step 2 - Configure access

Step 3 - Check operation

The installation process requires administrator-level access to the QM Server and Box Office Server, and should

take less than an hour.

Requirements The Standard Reports must be licensed and installed on a fully-working Monitor Master QM system, complete

with Box Office, Projector and one or more Agents.

The Standard Reports require Box Office version 1.5SP1 or later.

Step 1: Install the Dashboards The dashboards and other components used by the Standard Reports are installed with Box Office v1.5SP1 (or

later). If you have an earlier version of Box Office, you must upgrade to this version (or later) to use these reports.

Step 2: Configure access

Open Access

If Box Office is configured for Open Access, all users will be able to run the Standard Reports - you do not need to

do anything to give them access and can skip this Step.

Controlled Access

If Box Office is configured for Controlled Access, initially no-one will be able to run the Standard Reports. To allow

Users to access the reports, you must add the report dashboards to their Roles, as follows.

Hint: You may need to create suitable Roles first, by using the Configure / Advanced / Roles menu option, as

described in the Developing Reports with Box Office guide.

1. Login to Quality Manager as administrator.

2. Select the Home option from the Reporting menu, to launch Box Office.

3. Select the Configure / Advanced / Roles menu option:

Standard Reports


4. Click the Role to provide with access.

5. Click Dashboards and then click the Add button near the bottom of the window (not the one near the top),

to add a default dashboard:

6. Click the drop-down list on the right to list the dashboards on your system:

Chapter 2 - Installing the Standard Reports


7. Scroll down to the dashboard to add and click it. The Standard Script dashboard names all start with [STD]

8. Check you've chosen the right dashboard:

9. Click Apply (not OK) to add it to this Role:

Standard Reports


10. If you want to add any more dashboards to this Role, repeat the process from adding a blank dashboard

(action 5 above).

11. When you've finished adding dashboards to this Role, click OK to save the changes and quit.

FTP access

The Standard Scripts use FTP to upload IP Trace and Radio Trace files to the Projector, using the default FTP setup

(documented in the Installing & Managing Monitor Master guide - see Adding an FTP Server).

If FTP setup is non-default, you need to create a SiteConfig.xml file on the QM Server to specify the setup. This

process is documented in the Installing & Managing Monitor Master guide - see Adding an FTP Server.

Step 3: Check operation Before announcing availability of Standard Reports to your Users:

Log in to QM as one of the Users who should have access

Select the Reporting / Home menu option to launch Box Office

Can you can see the dashboards in Box Office?

Can you run the Result Summary dashboard?

If you encounter any problems, resolve them before "going live".


Chapter 3 - Using Dashboards

Read this chapter if you need to run the Standard Reports AND have never used Box Office before.

This chapter describes how to launch, control and tune the reports that present results from the Standard Scripts. If

you are already familiar with using Box Office reports, you probably don't need to read this chapter and can go

straight to Chapter 4, The Standard Reports.

For more detailed information on the controls provided by Box Office (the reporting GUI), see the Using Box Office

Reports user guide.

Overview

When you run a script, it reports messages, timings and other detailed metrics to the results database on the Agent

that ran it. The Agents continually distil this information to send summary results to the Monitor Master data

warehouse, Projector.

You can access a simple report on each Task from the Task Control page, (see your Standard Scripts Guide), but this

is just provided as a quick check on successful completion of the Task.

The main Monitor Master reporting technology is called Box Office, which provides a browser-based GUI for

selecting and configuring the reports you want and viewing the reports you've output. A report is also known as a

Dashboard. A dashboard can output information in many different objects called Viewlets. Viewlets range from

simple lists and gauges, to line graphs and Gantt charts to complex pivot tables.

Dashboards and Viewlets can also provide a range of user controls, including filtering, expand/collapse, zoom

level, and click-through to related reports.

Box Office includes a package of simple dashboards specifically designed to report results from the Standard

Scripts. These are the Standard Reports:

Result Summary

Graphical Overview

All Service Graphs

Cumulative Duration

Failure Overview

Completion Failure List

Service Failure List

System Failure List

These are described in detail in Chapter 4.

Box Office also includes many other, more general-purpose, reports. These are described in the Using Box Office

Reports guide.

Using Reports: Getting Started Launch QM and log in.

Select the Projector option from the Reporting menu:

Standard Reports


Wait for the Screening Room dashboard to be displayed

This dashboard lists recent testruns on your Monitor Master system, but we want to focus on the Standard

Scripts.

Click the button next to Dashboard in the top left corner of the window, to list all the dashboards that

work with Projector as the data source:

Click the button next to the Standard Scripts folder to open it:

Click Result summary to launch the top-level dashboard we want:



The Result Summary dashboard looks like this:

This dashboard provides a summary of the key results for each service, averaged over the selected period

(the last 24 hours, initially) as well as a range of controls and drill-down to other dashboards for more

detailed analyses

We'll now describe this dashboard, the information it provides and the controls it offers you. We'll describe these

controls in considerable detail, to help you get familiar with Box Office.

We'll describe each of the other dashboards available, in somewhat less detail, in Chapter 4, Using the Standard

Reports.

Using Reports: The Result Summary dashboard The Result Summary dashboard is the top-level dashboard in the suite of reports for the Standard Scripts. It

displays a summary of results for each service, averaged over the last 24 hours:

Standard Reports


This gives an instant indication of the health of each service and provides controls to refine the information and

links to more detailed reports.

We'll use this dashboard as your main introduction to using Box Office dashboards in general - and Standard

Reports in particular.

Using Reports: Tables The Result Summary dashboard displays a table (or grid) with four columns:

ServiceType is the name of the service (DNS, Email, etc) that this row applies to.

AdditionalDetail is a description of the aspect of the service being measured, where there is more than one

Standard Script for the service.

Average service availability % is the average service availability for this service, for the currently selected

period, expressed as a percentage. A single testrun will report that a service is available (100%) or not

available (0%); this metric averages those results over time to give a single figure that indicates the overall

health of the service. For example, if 1 test in 10 reports the service as unavailable, this figure would be 90%.

Each value is color-coded, ranging from green (for 100%) through yellow and orange down to red (0%). Any

color other than green indicates a potential problem.

Average end to end duration is the average time taken for the service test to complete, expressed in

seconds. This is a good indicator of how responsive the service is; how long users are having to wait for

their emails, downloads, etc.

Dashboards can display results using a wide range of devices: tables, graphs, lists, maps, gauges, Gantt charts, tree

diagrams, and more. These devices are called Viewlets.

Using Reports: Hover To count the number of testruns that a figure in the table represents, hover over it:

Move the mouse pointer to the figure

Wait a few seconds for a tooltip to be displayed, like this:



This example shows that 1.6s is the average result from 66 testruns.

(You'll see shortly that you can get much more information about a value by clicking it. Have a look at Click-

through if you can't wait!)

Using Reports: the Sidebar

On the left of each Dashboard is the Sidebar, which provides controls to adjust the report

to your requirements.

The Sidebar is divided into three main sections:

Dashboard - lists all the other dashboards you can use with the current data source

Source - lists all the data sources (databases) you can report on

Parameters - lists all the filtering controls you can use with the current dashboard

You can ignore the first two (Dashboard and Source) for now; let's look at Parameters.

(You can collapse the Dashboard and Source sections of the Sidebar by clicking their

buttons.)

Using Reports: Parameters

Parameters are controls in the Sidebar that you can use to adjust the filters applied to

the data source to extract the data you want to report on - and filter out the data you

aren't interested in. You can turn filters on and off, and change the filtered value by

using these controls.

The Summary Report dashboard has a fairly typical set of Parameters:

Time ago - select data for the last 5 minutes, hour, day, week, etc

Start time - select data from after a specific time and date

End time - select data from before a specific time and date

Operator - choose data from a specific operator or operators. To select

multiple operators use the Ctrl and Shift keys while clicking

Location - choose data from a specific (probe) location

Note: If you use Time ago, you must turn off Start time and End time; and if you

use Start time and End time, you must turn off Time ago. To turn a filter on or off

Standard Reports


you use its checkbox - as described in the next section.

Controlling a filter

Each parameter has a checkbox next to it. To turn on a filter you must tick its checkbox; to turn off a filter, you

must un-tick its checkbox:

This parameter is ON:

This parameter is OFF:

Time ago Time aggregation, and Service type are ticked, so these are the only filters being used - initially.

(If a checkbox is gray, you can't turn it on or off; it's FIXED the way it's displayed. You can however change its

value i.e. the value used by the filter.)

Adjusting a filter

You adjust a filter by adjusting the displayed value. There are different types of control, for different types of

value. The Summary Report dashboard uses these:

This is a single-row DROPDOWN LIST:

Time ago, Time aggregation and Service type all use a dropdown list.

To select an option from a single-row dropdown list, click the to list the options and choose the one you want

by clicking it.

This is a DATE-TIME:

Start time and End time both use a date-time control, for you to choose a date and a time.

To set the date:

Click the button to display the calendar

Select the right year from the dropdown

list on the right

Select the right month from the dropdown

list on the left

Click the day



To set the time, you can click the buttons next to the displayed hour and minutes and select the appropriate

hour and minute.

This is a multi-row LOOKUP TABLE:

To select an option from a multi-row lookup table, click the option. You may need to scroll it first.

The Operator parameter also allows you to select multiple values: simply click the first value, then hold down Ctrl

and click to add more.

(Dropdown lists and Lookup tables are used the same way - the real difference is that a Dropdown list use static

data, while a Lookup table use dynamic data read from the database or environment. Each is available in single-

row, multi-row and multi-value variants, which are used a little differently.)

Applying your changes

After making any changes to the parameters, you need to re-run the dashboard to get a new report. To do so, click

the Apply button, near the bottom of the Sidebar.

Examples

Initially, a Summary Reports dashboard displays results for the previous day, for all operators and all locations.

To display results for the previous week:

Change Time ago value from 1 day to 7 days

To display results since March 21, 2009:

Disable the Time ago parameter (un-tick its checkbox)

Enable the Start time parameter (tick its checkbox)

Set its date to March 21, 2009, and time to 00:00

To display results from March 21, 2012:

Disable the Time ago parameter

Enable the Start time parameter and set its date to March 21, 2012 and time to 00:00

Enable the End time parameters and set its date to March 22, 2012, and time to 00:01

To display results for Operators X and Y:

Enable the Operator parameter (tick checkbox)

Click the X operator in its list

Hold down Ctrl and click the Y operator

Using Reports: the Viewlet menu Notice the button at the end of Viewlet title bar. Hover over this to display the Viewlet menu:

Standard Reports


The Export sub-menu provides three options:

Select New web page to display the Viewlet as a separate web browser page.

Select Excel web query to create a database query that will extract the same data as the dashboard.

You can use this to export data to Excel as described in the Using Box Office Reports user guide.

Select As PDF to create a PDF document from the Viewlet. This an excellent way to keep a copy of

the displayed results for later use - for example, to email to another person.

The Edit option allows you to edit the definition of this Viewlet. This is for Box Office developers only. See

the Developing Reports with Box Office user guide.

The Query sub-menu provides two options:

These advanced options are described in the Developing Reports with Box Office user guide. You shouldn't need them

when using the Standard Reports.

Using Reports: Viewlet controls Notice the group of small icons at the start of the Viewlet title bar:

Click to unfold (expand) a folded Viewlet.

Click to fold (collapse) an unfolded Viewlet.

Click to zoom in - to double the Viewlet size.

Click to zoom out - to halve the Viewlet size.

Using Reports: Click-through To get more details about a result, you can use click-through:

Click the result (value) you want to examine: a cell of a table or pivot; a data point on a graph, a needle on a

gauge or a flag on a map..

Wait for a new dashboard to be displayed, in another Browser window. In general this will display a

dashboard that shows how the selected value was derived.

For example: click the Service Availability or Overall Duration value for the GPRS Context Activation on the

Result Summary to display the Graphical Overview dashboard for this service, like this:



This graph shows how the selected metric (GPRS Context Activate end-to-end overall duration) varied over the

selected period. It has separate curves for the average, minimum and maximum values for each hour of the period,

as well as bars showing the average service availability for each hour.

(The tables beneath the graph lists any transactions and testruns that reported a service failure. It starts off

collapsed; click its button to expand a collapsed table.)

If you click the Service Availability or Overall Duration value for a different service, you'll see the same dashboard,

but for that service.

You can use a dashboard displayed by click-through just like any other dashboard - it can have Parameters,

Viewlet Controls, Viewlet menus, etc. It can even provide click-through to more detailed reports!

Dashboards don't have to provide click-through, but most do.

Hint: If a value provides click-through, the mouse pointer changes to when you hover over it.

Report hierarchy

Most of the Standard Script dashboards are accessible direct from the Dashboards menu:

The click-through dashboards are only used to provide details of a value you've clicked.

Using Reports: Printing To print the entire report (as shown on the screen):

Select the appropriate print function from your browser's toolbar or menu

To print the output from a single Viewlet:

Hover over the button in the Viewlet title bar and select the Export New web page option from the

popup menu

Standard Reports


Once the Viewlet has opened in another browser window, use the appropriate IE or PDF Viewer print

function

You can also use the Export As PDF function, described next, to save a "printed" report.

Using Reports: Filing a report To store a report exactly as displayed on the screen (complete with the current data), so you can view it later or

send it to colleagues, save it as a web page:

Select the appropriate Save As option from the Internet Explorer Page (IE7) or File (IE6) menu. We

recommend saving as an MHT file.

Or, to save a single Viewlet exactly as displayed on the screen (complete with the current data), save it as a PDF:

Hover over the button in the Viewlet title bar and select the Export As PDF option from the popup menu

Once the PDF has opened in another browser window, use the button to save it as a file

These two methods produce a static report - in effect, you've "printed" the report to a file. The report will not

change to use new data when it's available; it's a permanent record, a snapshot.

Using Reports: Re-using a report To save a report so you can run it again (with the same parameter settings but new data), save it as a Favorite:

Select the Dashboards / Favorites / Add this option from the Box Office menu to save it as a Box Office

Favorite, which will allow all Box Office users to use it (via the Dashboards menu).

Note: This requires Developer access rights and is described further in the Using Box Office Reports user

guide.

Or select the Internet Explorer Add to Favorites option to save it as an IE Favorite, which will allow you to

use it (but not other Box Office users)

These two methods produce a dynamic report that you can run again easily, using new data. You're not really

saving a report, you're creating a shortcut to run the report again.

Using Reports: Scheduling a report While the most obvious way to use the Standard Script reports is by running the Dashboards interactively when

you need them (ad hoc), you can also arrange to run a report automatically at the same time every day, as a Box

Office Job. This can save you a lot of time and effort, as it runs automatically, sending the results by email to the

people you've chosen.

Here's how:

Hint: For a more detailed description of scheduling reports, see the Jobs chapter in the Developing Reports with Box

Office guide - or search the Help for "adding a job".

Login to Box Office as a developer

Select the Configure / Advanced / Jobs option to list your current Jobs:



Click the Add button. This will display a default Job definition for you to complete:

Type a more suitable job name in the first field and click Rename to save the change.

Type the Title to use in the Job List, such as Daily Service Failures List.

Ignore the Notes field.

Tick the Enabled checkbox.

Ignore the Regime field. (You can use this field to schedule Jobs in groups - refer to the Developing Reports

with Box Office guide for details).

Select the dashboard to run from Dashboard

Select the source to use from Source - your Projector

The next field, Extra parameters, is quite tricky. Here you must type the parameters the dashboard needs to

produce the report you want, in the format &parameter=value&parameter=value... The simplest way to do this

is to run the dashboard first with the correct parameters selected, click Apply, and copy them from the end

of the URL displayed on the Address bar. See Example parameters, below.

Note: When configuring the parameters, you must use the Time ago parameter to define the time period.

Do not use the Start time and End time parameters, or you'll get the same report every time you run the

report!

Standard Reports


If a dashboard times out, increase the timeout by adding the parameter &sqlcommandtimeout=nnn where

nnn is the timeout in seconds

Leave Alternative URL blank.

In the Email addresses field type the email addresses, separated by commas, to send the output to

In Mail format select the format to send the output in. PDF file attachment is suitable for most applications:

Ignore the This machine field

Click OK to save the changes and return to the Job List

Once you have created a Job it should run the next night, provided Jobs have been scheduled. See Scheduling Jobs.

Example Parameters

For example, to produce a list of all SMS service failures from the previous day:

Choose the Service failure list dashboard (from Standard Scripts / Admin).

Choose the Projector database source.

Enable the Time ago parameter and choose 1 day.

Choose SMS from the Service type parameter.

Make sure the other parameters are disabled.

Click Apply and you will see text like this on the end of the URL in the address bar:

?dashboard=Argo.StandardScripts.Overview.ServiceFailures &source=Argo.MMProjector.QM

&csvt=SMS+ &ta=86400

I've added some spaces to make it easier to read, so you can see that:

- Time ago is passed as &ta=nnn where nnn is the time ago in seconds

- Service type is passed as &csvt=name

Copy the &csvt=SMS+&ta=86400 part from the Address bar to the Extra parameters field.

Scheduling Jobs

Box Office Jobs are scheduled by using the Windows Scheduler on the Box Office server. During the installation

process a single task should have been scheduled to run every day (at 02:00am or 04:00am). If it was, this will run

all the enabled Jobs by default at the selected time. Easy!

If you're not sure whether Box Office scheduling has been set up, ask your Monitor Master administrator.

If Box Office Job scheduling has not been set up, or you want to change the times, or run different Jobs at different

times, see the Jobs chapter in the Developing Reports with Box Office guide.

Using Reports: Radio Trace Note: Radio Trace is an advanced optional technology, intended to help you diagnose low-level problems. It is not

intended for casual users.

Here are some notes to help you:

The appropriate version of Radio Trace for the modem (Type 1 or Type 2) must be installed, licensed and

working on the Agent

A system FTP server must be installed on Projector to store the trace files

The test script must use a modem that supports Radio Trace, such as the SW8785V or SW5727V



Include the trace keyword in the Standard Script's Option parameter to turn on Radio Trace, capture trace

data and report additional metrics in the results

If you want the trace stored for later analysis, include the upload keyword in the Option parameter

You should only use Radio Trace when you need it, to investigate failures. Don't turn it on as a matter of

routine or as part of your regular (scheduled) testing and monitoring

Key markers are reported as metrics (see the script descriptions in Chapter 4 for details) for you to use in

your own reports; these metrics are not reported by the Standard Reports

Uploaded files are stored in the FTP root directory on the Projector; the first 5 characters of the name is

based on the Agent name, to help you choose the correct file

You can analyze the uploaded trace using tools such as TEMS Discovery

TEMS Discovery requires filenames in a specific format - do not rename these files!

Using Reports: IP Trace Note: IP Trace is an advanced technology, intended to help you diagnose problems with TCP/IP connections. It is

not intended for casual users.

Here are some notes to help you:

The appropriate version of IP Trace for the modem (Type 1 or Type 2) must be installed, licensed and

working on the Agent

A system FTP server must be installed on Projector to store the trace files

IP Trace network capture only works for TCP/IP connections.

Include the pcap keyword in the Standard Script's Option parameter to turn on IP Trace

If you want the trace stored for later analysis, include the upload keyword in the Option parameter

You should only use IP Trace when you need it, to investigate failures. Don't turn it on as a matter of routine

or as part of your regular (scheduled) testing and monitoring

Uploaded files are stored in the FTP root directory on the Projector; the first part of the name is the Agent

name and testrun id, to help you choose the correct file; the file extension is .pcap

You can analyze the uploaded network capture file with a tool such as Ethereal.

Using Reports: Problems? If you encounter problems using the Standard Reports:

Check the instructions for using dashboards, in Chapter 3

Check the description of the particular Standard Report, in Chapter 4

Ask your Monitor Master Administrator (or other experienced Box Office user) for help

Visit the TEMS Customer Portal, https://customerportal.tems.infovista.com

https://customerportal.tems.infovista.com/


Chapter 4 - Using the Standard Reports

Read this chapter if you need to use the Standard Reports to report results from the Standard Scripts, or need to interpret the

information provided by those reports.

Box Office offers a range of Dashboards specifically designed for reporting results from the Standard Scripts, which

we call the Standard Reports. These can be accessed via the Dashboards/Monitor Master Projector/Standard

Scripts menu option:

To see more about a Standard Report, click its name:

Result Summary

ETSI / Service / FTP Download

ETSI / Service / FTP Upload

ETSI / Service / HTTP

ETSI / Service / MMS

ETSI / Service / SMS

ETSI / Service / Voice

ETSI / Service / WAP

ETSI / KPI / KPI Summary

ETSI / ETSI KPI Overview

Admin / Failure Overview

Admin / Completion Failure List

Admin / Service Failure List

Admin / System Failure List

Graphical Overview

All Service Graphs

Cumulative Duration



The previous chapter also includes a detailed description of the Result Summary Dashboard, as an introduction to

launching and using Box Office reports.

You can launch these Dashboards directly, by selecting options from the Dashboard menu or Sidebar. You can

launch some other Dashboards by click-through from one of these Dashboards, to provide more details of a clicked

value. The click-through Dashboards are also described below.

The Result Summary dashboard The Result Summary dashboard is the top-level dashboard in the suite of Standard Reports. It displays a summary

of results for each service, initially averaged over the last 24 hours.

This gives an instant indication of the health of each service and provides controls to refine the information and

links to more detailed reports.

Viewlets

This dashboard displays a single Complex Pivot Viewlet, summarizing a lot of results in four columns:




Average service availability % is the average service availability for this service, for the currently selected period,

expressed as a percentage. A single testrun will report that a service is available (100%) or not available (0%); this

metric averages those results over time to give a single figure that indicates the overall health of the service. For

example, if 1 test in 10 reports the service as unavailable, this figure would be 90%.

Each value is color-coded, ranging from green (for 100%) through yellow and orange down to red (0%). Any color

other than green indicates a potential problem with the service.

Average end to end duration is the average time taken for the service test to complete, expressed in seconds. This

is a good indicator of how responsive the service is; how long users are having to wait for their emails, downloads,

etc.

Standard Reports


Hover over a figure to display a count of the number of testruns that it represents:

This example shows that 1.6s is the average result from 66 testruns.

Parameters

The Result Summary dashboard has a fairly typical set of Parameters:




Operator - choose data from a specific operator or operators. To select multiple operators use the Ctrl and

Shift keys while clicking


Note: If you use Time ago, you must turn off Start time and End time; and if you use Start time and End time, you

must turn off Time ago. (Turn a parameter on or off by clicking its checkbox.)

Click-through

Click the Service Availability or Overall Duration value in any row to display the Graphical Overview dashboard

for that service, like this:

This dashboard includes a graph that shows how results for the selected service (e.g. GPRS Context Activate)

varied over the selected period, as well as lists of any testruns that reported a service. It is described more fully

below.

The FTP Download dashboard The FTP Download dashboard can be launched by selecting Standard Scripts / ETSI / Service / FTP Download

from the navigation (Sidebar or Dashboards menu). It is not displayed by click-through.



This displays results from the FTP Download standard script, initially for the last 24 hours and provides controls to

refine the information and links to more detailed reports.

Viewlets

This dashboard uses four different Chart Viewlets.

The FTP Download Session Time (ETSI) Viewlet displays the time (in seconds) to complete each download,

averaged over each timeslot (initially, hour):

Standard Reports


The three curves show the average, minimum and maximum time within each timeslot. In the example results

above there are some rather slow downloads between 8:00 and 20:00 on January 25th.

The FTP Download Failure (ETSI) Viewlet displays the percentage of downloads that failed to complete within

each timeslot (initially, hour):

In the example results above, none of the downloads failed.

The FTP Download Mean Data Rate (ETSI) Viewlet displays the data download rate (in Mbit/second) for each

timeslot (initially, hour):



The three curves show the average, minimum and maximum data download rate within each timeslot. The results

above show minimum data rates below 200Mbit/second between 9:00 and 18:00 on both days.

The FTP Mean Data Rate (ETSI) Cumulative Distribution Viewlet displays the percentage of downloads against

the data rate:

The frequency distribution curve (in red) shows the percentage of downloads that achieve each data rate. In the

example results above you can see that most downloads achieve 400-600 kbit/second.

Standard Reports


The cumulative distribution curve (in blue) shows the percentage of downloads with a data rate less than or equal

to the rate on the X-axis. In the example results above, approximately 30% of the downloads achieved a data rate of

400 kbit/second or less.

Parameters

The FTP Download dashboard has these Parameters:




Time aggregation - select the timeslot length over which to average results



Click-through

Click a datapoint on any of the line graphs to display a list of the testruns contributing to that value, on the List of

Testruns dashboard.

Click any testrun in this dashboard to display details on the Testrun Details dashboard.

The FTP Upload dashboard The FTP Upload dashboard can be launched by selecting the Standard Scripts / ETSI / Service / FTP Upload

option from the navigation (Sidebar or Dashboards menu). It is not displayed by click-through.



This dashboard displays results from the FTP Upload standard script - initially for the last 24 hours - and provides

controls to refine the information and links to more detailed reports.

Viewlets

This dashboard displays results on four different Chart Viewlets.

The FTP Upload Session Time (ETSI) Viewlet displays the time (in seconds) to complete each upload, for each


Standard Reports


The three curves show the average, minimum and maximum time within each timeslot. In the example results

above there are some rather slow uploads around 17:00.

The FTP Upload Failure (ETSI) Viewlet displays the percentage of uploads that failed to complete within each


In the example results above, none of the uploads failed (although there's no data for 11:00-14:00).

The FTP Upload Mean Data Rate (ETSI) Viewlet displays the data upload rate (in Mbit/second) for each timeslot

(initially, hour):



The three curves show the average, minimum and maximum data upload rate within each timeslot. The results

above show the minimum upload rate dropped below 250Mbit/second within several timeslots.

The FTP Upload Mean Data Rate (ETSI) Cumulative Distribution Viewlet displays the percentage of uploads

against the data rate:

Standard Reports


The cumulative distribution curve shows the percentage of uploads with a data rate less than or equal to the rate

on the X-axis. In the example results above, approximately 30% of the downloads achieved a data rate of 300

kbit/second or less.

Parameters

The FTP Upload dashboard has a fairly typical set of Parameters:







Click-through


Testruns dashboard.


The HTTP dashboard The HTTP Download dashboard can be launched by selecting Standard Scripts / ETSI / Service / HTTP

Download from the navigation (Sidebar or Dashboards menu). It is not displayed by click-through.



This displays results from the HTTP Download standard script, initially for the last 24 hours and provides controls

to refine the information and links to more detailed reports.

Viewlets

This dashboard displays results on three Viewlets.

This dashboard uses three different Graph Viewlets to display results.

The HTTP Session Time (ETSI) Viewlet displays the time from beginning to end of the HTTP session, within each

timeslot:

Standard Reports


The three curves show the average, minimum and maximum session time within each timeslot.

The example results shown above show that the session time is fairly consistent during the test period, with peaks

around 12:00-13:00 and 18:00-19:00.

The HTTP Session Failure Ratio Viewlet displays the percentage of sessions that failed, within each timeslot:



The example results shown above show a 0% failure ratio throughout most the day (no sessions were lost, once

started), with 1-in-4 failing in the 02:00 and 08:00 timeslots..

The HTTP Session Time Cumulative Distribution Viewlet displays the percentage of sessions completed with

each session time:

The frequency distribution curve (in red) shows the percentage of sessions that complete in a specific time. In the

example results above, most sessions take 21-27 seconds.

The cumulative frequency curve (in blue) shows the percentage of sessions that are complete within each time. In

the example results shown above, a few sessions lasted just 22 seconds and 90% were completed within 29.5

seconds.

Parameters

The HTTP dashboard has a fairly typical set of Parameters:







Click-through


Testruns dashboard.

Standard Reports



The MMS dashboard The MMS dashboard can be launched by selecting the Standard Scripts / ETSI / Service / MMS option from the

navigation (Sidebar or Dashboards menu). It is not displayed by click-through.

This displays results from the MMS standard script, initially for the last 24 hours, and provides controls to refine

the information and links to more detailed reports.



Viewlets


The MMS End-to-End Delivery Time Viewlet displays the time taken to deliver an MMS from one probe to

another, within the selected time period:

The three curves show the average, minimum and maximum delivery time within each timeslot. The example

results shown above show that delivery time is fairly consistent during the test period, with peaks around 13:00

and 17:00.

The MMS End-to-End Delivery Time (raw data) Viewlet is collapsed initially. Click its button to see the data that

the graph is based on:

Standard Reports


The MMS End-to-End Failure Ratio Viewlet displays the percentage of messages that were not received, within

each timeslot:

The example results shown above show non-zero failure ratios between 00:00 and 05:00 and around 23:00.

The MMS End-to-End Delivery Time Cumulative Distribution Viewlet displays the percentage of messages

delivered against delivery time:



The cumulative frequency curve shows the percentage of messages delivered within a specified time. In the

example results shown above, approximately 75% of messages were delivered within 40 seconds and 95% within

60 seconds.

Parameters

The MMS dashboard has a fairly typical set of Parameters:






Click-through

Click a datapoint on any of the line graphs or a row in the raw data table to display a list of the testruns

contributing to that value, on the List of Testruns dashboard.


The SMS dashboard The SMS dashboard can be launched by selecting Standard Scripts / ETSI / Service / SMS option from the


Standard Reports


This displays results from the SMS standard script, initially for the last 24 hours, and provides controls to refine the

information and links to more detailed reports.

Viewlets


The SMS End-to-End Delivery Time Viewlet displays the time taken to deliver an SMS from one probe to another

during the selected period:



The three curves show the average, minimum and maximum delivery time within each timeslot. The example

results shown above show that delivery time is fairly consistent during the selected period, varying between 2.5

and 4 seconds.

The SMS Completion Failure Ratio Viewlet displays the percentage of messages that were not received, within

each timeslot:

Standard Reports


The example results shown above show a 0% failure ratio throughout the day i.e. no messages were lost, once sent;

an excellent result.

The SMS End-to-End Delivery Time Cumulative Distribution Viewlet displays the percentage of messages

delivered against delivery time:

The cumulative frequency curve shows the percentage of messages delivered within a specified time. In the

example results shown above, approximately 50% of messages were received within 3.3 seconds and 95% within

3.8 seconds.

Parameters

The SMS dashboard has a fairly typical set of Parameters:







Click-through


Testruns dashboard.




The Voice dashboard The Voice dashboard can be launched by selecting Standard Scripts / ETSI / Service / Voice from the navigation

(Sidebar or Dashboards menu). It is not displayed by click-through.

This displays results from the Voice Availability and Voice Quality standard scripts, initially for the last 24 hours,

and provides controls to refine the information and links to more detailed reports.

Standard Reports


Viewlets

This dashboard uses six different Graph Viewlets to display results.

The Telephony Setup Time Viewlet displays the time taken to set up calls, within each timeslot:

The three curves show the average, minimum and maximum setup time within each timeslot.

The example results shown above show that setup time increases steadily after mid-day.

The Telephony Service Non-Accessibility Viewlet displays the percentage non-accessibility (failure to place a call)

within each timeslot:



The example results shown above show 100% accessibility throughout the day, except in the 13:00 and 23:00

timeslots.

The Telephony Setup Time Cumulative Distribution Viewlet displays the percentage of calls against setup time

(in seconds):

Standard Reports


The example results shown above show that the shortest setup is around 6 seconds, with 50% of calls set up within

7.2 seconds, and 100% set up (or failed) within 10.2 seconds.

The Telephony Cut-off Call Ratio Viewlet displays the percentage of calls that were cut off within each timeslot:

The example results shown above show a 0% cut-off ratio throughout the day i.e. no calls were cut off, once set up;

an excellent result.

The Telephony Speech Quality on Call Basis Viewlet displays the call quality (PESQ score) for calls within each

timeslot:



The three curves show the average, minimum and maximum speech quality scores within each timeslot.

The example results shown above show a marked drop in call quality in the second half of the day.

The Telephony Speech Quality on Call Basis Cumulative Distribution Viewlet displays the percentage of calls

achieving each level of call quality score:

Standard Reports


The frequency distribution curve (in red) shows the percentage of calls that achieve each call quality score. In the

example results shown above, most calls achieve a score of 1.5-2.0 or 2.8-3.1.

The cumulative frequency curve (in blue) shows the percentage of calls that achieve or better each call quality

score. In the example results shown above, approximately 45% of calls achieved a score of less than 2.0.

Parameters

The Voice dashboard has a fairly typical set of Parameters:







Click-through


Testruns dashboard.


The WAP dashboard The WAP dashboard can be launched by selecting Standard Scripts / ETSI / Service / FTP Download from the




This displays results from the WAP standard script, initially for the last 24 hours, and provides controls to refine

the information and links to more detailed reports.

Viewlets


The WAP Page Session Time Viewlet displays the time taken to visit a page, within each timeslot:

Standard Reports


The three curves show the average, minimum and maximum time within each timeslot.

The example results shown above show a marked peak around 21:00 and trough around 01:00.

The WAP Page Session Failure Ratio Viewlet displays the percentage of sessions that failed, within each timeslot:

The example results shown above show a 0% failure ratio throughout most of the day (no sessions failed, once set

up), with 100% failure rates at 23:00 and 03:00.



The WAP Session Time Cumulative Distribution Viewlet displays the percentage of sessions completed within

each time:

In the example results shown above, 75% of sessions were complete within 18 seconds., and 90% within 26

seconds.

Parameters

The HTTP dashboard has a fairly typical set of Parameters:







Click-through


Testruns dashboard.

Standard Reports



The KPI Summary dashboard The KPI Summary dashboard can be launched by selecting the Standard Scripts / ETSI / KPI / KPI Summary

option from the navigation (Sidebar or Dashboards menu). It is not displayed by click-through.



This displays results from all standard scripts for a specified ETSI Metric and time period, aggregated over time -

initially Attach Setup Time for the previous day, aggregated hourly - and provides controls to refine the

information and links to more detailed reports.

Viewlets

This dashboard uses three Viewlets to display results.

The Aggregated ETSI KPIs Viewlet lists the minimum, average, and maximum values for the selected Metric for

each standard script reporting them:

The example results shown above show just two standard scripts have reported values for the Attach Time Setup

Metric. The maximum value is picked out in red.

The KPI Aggregates Viewlet plots the Metrics values for each timeslot, aggregated across all standard scripts:

The three curves show the average, minimum and maximum values within each timeslot. The example results

shown above show a marked peak around 15:00.

The Cumulative Distribution Viewlet displays the percentage of values against value:

Standard Reports


The frequency distribution curve (in red) shows the percentage of metric values for each value. In the example

results above you can see that most values lie in the range 0.7-2.8, with the peak (most frequent value) at 2.1.

The cumulative frequency curve (in blue) shows the percentage of metric values that are less than or equal to a

specific value. In the example results above, approximately 15% are 1.7 or less, while 99% are 2.5 or less.

Parameters

The KPI Summary dashboard has these Parameters:




Time aggregation - select the timeslot over which to average results

ETSI Metric type - choose the Metric type to report on by typing its name

Test - choose the name of the test. To select multiple tests use the Ctrl and Shift keys while clicking



Click-through

Click a datapoint on the KPI Aggregates Viewlet to display a list of the testruns contributing to that value, on the

List of Testruns dashboard.




The ETSI KPI Overview dashboard The ETSI KPI Overview dashboard can be launched by selecting the Standard scripts / ETSI / ETSI KPI Overview

option from the navigation (Sidebar or Dashboards menu).

Its parameters are initially set to their defaults, to list the average value for all ETSI KPIs reported by the Standard

Scripts in the last 24 hours, aggregated hourly. You can use parameters in the sidebar to change the time period

and aggregation to and filter by test or metric type.

Viewlets

This dashboard has a single viewlet, ETSI KPI Overview (a Complex Pivot):

Standard Reports


The ETSI KPI Overview viewlet lists each test, the ETSI KPIs it reported, and the average value for each timeslot

(aggregation period) and overall average value (for the selected time period).

Individual cells are colored according to their value, to highlight any failures or poor performance.

Parameters

The ETSI KPI Overview dashboard has the following Parameters:




Time aggregation - choose the period over which to average data, for the graph: 5 minutes, 10 minutes, an

hour, etc.

Test - choose the name of the test. To select multiple tests use the Ctrl and Shift keys while clicking

ETSI Metric type - choose the Metric type to report on by typing its name

Notice that the only parameter that's ticked initially is Time aggregation.

Click-through

Click any average KPI value to click through to a list of the testruns that contributed to that value:

This lists the service availability, start time, name, Agent, test type and value reported, for each testrun.

Click any value for a detailed report on that testrun:



In the Trace Files viewlet, click a icon to access that saved trace file.

The Graphical Overview dashboard The Graphical Overview dashboard can be launched by selecting Standard scripts / Graphical overview from the

navigation (Sidebar or Dashboards menu), or by clicking a Service (row) in the Result Summary dashboard.

If you launch it from the navigation, its parameters are set to their defaults; if you launch it by click-through, its

parameters are set from the Result Summary to show details for the clicked Service.

Standard Reports


Viewlets

This dashboard displays two or three Viewlets:

A graph showing how the end-to-end overall duration for the selected service varied over the selected

period. It has 3 separate curves, for the average, minimum and maximum values for each hour of the period,

as well as bars showing the average service availability for each hour.

A table beneath the graph that lists any of these testruns that reported a service failure. It starts off

collapsed; click its button to expand the table.

If there are no failures to report, you get a simple message:

If there are failures to report, two tables are displayed, like this:



Parameters

The Graphical Overview dashboard has the following Parameters:





hour, etc.

Service type - choose the Service to report on




Notice that the Time aggregation and Service type checkboxes are gray and ticked. This means that these filters

are turned on and you cannot turn them off - you can only change the filter values used.

This parameter is ALWAYS ON:

(Can you see why you're not allowed to turn these parameters off?)

Standard Reports


Click-through

Click any data point on an overall duration curves to display the Cumulative duration dashboard for this period

(described below), which shows the distribution of duration values.

Click a service availability bar to list the service failures that produced that bar, on the Service failure list

dashboard (described below).

Click any row in the List of failing transactions to display the Service Failure list dashboard (described below) for

this entire period.

Click a row in the List of failing testruns to display details of that testrun on the Projector Testrun summary

dashboard (a Standard Report described in the Using Box Office Reports guide).

The All Service Graphs dashboard The All Service Graphs dashboard can be launched by selecting Standard Scripts / All service graphs from the


It displays the average overall duration reported for each service type and time period, with each service on a

separate graph.

When you launch it, its parameters are set to their defaults (described below).

Viewlets

This dashboard displays ten Viewlets, each of which is a line graph of average overall duration for each period, for

a different service. Where there is more than one Standard Script reporting on a service, the times are plotted

together on the same graph (like the Email service type), above.

The Viewlets are for:



DNS

Email (IMAP and POP3)

FTP (Upload and Download)

GPRS (Attach and Context Activate)

MMS

Ping

SMS

Video

Voice

WAP

Web

Parameters

The All service graphs dashboard has the following Parameters:





hour, etc.




Notice that the Time aggregation checkbox is gray and ticked. This means that this filter is turned on and you

cannot turn it off - you can only change the filter values used.

Click-through

Click any data point on the overall duration curves to display the Graphical overview dashboard (described

above) for this period, which shows the average, maximum and minimum overall duration for each aggregation

period - as well as service availability.

The Cumulative Duration dashboard The Cumulative Duration dashboard can be launched by selecting Standard Scripts / Cumulative duration from

the navigation (Sidebar or Dashboards menu), or by clicking a data point in the Graphical Overview dashboard.


parameters are set from the Graphical Overview to show details for the clicked Service.

Viewlets

The Cumulative duration dashboard shows the distribution of overall duration values for the selected service, as

two curves: frequency and cumulative. The first (frequency) clearly shows which values were common and which

were not, while the second clearly shows what percentage of results achieved any given value:

Standard Reports


For example, the above graph shows:

From the frequency curve, around 30% of the tests

returned a DNS duration of around 1.7 seconds

From the cumulative curve, around 89% of all DNS

lookups were complete in around 2.2 seconds.

Hover over any point on the frequency curve to display a count of results with that duration, expressed as a

percentage.

Hover over any point on the cumulative curve to display the percentage of tests that reported that value or less.

Parameters

The Cumulative duration dashboard has the following Parameters:










Notice that the Service type checkbox is gray and ticked. This means that this filter is turned on and you cannot

turn it off - you can only change the filter values used i.e. which Service to report on.

Click-through

The Cumulative duration dashboard does not support click-through.

The Failure Overview dashboard The Failure Overview dashboard can be launched by selecting Standard Scripts / Admin / Failure overview from

the navigation (Sidebar or Dashboards menu).

Its parameters are set to their defaults (shown below).

Viewlets

The Failure overview dashboard provides a color-coded summary of the average service availability, system

availability and completion for each of the service types, for the selected period:

The report has five columns:




Average service availability % is an indicator of service reliability as experienced by users. It is calculated by

averaging the service availability reported by each testrun over the selected period and expressed as a percentage.

A single testrun reports a service as available (100%) or not available (0%); averaging this metric over time gives a

single figure that summarizes the health of the service for that period. For example, if 1 test in 10 reports the service

as unavailable, this figure would be 90%.

Values are color-coded, ranging from green (for 100%) through yellow and orange down to red (0%). A color other

than green highlights a problem with the service, such as service failure or network congestion.

If a testrun reports a system or completion error, it does not report service availability.

Average system availability % is an indicator of test system reliability - the health of your Monitor Master system.

It is calculated by averaging the system availability reported by each testrun over the selected period and

expressed as a percentage. System availability indicates whether the resources the script needed to run were

Standard Reports


available. A single testrun reports system availability as 100% or 0% (not available); averaging these results over

time gives a single figure that indicates the health of the test system for that period. For example, if 1 test in 10

reports the system as unavailable, this figure would be 90%.

Values are color-coded just like Average service availability %. A color other than green highlights a problem with

the test resources, such as a busy modem or overworked probe.

Average completion % is an indicator of test script reliability - how often the testrun completes normally. It is

calculated by averaging the completion reported by each testrun over the selected period and expressed as a

percentage. If a script ends normally, it reports a completion of 100%; if it fails (because of a scripting error), it

reports a completion of 0%.

Values are color-coded just like Average service availability %. A color other than green highlights a problem with

the script, such as a missing data file, badly-configured resource, or manual cancellation of the task.

Hover over a value to display a count of the number of testruns that the value is based on.

Parameters

The Failure overview dashboard has the following Parameters:

Time ago - select data for the last 5 minutes, hour, day, week, etc.






Click-through

Click a value to display details on the Service failure dashboard, System failure dashboard, or Completion failure

dashboard.

The Completion Failure List dashboard The Completion Failure dashboard can be launched by selecting Standard scripts / Admin / Completion failure

list from the navigation (Sidebar or Dashboards menu), or by clicking an average completion figure in the Failure

Overview dashboard.


parameters are set from the Failure Overview dashboard to show details for the clicked Service.

Viewlets

The Completion failure list dashboard provides details of any testruns that did not complete, for the selected time

period, service type, operator and location:



Each failed testrun is listed as a separate row, with details in five columns:

Time: the time and date the testrun started

Test: the test (script) name

Task: the Task Id

Duration: how long the test ran before exiting, in seconds.

Failure exit expression: the exit expression provided by the script, as an aid to failure diagnosis.

You can sort by any column by clicking the column header. For example:

Click Time once to sort into ascending order of start times

Click Time again to sort into descending order of start times

If the list is long, use the controls at the end of the table to scroll through it.

Parameters

The Completion failure list dashboard has the following Parameters:










Click-through

Click any row in the List of failing testruns to display details of that testrun on the Projector Testrun summary


Standard Reports


The Service Failure List dashboard The Service Failure List dashboard can be launched by selecting Standard Scripts / Admin / Service failure list

from the navigation (Sidebar or Dashboards menu), or by clicking an average service availability figure in the

Failure Overview dashboard.



Viewlets

The Service Failure List dashboard provides details of any service failures reported for the selected time period,

service type, operator and location:

This consists of two Viewlets.

The failing transactions table lists the transactions (stages in the script) that have reported a service failure, as:

Diagnostic: the diagnostic message returned by the transaction

Text: additional text output by the transaction

Transaction: one or more columns giving the number of times each transaction reported a failure

The list of failing testruns table lists each of the testruns that reported a service failure, with details in five

columns:



Task: the Task Id

Duration: how long the test ran for, in seconds.

Diagnostic: the diagnostic message output by the script, as an aid to failure diagnosis.

You can sort by any column by clicking the column header. If the list is long, use the controls at the end

of the table to scroll through it.

Parameters

The Service Failure List dashboard has the following Parameters:












Click-through

Click the count in the List of failing transactions to display the testruns that made up that count, in the Service

failure list dashboard.



The System Failure List dashboard The System Failure List dashboard can be launched by selecting Standard Scripts / Admin / System failure list

from the navigation (Sidebar or Dashboards menu), or by clicking an average system availability figure in the

Failure overview dashboard.



Viewlets

The System Failure List dashboard provides details of any system failures reported for the selected time period,

service type, operator and location:

The list of failing testruns table lists each of the testruns that reported a system failure, with details in five

columns:



Standard Reports


Task: the Task Id

Duration: how long the test ran for, in seconds.

Diagnostic: the diagnostic message output by the script, as an aid to failure diagnosis.

You can sort by any column by clicking the column header. If the list is long, use the controls at the end

of the table to scroll through it.

Parameters

The Service Failure List dashboard has the following Parameters:










Click-through



Standard Reports


Index

A

Access to

Standard Reports, 8

Adjusting a filter, 17

C

Click-through, 19

Configuring

access to dashboards, 8

access to Standard Reports, 8

Controlled Access, 8

Controlling

filters, 16

Creating

Box Office Jobs, 21

Box Office Schedules, 21

PDFs from reports, 21

Customer Support, 24

D

Dashboards for Standard Scripts

Launching, 12

The All Service Graphs dashboard, 63

The Completion Failure List dashboard, 67

The Cumulative Duration dashboard, 64

The ETSI KPI Overview dashboard, 58

The Failure Overview dashboard, 66

The FTP Download dashboard, 27

The FTP Upload dashboard, 31

The Graphical Overview dashboard, 60

The HTTP dashboard, 35

The KPI Summary dashboard, 55

The MMS dashboard, 39

The Result Summary dashboard, 14, 26

The Service Failure List dashboard, 69

The SMS dashboard, 42

The System Failure List dashboard, 70

The Voice dashboard, 46

The WAP dashboard, 51

Using, 12

using click-through, 19

using the Standard Reports, 25

E

ETSI compliant metrics, 5

ETSI limitations, 6

ETSI Metrics, 5

Example

schedule, 22

Standard Report Parameters, 16

Standard Report schedule, 21

F

Filing a report, 21

Filters, 16

FTP Access, 11

I

Installing

Standard Reports, 8

K

KPIs, 56

L

L3 Trace. See Radio Trace

Layer 3 Trace. See Radio Trace

M

Metrics

ETSI Metrics, 5

ETSI Metrics summary, 58

Standard Metrics, 5

Standard Reports


O

Open Access, 8

Overview

Standard Reports Overview dashboard, 60

P

Parameters, 16

Printing

Standard Reports, 20

Problems, 24

R

Radio Trace

using in Standard Scripts, 23

Report hierarchy, 20

Reports

filing, 21

installing, 8

printing, 20

re-using, 21

saving, 21

scheduling, 21

storing, 21

Re-using a report, 21

S

Saving a Standard Report, 21

Scheduling

Standard Reports, 21

SiteConfig.xml File, 8

Standard Metrics, 5

Standard Reports, 25, 27, 31, 35, 39, 42, 46, 51, 55, 60

Click-through, 15

Hover, 15

Installing, 8


















Using, 12

using click-through, 19

using Standard Reports, 25

Standard Scripts

ETSI Metrics, 5

Overview, 5

Reports, 5

Standard Metrics, 5

Storing a report, 21

Support, 24

T

TEMS Discovery, 23














Standard Reports






U

Using

Click-through in Standard Reports, 19

IP Trace in Standard Reports, 24

Parameters in Standard Reports, 16

Radio Trace in Standard Reports, 23

Schedules for Standard Reports, 21

Sidebar in Standard Reports, 16

Standard Reports, 12, 25

Tables in Standard Reports, 15

Viewlet controls in Standard Reports, 19

Viewlet menus in Standard Reports, 18

V

Viewlet controls, 19

Viewlet menu, 18

tems™ monitor master v10.4 - mantica-qm.ant...

Documents