kpi measurement monitoring and analysis guide

7/29/2019 KPI Measurement Monitoring and Analysis Guide

1/35

EUROPEAN ORGANISATION FOR THE SAFETY OFAIR NAVIGATION

EUROCONTROL

EUROPEAN AIR TRAFFIC MANAGEMENT PROGRAMME

KPI Measurement,Monitoring and Analysis

Guide

AIM/AEP/S-LEV/0008

Edition : 0.1Edition Date : 30 Nov 2001Status : DraftClass : General Public


2/35


3/35

DOCUMENT IDENTIFICATION SHEET

DOCUMENT DESCRIPTION

Document TitleKPI Measurement, Monitoring and Analysis Guide

EWP DELIVERABLE REFERENCE NUMBER

PROGRAMME REFERENCE INDEX EDITION : 0.1

AIM/AEP/S-LEV/0008 EDITION DATE : 30 Nov 2001

Abstract

This guide provides an introduction to Key Performance Indicator (KPI) measurement, monitoring andanalysis.

KeywordsQuality Management Service Level Performance Indicator

KPI Measurement Data Collection AnalysisMonitoring

CONTACT PERSON : P. Bosman TEL : 3333 UNIT :

DOCUMENT STATUS AND TYPE

STATUS CATEGORY CLASSIFICATION

Working Draft o Executive Task o General Public

Draft Specialist Task o EATMP o

Proposed Issue o Lower Layer Task Restricted o

Released Issue o

INTERNAL REFERENCE NAME : AHEAD Electronic Filing System


4/35

AIM/AEP/S-LEV/0008 KPI Measurement, Monitoring and AnalysisGuide

Page iv Draft Edition: 0.1

DOCUMENT APPROVAL

The following table identifies all management authorities that have successively approvedthe present issue of this document.

AUTHORITY NAME AND SIGNATURE DATE

Author/EditorErtan Ozkan 30 Nov 2001

Programme

Manager

Conrad Cleasby 30 Nov 2001

Quality Assurance


5/35

KPI Measurement, Monitoring and AnalysisGuide

AIM/AEP/S-LEV/0008

Edition: 0.1 Draft Page v

DOCUMENT CHANGE RECORD

The following table records the complete history of the successive editions of the presentdocument.

EDITION DATE REASON FOR CHANGESECTIONS

PAGESAFFECTED

0.1 30 Nov 2001 Creation All


6/35


Page vi Draft Edition: 0.1


7/35


AIM/AEP/S-LEV/0008

Edition: 0.1 Draft Page vii

TABLE OF CONTENTS

1. INTRODUCTION......................................................................................................1

1.1 Purpose and scope ................................................................................................1

1.2 References..............................................................................................................1

2. DATA COLLECTION ...............................................................................................2

2.1 Data Collection Plan...............................................................................................2

2.2 Measurement Techniques......................................................................................2

2.2.1 Event-Driven Measurement ........................................................................3

2.2.2 Sampling-Based Measurement...................................................................5

2.2.3 Simulation...................................................................................................5

2.3 Designing a Data Collection System.....................................................................5

3. MONITORING..........................................................................................................7

3.1 Operational Report .................................................................................................7

3.2 Real-Time Reports..................................................................................................8

3.3 Executive Summaries.............................................................................................8

3.4 Customer Reports ..................................................................................................8

4. ANALYSING AND INTERPRETING KPIS ...............................................................9

4.1 Variations (Trend) Analysis ...................................................................................9

4.2 Identifying Relationships (Cause-Effect Analysis).............................................11

4.2.1 Scatter Diagrams......................................................................................11

4.2.2 Stratification..............................................................................................12

4.3 Capability Analysis...............................................................................................13

4.4 Capacity Analysis.................................................................................................13

4.5 Considering Context ............................................................................................14

4.6 Establishing Priorities..........................................................................................15

5. CONCLUSIONS AND FUTURE WORK.................................................................16

QUALITY TOOLS................................................................................................................17

A.1 Cause and Effect Diagram ...................................................................................17

A.1.1 Usage .......................................................................................................17

A.1.2 Steps ........................................................................................................ 17

A.1.3 Example.................................................................................................... 19

A.2 Check Sheet..........................................................................................................19

A.2.1 Usage .......................................................................................................19

A.2.2 Steps ........................................................................................................ 19

A.2.3 Example.................................................................................................... 20

A.3 Pareto Chart..........................................................................................................20

A.3.1 Usage .......................................................................................................20

A.3.2 Step-by-Step Example .............................................................................. 21


8/35


Page viii Draft Edition: 0.1

A.4 Scatter Diagram....................................................................................................22

A.4.1 Usage .......................................................................................................23

A.4.2 Steps ........................................................................................................ 23

A.4.3 Example.................................................................................................... 24

A.5 Run Chart..............................................................................................................24

A.5.1 Usage .......................................................................................................24

A.5.2 Steps ........................................................................................................ 24

A.6 Affinity Diagram....................................................................................................25

A.6.1 Usage .......................................................................................................25

A.6.2 Steps ........................................................................................................ 25

A.7 Nominal Group Technique...................................................................................26

A.7.1 Usage .......................................................................................................26

A.7.2 Steps ........................................................................................................ 26

A.7.3 Example.................................................................................................... 26

A.8 Interrelations Digraph ..........................................................................................26

A.8.1 Usage .......................................................................................................27

A.8.2 Steps ........................................................................................................ 27

A.9 Force Field Analysis.............................................................................................27

A.9.1 Usage .......................................................................................................27

A.9.2 Steps ........................................................................................................ 28

A.10 Matrix Diagram .....................................................................................................28

A.10.1 Usage .......................................................................................................28

A.10.2 Steps ........................................................................................................ 28


9/35


AIM/AEP/S-LEV/0008

Edition: 0.1 Draft Page 1

1. INTRODUCTION

1.1 Purpose and scope

This guide explores issues related to Key Performance Indicator (KPI)measurement, monitoring and analysis. The main objective is to provide necessarybackground required for deploying a performance measurement system within thecontext of quality and service level management.

This guide is organised as follows:

Chapter 2 presents the issues related to data collection: brief summary ofdata collection, data collection plan and measurement techniques.

In the Chapter 3 how to monitor KPIs is discussed and particularly reportingissues are explored.

Chapter 4 gives some guidelines in order to analyse your KPIs.

Conclusion and future work are given in the chapter 5.

Appendix A contains summary of quality tools that may be useful in thedeployment of a performance measurement system.

1.2 References

[1] Foundations of Service Level Management, April 2000, Rick Sturm, WayneMorris and Mary Jander, SAMS Publications, ISBN 0-672-31743-5.

[2] Change Management the 5-step action kit, C. Rye, ISBN 0749433809, KoganPage Limited, 2001.

[3] Operational Performance Measurement Increasing Total Productivity, W.Kaydos, ISBN 1574440993, CRC Press LLC, 1999.

[4] Applications of Performance Measurement, Paul Arveson, The BalancedScorecard Institute, http://www.balancedscorecard.org/appl/index.html, 1998.

[5] Basic Tools for Process Improvement, Module 7, Data Collection, 1996.


10/35


Page 2 Draft Edition: 0.1

2. DATA COLLECTION

Data collection helps you to assess the health of your system and processes. To doso, you must identify the Key Performance Indicators (KPIs) you will measure, howyou will measure them and what you will do with the data you collect.

Every improvement effort relies on data to provide a factual basis for makingdecisions for improvement. Data collection enables a team to formulate and testworking assumptions and develop information that will lead to the improvement ofthe KPIs of the product, service or system. Data collection improves your decision-making by helping you focus on objective information about what is happening,rather than subjective opinions. In other words, I think the problem is... becomesThe data indicate the problem is...

To collect data uniformly, you will need to develop a data collection plan. Theelements of the plan must be clearly and unambiguously defined. Data collectioncan involve a multitude of decisions by data collectors. When you prepare your data

collection plan, you should try to eliminate as many subjective choices as possibleby operationally defining the parameters needed to do the job correctly. Your datacollectors will then have a standard operating procedure to use during their datacollection activities.

2.1 Data Collection Plan

While preparing data collection plans you have to answer the following questions:

Why do we want the data? What will we do with the data after we havecollected them? You must decide on a purpose for collecting the data. Thisaction focuses you on the specific quality characteristic you want to improve,

and sets the stage for where you will collect the data.

Where will we collect the data? The location where data are collected mustbe identified clearly.

Who will collect the data? The answer is simple: Those closest to the data(e.g., the process workers) should collect the data. These people have thebest opportunity to record the results. They also know the process best andcan easily detect when problems occur. But remember, the people who aregoing to collect the data need training on how to do it and the resourcesnecessary to obtain the information such as time and measurement tools.

How do we collect the right data? You have to specify the amount andfrequency of data collection. However you need to remember that you arecollecting data for the purpose of future improvement efforts. Therefore youhave to take into consideration the cost of obtaining the data, the availabilityof data and the consequences of decisions made on the basis of the datawhen determining how much data should be obtained and how frequently itshould be collected.

2.2 Measurement Techniques

There are three main techniques for collecting data:

Event-Driven Measurement

Sampling Based Measurement


11/35


AIM/AEP/S-LEV/0008


Simulation

2.2.1 Event-Driven Measurement

In case of event-driven measurement the times at which certain events happen arerecorded and then desired statistics are computed by analysing data. Sometimesthis is called as event management in the literature.

Although the event-driven measurement varies from company to company, there arethree distinct and common steps:

To detect events (e.g., failure of a computer, error in a publication)

To record time and nature events (e.g., to record time and nature ofcomputer failure or error in a publication)

To take corrective actions by using procedure that outline how the event

should be managed (e.g., to make the computer up and running again or tocorrect error in a publication)

Events can be detected and recorded by

Agents

Human beings

Agent. An agent is a piece of software designed to collect data about the status andfunctionality of a device, system or application for reporting purposes.Among manytools the popular examples are First Sense, Empire, OpenView (HP), etc.

Agents capture data directly from the hardware elements underlying the service(network, bridges, routers, switches, hubs, etc.) or they gather input from softwareprograms that affect overall service availability (applications, databases,middleware, etc.) They report events directly as they occur. Examples includehardware and software failures, broken routers, etc.

Human Beings. In this case the event is detected by the people involving in theprocess. The recording is generally done by using checksheets. Checksheets arestructured forms that enable people to collect and organise data systematically.Checksheets may be computerised (e.g., workflow management or documentmanagement systems contain similar forms).

Because each checksheet is used for collecting and recording data unique to aspecific process or system, it can be constructed in whatever shape, size and formatare appropriate for the data collection task at hand. There is no standardised formatthat you can apply to all checksheets. Instead, each checksheet is a form tailored tocollect the required information. However you may use the following guidelines whiledeveloping useful checksheets:

Involve the process workersin developing the checksheet for their process.

Label all columns clearly. Organise your form so that the data are recordedin the sequence seen by a person viewing the process. This reduces the

possibility of data being recorded in the wrong column or not being recorded.

Make the form user-friendly. Make sure the checksheet can be easilyunderstood and used by all of the workers who are recording data.


12/35



Week 14- 2001

Monday Tuesday Wednesday Thursday Friday Total1. Input Hopper Faults

5

2. A4 Paper Tray Fault10


4. Internal Paper Jams25

5. Duplex Processing Faults5

6. Stapling Faults0

7. Output Collation Faults2

Figure 1. A Sample Checksheet for a Photocopy Machine

Create a format that gives you the most information with the least amount ofeffort. For example, design your checksheet so that data can be recordedusing only a check mark, slant mark, number or letter.

Provide enough space for the collectors to record all of the data.

Designate a place for recording the date and time the data were collected.

These elements are required when the data are used with Run Charts orother tools which require the date and time of each observation.

Provide a place to enter the name of the individual collecting the data.

1

4

5

7


13/35


AIM/AEP/S-LEV/0008


Allow enough space so data collectors can write in comments on unusualevents.

2.2.2 Sampling-Based Measurement

In the sampling-based measurement one selects a sample of a certain product orpublication and checks the selected sample. For example in order to measurecorrectness, a set of AIS publications (i.e., not all) is taken and checked forcorrectness. The correctness is determined by using this selected sample.

This technique is particularly useful when you dont have enough tools or manpowerin order to collect data.

2.2.3 Simulation

This technique is generally used in automated environments in order to measureservice availability. The basic idea is to generate synthetic service/product requestsat regular intervals and collecting availability and performance measures based ontracking these requests.

This technique has some certain advantages:

It does not require external participation (e.g., your customers)

It does not require real-time agents that can be heavy and expensive interms of computing power and investment.

2.3 Designing a Data Collection System

The design of a data collection system depends on the following:

What has to be collected?

What mechanisms are already in place?

If there are many unknowns about what data is required, it is probably best to startwith a manual data collection system. After the system has been used and thesystem design has stabilised, it can be automated.

The first problem is to get everyone to report reasonably accurate data. This will bea challenge, no matter how much instruction is provided. Most people get into the

routine very quickly, but some will require considerable support and instruction.Intensive follow-up and checking of detail is needed to be sure all problems arereported and that they are reported correctly. During first several weeks ofimplementation, supervisors should review all data sheets and look into any entriesthat look questionable.

When KPIs are first implemented, there will many questions about what they mean,where they come from and how they should be interpreted, even if all this wasexplained before starting. Managers and the system developers should carefullylisten to any questions and objections because they may indicate where the systemneeds to be improved.

If KPIs are not being used and analysed, there are only four possible explanations:

There is a lack of leadership of the program

The users dont understand information


14/35



The information is not relevant users needs.

The information is incorrect or unreliable

The following should be considered during the design of a Data Collection System:

Make it Easy to Report Data. Make it as easy as possible to record or enter data.Dont add steps and people to a production process to capture necessary data.Build reporting into the process by modifying forms and procedures.

Dont Overkill. Dont take the approach of collecting every bit of available data andrearranging it into massive reports that no one can use. Instead first determine whatinformation is needed and then develop the system to supply it.

Reports and graphs should be designed for specific users and purposes. Sincedifferent managers must make different decisions, they need different information.Give every the graphs and summary reports of all KPIs that are relevant to them.

Decentralise the Measurement System. Dont try to build a centrally controlled,one-size-fits-all system. It will be cumbersome, slow and inefficient. There are goodreasons for decentralising measurement systems:

Measures and data systems needed in different processes and sub-processes are diverse that building a system to accommodate all the needswould be practically impossible.

Systems customised for different functions are more efficient and moreeffective than more general solutions

The best approach appears to be to take a decentralised approach but keep closelycoupled functions under the same umbrella so the data will share a commonstructure and can be easily interrelated.

Level of Detail. The amount of detail needed to identify the root causes ofproblems is typically more than what is required to establish accountability. In theoryany thing (e.g., process) could be measured so extensively that the root cause ofany problem could be quickly isolated. However, in general, it is very expensive.Therefore it is required to select right level of details that makes the best trade-offbetween how much data to collect and how often the detailed data is need. Anotherapproach is to find the root-cause iteratively. In the iterative the probable causes canbe identified in suspected areas and additional data can be collected to finally

determine the real causes of problems.

Combining Measure to Create a Single Composite Index. All managers would

like to have one KPI that would indicate when everything was not in fine shape andtell them what to do about it. Unfortunately it is not possible since complex systemscannot be controlled with simple measurement systems. However it is still useful tocomposite KPIs for a department or a process since they can help keep the relativeimportance of individual KPIs. The easiest way of constructing a composite KPI is toassign a weighting factor to each component and calculate the weighted average.

Security of Confidential Information. For KPIs to be effective as motivator,everyone must be kept abreast of performance. While it is best for KPIs to be

available for everyone, it may be necessary to keep some information confidentialfor competitive reasons.


15/35


AIM/AEP/S-LEV/0008


3. MONITORING

KPI reporting is an important communication vehicle while monitoring KPIs.However the biggest danger with KPI reporting is to create information overload bydeveloping and distributing too many reports, charts and tables to too many people.The quality of information (and its value) is, in general, inversely proportional to thevolume information.

For effective communications, reports has to fulfil the following requirements:

Relevant to the person receiving it. This requirement has two aspects: (1)Making sure that managers get all information that is relevant to them (2)and also they get nothing that is not relevant to them. Information notneeded or not used is just another form of waste.

Well organised. Cause-effect relationships, process relationships and therelative importance of KPIs to organisation or operating unit should be

readily apparent.

Understood by those using it. Information that isnt understood is justanother form of waste (useless noise)

Kept as brief as possible. Since everyones time is limited and valuable,the shorter a report is the more likely it will be used. Wading through pagesof numbers to find important points is not effective use of any managerstime.

Reports should provide information to lines of business and customer community.The audience for certain information affects the format of reports, different reports

may be required to cover different aspects of KPIs and to satisfy the interests andfocus of various audiences. The following types of reports are generally used formonitoring:

Operational Reports

Real-Time Reports

Executive Summaries

Customer Reports

3.1 Operational Report

The format and content of the operational reports vary with the purpose of analysisto be done (Please see the following chapter for details and examples):

Trend Analysis. Such reports should present a single KPI as a function oftime. The report should be supported by a simple line graph or Run Chart inorder to make trend analysis.

Cause-Effect Analysis. In case of cause-effect analysis it is checkedwhether there is a relationship between two or more KPIs. Therefore suchreports should contain values of KPIs in question. They are generallysupported with a scatter diagram or stratification table.

Capability Analysis. Such reports are used to keep track of the KPI valueswhen you start making a change in order to improve the system or


16/35



processes. These reports will be very similar to ones used in the trendanalysis. However in this case the KPI values are presented as a function ofchange (i.e., before and after the change(s)) not time.

Capacity Analysis. These reports are only applicable to KPIs that are usedto measure capacity (e.g., number of publications per month) and associatedquality KPIs (e.g., rate of errors in publications). Such reports are used tofind out where the capacity saturates and at which capacity you can stillproduce high quality products.

3.2 Real-Time Reports

Real-time reporting and proactive notification of problems increase customers'confidence and flexibility. They generally cover service provision problemshappened due to changes in the environment such as:

Scheduled outages

Unavailability under heavy security attacks (such as virus, hackers, etc.)

Strikes

Etc.

Such kind of notification should show which end customers are affected as well asapplications, locations and lines of business that are affected. It should also showthe nature of the problem and its symptoms along with when service is anticipated toreturn to normal.

3.3 Executive Summaries

You should keep in mind one thing when tailoring reports for executive managers:they have no time!

Such reports should provide an overall assessment of achieved performance levelsincluding quantitative and qualitative reports. It should provide quick summaries ofperformance levels and make effective use of graphs and charts to impart thisinformation. Relations achieved performance level with any business impact is animportant aspect of the executive summary.

The executive summary should be self-contained, particularly for end-of-period

reports aimed at senior management and lines of business. If there are KPIs forwhich problems have been experienced, they should be highlighted with referencesto any supporting documentation or detailed reports.

3.4 Customer Reports

These reports should provide customers with summarised reports on service andproduct delivery. If there are KPIs for which problems have been experienced andwhich are important to customers (e.g. service availability), they should behighlighted. These reports should also cite any steps it takes to improve customerservice.


17/35


AIM/AEP/S-LEV/0008


4. ANALYSING AND INTERPRETING KPIS

You may design KPIs very well and in a reliable manner. However if the data andinformation are not properly analysed and interpreted, the benefits will be limited.

Main objectives of KPI Analysis are as follows:

Identifying opportunities and problems

Determining priorities

Taking action to improve

Making decisions to re-allocate resources

Changing or adjusting strategy

Providing feedback to change behaviour

Recognising and rewarding accomplishments

Although rigid rules for analysing and interpreting KPIs and their related data cannotbe defined, some guidelines will help assure that the data is analysed correctly andthe right conclusions are drawn.

4.1 Variations (Trend) Analysis

All KPIs will exhibit some variation. At the lower levels of detail, this variation can bequite large even if everything is under control. The first rule to follow when

interpreting KPIs is to not react to short-term deviations until reasons for thedeviation are understood. If the deviation is within the normal range, there has beenno change in performance at all. If it is a very large deviation, something unusualhas happened and the cause should be determined. In most cases, specialproblems or circumstances are known by those responsible for the KPI.

A simple line graph orRun Chart will provide a good sense of the normal variation.This is one reason why KPIs should always be put on run charts instead of relyingsolely on reports.

In order explore how the variance can be analysed; lets assume that you measurethe time to reach your office each morning:

Measurements for Time to Reach (Minutes)

Day 1 2 3 4 5 6 7 8 9 10

Time 25.3 22.1 24.4 26.8 27.3 26.6 24.2 22.0 21.3 23.9


18/35



10

12

14

16

18

20

2224

26

28

30

1 2 3 4 5 6 7 8 9 10

Days

TimetoReach(mins)

Figure 2. Example of Stable Trend

Although there is a fluctuation between 21.3 and 27.3 minutes, the trend is quitestable around median value 24.3 and data points do not show a particular andsteady trend. Therefore there is no reason that you try to find out why it took you27.3 minutes in the day 5.

Lets assume that you continue your measurements next ten days and you obtainthe following measures.

Measurements for Time to Reach (Minutes)

Day 11 12 13 14 15 16 17 18 19 20

Time 18.1 17.6 17.2 15.1 14.4 14.0 12.6 12.2 14.5 15.3

New measurements show a descending trend. The trend is confirmed with

reasonable number of consecutive data points. You can conclude that your time toreach office has been reduced.


19/35


AIM/AEP/S-LEV/0008


10

12

14

16

18

20

22

24

26

28

30

1 3 5 7 9 11 13 15 17 19

Days

TimetoReach(mins)

Figure 3. Example of Change in Trend

4.2 Identifying Relationships (Cause-Effect Analysis)

Identifying relationships between variables is important for understanding how athings work and also the causes of problems. Looking for relationships should bepart of analysing any KPI, especially those that have several external or internalvariables that might affect the performance. This also includes customers, becauseparticular customers can influence quality measures such as complaints and general

satisfaction.

While identifying relationships between two variables X and Y there are threepossibilities that should be considered:

X and Y are not related at all. The apparent relationship is the result of purecoincidence.

X and Y are related, but X does not cause Y or vice-versa. Instead they areaffected by another variable(s).

X and Y have a cause-effect relationship.

There are two popular methods used for identifying relationships: scatter diagramsand stratification.

4.2.1 Scatter Diagrams

Scatter Diagrams are a simple way of identifying whether a relationship existsbetween two variables and, if so, the strength of the influence of one variable uponthe other, e.g. the effect of increases in temperature on the consumption of domesticdrinking water. Analysis with Scatter Diagrams is done through plotting the datafrom each data set on a graph. Horizontal axis of the graph is scaled for the causevariable and vertical axis for the effectvariable (maximum values applicable to eachaxis will be determined by reference to values within the data sets).

If there is a strong relationship between two variables, it will probably be indicatedby the Scatter Diagram. If the relationship is weak, it may not be readily apparent,

Trend


20/35



especially if there are other factors influencing the behaviour of the variable ofinterest.

Figure 4. Correlation Examples with Scatter Diagrams for variables X and Y.

4.2.2 Stratification

Stratifying data is another way of identifying relationships between variables.Stratification consists of cutting the data into layers according to the differentvariables in question. For example, to analyse customer complaints about toasters itmight be appropriate to look at them by model number, which plant made them andwhich retail chain sold them. Lets take the following table as an example:

Model % Plant % Chain %

106A 1.1 1 0.9 K 0.6

117B 2.6 2 0.8 S 0.8

419B 0.8 3 2.4 P 0.7

777A 0.5 4 0.9 T 0.8

W 2.1

The analysis indicates a possible relationship with model 117B, plant 3 and storechain W but these relationships are not certain and require further investigation

since it is possible that:

The chain W sold more 117B units than others or


21/35


AIM/AEP/S-LEV/0008


The plant 3 produced more117B units or

There is nothing wrong with 117B units but chain W mistakenlymisrepresented the units in its advertisements.

Therefore although stratification illustrates possible relationships it is not definitive atall and a further investigation should be carried out.

4.3 Capability Analysis

Understanding process capability is important for both control and planningpurposes. For large and complex processes the capability question is Whatperformance level can be maintained by the process? In order to determine thisperformance level it may be necessary to select a period that seems to representnormal operating conditions and derive a performance target from this period.

However this target only applies to the process in its current configuration. Often

real question is At what point should attempts to incrementally improve thecapability of a process be abandoned in favour of a radical restructuring (or re-engineering) of the process. Lets take the following table, which illustratesimprovements provided by each successive change in a production process, as anexample:

Event Ratio of Errorsin Publications

Start 10.0

Change 1 5.0

Change 2 4.1

Change 3 3.4

Change 4 2.8

It might seem further significant improvements are not feasible the after change 4.However it should be noted that determining when a process has reached itspractical limit for incremental improvement is a matter of judgement. If the personmaking that judgement understands how well the process is performing and itsimprovement history, that determination will probably be quite accurate.

4.4 Capacity Analysis

How much work a process can do in a given period is its production capacity. It is atype of capability. Knowing this figure is obviously essential for effective planningand management. When production capacity is exceeded, the following will happen:

If the process is limited by equipment capacity, work will pile up in front of thelimiting steps of the process production delay will increase.

If the process is limited by labour capacity, work will pile up in front of thebottleneck step(s) in the process, but the work may also get done while the

quality of the work suffers rework and rejects will increase.

Identification of capacity for the first case is relatively straightforward. The saturationpoint can be determined after plotting production data (e.g., number of publications):


22/35



Figure 5. Capacity Analysis (I)

For the second case an estimate of production capacity can be derived from qualityand production data:

Figure 6. Capacity Analysis (II)

4.5 Considering Context

KPIs do not exist in a vacuum. They are affected by anything that affects anorganisation or its production processes. Weather, strikes, supply line disruptions,unusual customer requests, competitors actions and many others can cause largedeviations in the KPIs. That is why it is a good practice to note significant changes inenvironmental factors or unusual circumstances on charts when they occur. Besidesexplaining what caused particular behaviour, these notes can help managers predictwhat will happen under similar conditions in the future.

However although context is important making allowances for poor performance canalso be overdone. Minor events should never be used to make excuses for largechanges in the performance. The same is true when performance is unusually good.

This could also be caused by favourable circumstances. Even if the circumstancesare not controllable, understanding what happened can lead to new opportunities.

ProductionUnits/Month

TimeMonths

Range whereproduction saturates

Error Rate

%

ProductionUnits/Month

TimeMonths

Errors start increasing in this region.The capacity cut-off is a value in that

region


23/35


AIM/AEP/S-LEV/0008


4.6 Establishing Priorities

Because there will always be more problems and opportunities than there areresources available to pursue them, managers must always think in terms ofpriorities. Priorities for improving performance or changes in these priorities shouldbe one of the regular outputs of analysing KPIs. Assuming a measurement systemhas the capability of determining the relative impact of KPIs, priorities should berelatively clear in terms of costs or profit opportunities. However priority decisionsmust be supported with the following:

Potential risk

Investment required

Payback period

How well projects support strategic objectives

Availability of resources

Etc.

Priorities must be evaluated from the broader perspective of the total organisation toavoid sub-optimisation and to assure resources are allocated to the areas of mostreturn.


24/35



5. CONCLUSIONS AND FUTURE WORK

This guide discusses issues and problems that can be encountered duringmeasurement, monitoring and analysis of KPIs.

This guide covers the following topics:

How to prepare data collection plans

Tips and tricks to design a data collection system

Measurement techniques

Monitoring and reporting

Different types of analysis that can be done via KPIs: trend analysis, cause-effect analysis, capability analysis, capacity planning, etc.

The provided information will assist AIS organisations in order to fulfil the associatedISO9000:2001 requirement (i.e., section 8). It will be also useful while implementingservice level management.

It should be noted that the main objective is not only to measure but also to takecorrective and preventive actions by analysing performance levels achieved forKPIs.

The document is still in its early stages and it requires some detailed examplesespecially for KPI monitoring and analysis.


25/35


AIM/AEP/S-LEV/0008


QUALITY TOOLS

This appendix is taken from The Quality Tools Cookbook by Prof. Sid Sytsma -Ferris State University and Dr. Katherine Manley - Ferris State University. Theyused the following references:

Brassard, M. and Ritter, D. (1994). The Memory Jogger, Goal/QPC,

Methuen, MA

Brassard, M. (1989). The Memory Jogger+, Goal/QPC, Methuen, MA

Brassard, M., Tucker, S. and Oddo, F. (1993). Total Quality Management inEducation, Goal/QPC, Methuen, MA

Byrnes, M., and Cornesky, R. (1994). Quality Fusion: Turning Total QualityManagement into Classroom Practice, Cornesky and Associates, PortOrange, FL

Chang, R. and Niedzwiecki, M. (1993). Continuous Improvement Tools Vol.1, Richard Chang Associates, Irv ine, CA

Chang, R. and Niedzwiecki, M. (1993). Continuous Improvement Tools Vol.2, Richard Chang Associates, Irv ine, CA

Cornesky, R. (1993). The Quality Professor, Magna Publications Inc.,Madison, WI

Cornesky, R., and McCool S. (1992). Total Quality Improvement Guide forInstitutions of Higher Education, Madison, WI

E. B. Dean's Internetpage at [email protected]

Ishikawa, K (1982). Guide to Quality Control, Quality Resources, WhitePlains, NY.

McCloskey, L.A, and Collett, D.N. (1993). TQM a Basic Text, Goal/QPC,

Methuen, MA

The tools given in this appendix are supposed to assist AIS organisation whileanalysing their KPIs.

A.1 Cause and Effect Diagram

A Cause and Effect Diagram (Fishbone Diagram) is an analysis tool to displaypossible causes of a specific problem or condition.

A.1.1 Usage

Identifying potential causes of a problem or issue in an orderly way (e.g.,Why is the production process suddenly producing so many defects?)

Summarising major causes under four categories (e.g., People, Machines,

Methods, and Materials or Policies, Procedures, People, and Plant)

A.1.2 Steps

Prepare a flip chart or an overhead transparency of the following template:


26/35



Error! Unknown switch argument.

Write the issue (problem or process condition) on the right side of the Causeand Effect Diagram.

Identify the major cause categories and write them in the four boxes on theCause and Effect Diagram. You may summarise causes under categoriessuch as:

o Methods, Machines, Materials, People

o Places, Procedures, People, Policies,

o Surroundings, Suppliers, System, Skills

Brainstorm potential causes of the problem. As possible causes areprovided, decide as a group where to place them on the Cause and EffectDiagram. It is acceptable to list a possible cause under more than one majorcause category.

Review each major cause category. Circle the most likely causes on thediagram.

Review the causes that are circled and ask "Why is this a cause?" Asking"why" will help get to the root cause of the problem.

Reach an agreement on the most probable cause(s).

A.1.3 Example


A.2 Check Sheet

The Check Sheet is a data-gathering and interpretation tool.

A.2.1 Usage

Distinguishing between fact and opinion (example: how does the communityperceive the effectiveness of the school in preparing students for the world ofwork?)

Gathering data about how often a problem is occurring (example: how oftendoes printing machine out of order?)

Gathering data about the type of problem occurring (example: What is themost common type of word processing error created by the employees-grammar, punctuation, transposing letters, etc.?)


27/35


AIM/AEP/S-LEV/0008


A.2.2 Steps

Clarify the measurement objectives. Ask questions such as "What is theproblem?", "Why should data be collected?", "Who will use the informationbeing collected?", "Who will collect the data?"

Create a form for collecting data. Determine the specific things that will bemeasured and write this down the left side of the check sheet. Determine thetime or place being measured and white this across the top of the columns.

Collect the data for the items being measured. Record each occurrencedirectly on the Check Sheet as it happens.

Tally the data by totalling the number of occurrences for each category beingmeasured.

A.2.3 Example


28/35



Week 14- 2001

Monday Tuesday Wednesday Thursday Friday Total

1. Input Hopper Faults5



4. Internal Paper Jams25

5. Duplex Processing Faults

5

6. Stapling Faults0

7. Output Collation Faults2

Figure 7. A Sample Checksheet for a Photocopy Machine

A.3 Pareto Chart

A Pareto Chart is a special form of a bar graph and is used to display the relativeimportance of problems or conditions.

A.3.1 Usage

Focusing on critical issues by ranking them in terms of importance andfrequency (e.g., Which problem with Product X is most significant to our

customers?)

Prioritising problems or causes to efficiently initiate problem solving (e.g.,Solution of what production problem will improve quality most?)

2

3

5

6


29/35


AIM/AEP/S-LEV/0008


Analysing problems or causes by different groupings of data (e.g., bydepartment, by machine, by team)

Analysing the before and after impact of changes made in a process (e.g.,Has the initiation of a quality improvement program reduced the number ofdefectives?)

A.3.2 Step-by-Step Example

Determine the categories of problems or causes to be compared. Begin byorganising the problems or causes into a narrowed down list of categories(usually 8 or less).

Select a Standard Unit of Measurement and the Time Period to be studied. Itcould be a measure of how often something occurs (defects, errors, tardies,cost overruns, etc.); frequencies of reasons cited in surveys as the cause ofa certain problem; or a specific measurement of volume or size. The time

period to be studied should be a reasonable length of time to collect thedata.

Collect and Summarise the Data. Create a three-column table with theheadings of "error or problem category", "frequency", and "percent of total".In the "error or problem category" column list the categories of problems orcauses previously identified. In the "frequency" column write in the totals foreach of the categories over the designated period of time. In the "percent oftotal" column, divide each number in the "frequency" column by the totalnumber of measurements. This will provide the percentage of the total.

Error Category Frequency Percentage

Punctuation 22 44%Grammar 15 30%Spelling 10 20%Typing 3 6%TOTAL 50 100%

Create the framework for the horizontal and vertical axes of the ParetoChart. The horizontal axis will be the categories of problems or causes indescending order with the most frequently occurring category on the far left(or at the beginning of the horizontal line). There will be two vertical axes-oneon the far left and one on the far right. The vertical axis on the far left pointwill indicate the frequency for each of the categories. Scale it so the value atthe top of the axis is slightly higher than the highest frequency number. Thevertical axis on the far right will represent the percentage scale and shouldbe scaled so that the point for the number of occurrences on the left matcheswith the corresponding percentage on the right.


Plot the bars on the Pareto Chart. Using a bar graph format, draw thecorresponding bars in decreasing height from left to right using the frequencyscale on the left vertical axis. To plot the cumulative percentage line, place adot above each bar at a height corresponding to the scale on the right

vertical axis. Then connect these dots from left to right, ending with the 100%point at the top of the right vertical axis.


30/35




Interpret the Pareto Chart. Use common sense-just because a certain problemoccurs most often doesn't necessarily mean it demands your greatestattention. Investigate all angles to help solve the problems-What makes thebiggest difference? What will it cost to correct the problems? What will it cost ifwe don't correct this problem?

A.4 Scatter Diagram

A Scatter Diagram is used to interpret data by graphically displaying the relationshipbetween two variables.

A.4.1 Usage

Validating "hunches" about a cause-and-effect relationship between types ofvariables (example: Is there a relationship between the production speed ofan operator and the number of defective parts made?)

Displaying the direction of the relationship (positive, negative, etc.) (example:Will increasing assembly line speed increase or decrease the number ofdefective parts made?)

Displaying the strength of the relationship (example: How strong is therelationship between assembly line speed and the number of defective partsproduced)

A.4.2 Steps

Collect two pieces of data (a pair of numbers) on a process, or product.Create a summary table of the data.

Draw a diagram labelling the horizontal and vertical axes. It is common thatthe "cause" variable be labelled the horizontal (X) axis and the "effect"variable be labelled the vertical (Y) axis. The values should increase up thevertical scale and toward the right on the horizontal scale. The scale on boththe X and Y-axes should be sufficient to include both the largest and thesmallest X and Y values in the table.

Plot the data pairs on the diagram by placing a dot at the intersections of theX and Y coordinates for each data pair.


Interpret the scatter diagram for direction and strength.

o Interpreting the direction: Data patterns may be positive, negative,

or display no relationship. A positive relationship is indicated by anellipse of points that slopes upward demonstrating that an increase inthe cause variable also increases the effect variable. A negative

relationship is indicated by an ellipse of points that slopes downwarddemonstrating that an increase in the cause variable results in adecrease in the effect variable. A diagram with a cluster of pointssuch that it is difficult or impossible to determine whether the trend is


31/35


AIM/AEP/S-LEV/0008


upward sloping or downward sloping indicates that there is norelationship between the two variables.

o Interpreting the strength: Data patterns, whether in a positive ornegative direction, should also be interpreted for strength byexamining the "tightness" of the clustered points. The more the pointsare clustered to look like a straight line the stronger the relationship.

A.4.3 Example


A.5 Run Chart

Run charts are used to analyse processes according to time or order. Run chartsare useful in discovering patterns that occur over time.

A.5.1 Usage

Simplest display of trends over time

Data plotted in time order

An aid to understanding basic characteristics of a process.

A.5.2 Steps

Gathering Data. To begin any run chart, some type of process or operationmust be available to take measurements for analysis. Measurements must

be taken over a period of time. The data must be collected in a chronologicalor sequential form. You may start at any point and end at any point. For bestresults, at least 25 or more samples must be taken in order to get anaccurate run chart.

Organising Data. Once the data has been placed in chronological orsequential form, it must be divided into two sets of values x and y. Thevalues for x represent time and the values for y represent the measurementstaken from the manufacturing process or operation.

Charting Data. Plot the y values versus the x values by hand or bycomputer, using an appropriate scale that will make the points on the graph

visible. Next, draw vertical lines for the x values to separate time intervalssuch as weeks. Draw horizontal lines to show where trends in the process oroperation occur or will occur.

After drawing the horizontal and vertical lines to segment data, interpret thedata and draw any conclusions that will be beneficial to the process oroperation. Some possible outcomes are:

o Trends in the chart

o Cyclical patterns in the data

o Observations from each time interval are consistent


32/35



A.6 Affinity Diagram

An Affinity Diagram is a creative process, used with or by a group, to gather andorganise ideas, opinions, issues, etc.

A.6.1 Usage

Adding structure to a large or complicated issue (example: what are thecentral issues in the development of a particular new product?)

Breaking down a complicated issue into broad categories (example: Whatare the major steps in the completion of a complex project?)

Gaining agreement on an issue or situation (example: How should a newproduct be marketed?)

A.6.2 Steps

State the issue or problem to be explored. Start with a clear statement of theproblem or goal and provide a time limit for the session-usually 45-60minutes is sufficient.

Brainstorm ideas for the issue or problem. Each participant should think ofideas and write them individually on index cards, sticky notes, or have arecorder write them on a flip chart.

Collect the cards or sticky notes, mix them up and spread them out (or stickthem) on a flat surface (e.g., desk or wall). Index cards can easily be securedto a wall with a putty-type adhesive.

Arrange the cards or sticky notes into related groups. For approximately 15minutes allow participants to pick out cards that list related ideas and setthem aside until all cards are grouped.

Create a title or heading for each grouping that best describes the theme ofeach group of cards.

A.7 Nominal Group Technique

A nominal group technique is a structured process that identifies and ranks themajor problems or issues that need addressing.

A.7.1 Usage

Identifying the major strengths of a department/unit/institution (example:making decisions by consensus when selecting problem solutions in abusiness)

Providing each participant with an equal voice (example: defusing adomineering influential employee who tends to control the discussion anddominate the process)

A.7.2 Steps

Request that all participants (usually 5-10 persons) write or say the problemor issue they feel is most important.


33/35


AIM/AEP/S-LEV/0008


Develop a master list of the problems or issues.

Generate and distribute to each participate a form that numbers in noparticular order the problems or issues. Request that each participant rankthe top five problems or issues by assigning a #5 points to their mostimportant perceived problem and #1 points the least important of their topfive.

Tally the results by adding the points for each problem or issue. The problemor issue with the highest number is the most important one for the total team.

Discuss the results and generate a final ranked list for action planning.

A.7.3 Example

Five possible solutions to a problem exist. Six people must decide which solutionshould be attempted first. The solutions are called A, B, C, D, and E. The people are

Bill, Bob, Henry, Peter, Paul, and Mary.

Each of the six people order the potential solutions. The following matrix isdeveloped.

Solution Bill Bob Henry Peter Paul Mary TOTAL

A 1 2 1 4 3 5 16

B 5 5 2 5 5 4 26

C 4 3 3 3 1 2 16

D 2 1 4 1 2 1 11

E 3 4 5 2 4 3 21

We would begin by trying solution B, followed by E.

A.8 Interrelations Digraph

An Interrelations Digraph is a graphical representation of all the factors in acomplicated problem, system, or situation. It is typically used in conjunction with oneof the other quality tools, particularly the affinity diagram. Frequently the headercards from the affinity diagram are used as the starting point for the interrelationsdigraph.

A.8.1 Usage

Identifying key or driver issues from a list of important issues.

Identifying the most important problems for solution when the number ofproblems exceeds the resources available to solve all of them.

Identifying the root cause of existing problems.

Identifying key factors needed to make a decision when there is insufficientinformation available to make a data-driven decision.

A.8.2 Steps

State clearly the issue or problem. Write it on a card and stick it to the centre

of a board.

Determine the factors related to the issue. Frequently these will be theheaders from a previously completed affinity diagram. Place cards containingthese factors in a circle around the issue card.


34/35



Determine if cause-effect relationships exist between any of the cards. If so,draw an arrow from the "cause" card to the "effect" card. Do this for allcause/effect relationships that you can find.

Analyse the relationships. Cards that have most arrows going from themtend to be root causes. Cards that have most arrows going to them are rooteffects.

A.9 Force Field Analysis

The Force Field Analysis is an analysis tool that uses a creative process for forcingagreement about all facets of a desired change.

A.9.1 Usage

Clarifying and strengthening the "driving forces" for change (example: What

things are "driving" us toward process improvement?)

Identifying obstacles or "restraining forces" to change (example: What is"restraining" us from achieving increased sales?)

Encouraging agreement on relative priority of factors on each side of thebalance sheet

A.9.2 Steps

1. Create a flip chart or overhead transparency with a form similar to the following:

2. Discuss and come to agreement with the group (usually 5-7 people) on the current

situationand the goal. Write this in the appropriate space on the form.

3. Brainstorm the "driving" and "restraining" forces. Driving forces are things (actions,skills, equipment, procedures, culture, people, etc.) that help move toward the goal.Restraining forces are things that can inhibit reaching the goal.

4. Sort on common themes and/or prioritise the driving and restraining forces.

5. Discuss action strategies to eliminate the restraining forces and to capitalize on thedriving forces.

A.10 Matrix Diagram

A Matrix Diagram is a tool that is used to systematically organise information thatmust be compared on a variety of characteristics in order to make a comparison,selection or choice.

A.10.1 Usage

Assigning tasks to complete a project.

Make comparisons between competing alternatives that involve multiplecharacteristics.

Prioritise combinations of new and old activities that maximised the numberof total objectives met.


35/35


AIM/AEP/S-LEV/0008

A.10.2 Steps

Determine the factors that are important for making a correct selection orassignment.

Select the type of matrix to be used. L-shaped matrices are used for two-factor comparisons; T-shaped are used for 3-factor comparisons in thatdisplay indirect and direct relationships; Y-shaped are used for 3-factorcomparisons showing direct relationship only.

Select the relationship symbols to be used.

Complete the proper matrix using the appropriate factors and symbols.

Examine the matrix and draw the appropriate conclusion.

End of Document

kpi measurement monitoring and analysis guide

Documents