experiences with indicators
DESCRIPTION
Presentation for the European SEPG conference in 2003.TRANSCRIPT
European SEPG
Experiences with IndicatorsThe Measurement Front-End
Robert van Lieshout
June 2003
About ImtechImtech
14.000 employees€ 2 billion turnover
Information TechnologyCommunication TechnologyIndustrial AutomationSystem TechnologyInstallation Technology
Agenda
What is an Indicator?Analyzing the measurement processPoor IndicatorsCriteria for Good IndicatorsConclusion
What is an indicator?
A measure or combination of measures that provides insight into a software issue or concept.
Indicators are often comparisons§ E.g. Planned versus Actual
Indicators are generally presented as graphs or tables.
Based on Practical Software and Systems Measurement: A Foundation for Objective Project Management
http://www.psmsc.com/
Analyzing the process
Indicators are an important aid in analyzing the measurement process.§ They are easily accessible§ They provide clues to the strengths and weaknesses of
the measurement process§ They are the interface between information provider and
information user
Indicators are an excellent starting point.
UnderstandabilityTest Progress Chart
0
100
200
300
400
500
600
700
26.1
27.1
28.1
29.1
30.1
31.1
32.1
Ho
urs
Planned
Target
Fini, NA
Fini, NOK
Fini, OK
Understandability
How easy is it to understand the indicator?
Understandability can be improved through:§ Clear definitions§ Good legends and labels§ Consistency among different indicators§ Less clutter
Confidence in Result
Our Product Competitor 1 Competitor 2Feature A Good Good FairFeature B Good Poor GoodFeature C Fair Poor FairFeature D Good Good Poor
Product Comparison
Company X
Confidence in Result
Do users have confidence in the indicator?§ Is the indicator unbiased?§ Are the users involved in the process?
Confidence can be improved through:§ Better understanding of the indicator§ Better understanding of its limitations§ Knowing how it is created (measurement process)
Fitness for PurposeEUROPE Current Tasks
0
2
4
6
8
10
12
14
16
11-Jan 12-Jan
Pro
cess
tim
e [d
ays]
Report, In Process
Report, Wait
Processing, In Process
Processing, Wait
Check, In Process
Check, Wait
Expose, In Process
Expose, Wait
Transport, In Process
Transport, Wait
Provide, In Process
Provide, Wait
Request, In Process
Request, Wait
Fitness for Purpose
Does the measure meet information needs?§ To what extent does the indicator measure what it is
supposed to?§ For predictive indicators: to what extent does it
demonstrate the ability to forecast?
Fitness for Purpose can be improved through:§ Choosing data that is easier to collect§ Use of proven indicators§ Improving understandability
Accuracy & ReliabilityReview Rate versus Document Size
0
5
10
15
20
25
30
35
40
45
50
0 10 20 30 40 50 60 70
Document Size (Effective Pages)
Rev
iew
Rat
e (E
ffec
tive
pag
es p
er h
ou
r)
Accuracy
Does the indicator conform to its definition?Accuracy can be improved through:§ Well defined measures§ Clearly communicating those definitions§ Reducing the amount of missing data§ Reducing the number of missed measurement
opportunities§ Reducing the frequency of finding inconsistent data
Reliability
How consistent are the results?§ What happens if the measurement is repeated?§ What happens if someone else tries to reproduce it?
Reliability can be improved through:§ Well defined measures § A well defined measurement process§ Using fewer subjective measurement methods
Model AssumptionsProject Effort versus Product Size
0
50
100
150
200
250
300
5 10 15 20 25 30 35 40 45 50
Product size (KLoc)
Pro
ject
Eff
ort
(Man
days
)
Model Assumptions
Have the model assumptions been satisfied?§ E.g. data distributions, sample size, scales§ Models often rely on assumptions about input data
Satisfaction of model assumptions can be improved through:§ Using simple techniques in preference to sophisticated
techniques§ Matching models with measurement context
Amount of UseQuality Performance to Agreed Reject Level
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Janu
ary
Febru
aryMarc
hApri
lMay
June Ju
ly
Augu
st
Septe
mber
Octobe
r
Novem
ber
Decem
ber
OK Not OK KPI to be calculated Target
Amount of Use
To which extent is the indicator used?§ Is it used as a basis for decision making?§ Does anyone complain if it is not available?
Many of the previous criteria have an influence on the amount of use§ E.g. Poor understandability results in less use
Good indicators
A good indicator is:§ Based on well defined measures§ Produced using a well defined process§ Used for decisions, predictions or conclusions§ Clear and easy to interpret
ConclusionAn indicator is:
§ Part of a measurement process used for decision making§ A good starting point for analysis
Characteristics of Good Indicators:§ Used regularly § Users have confidence in their result§ Fit for purpose§ Easily understandable§ Satisfy any model assumptions§ Accurate and reliable
By evaluating these characteristics, poor indicators and the processes that produce them can be improved.
