estimator_featured article ii

7/31/2019 Estimator_Featured Article II

1/22

Featured Article II

A Roadmap for Balanced Productivity Metrics: Are We There Yet?

By Jim Mayes (IT Metrics Strategies, August 2000)

The Balanced Approach

Philosophy

IT outsourcing continues to be a popular course of action for streamlining

businesses. These arrangements always include measurements of

performance and productivity improvement. The questions that we should be

asking are:

Why isnt it just as important to measure performance and productivity

related to in-house IT?

Why dont we develop in-sourcing contracts, as an alternative to out-

sourcing?

Whether IT is inhouse or outsourced, performance and productivity are

important.

You know how the kids in the back seat are always asking: Are we there yet?

Most companies make outsourcing decisions without even knowing their

current IT capability and are often surprised when they find out that they

were better off prior to outsourcing. Similarly, many companies embark on

software process improvement programs without knowing their existing level

of performance and without any metrics in place to measure productivity

improvement. Its like the old saying If you dont know where you are, a map

wont help. This applies to measuring progress as well. It is very difficult to

know if were there yet, if we dont know where we started and exactly

where we still need to go.


2/22

Definition of BPM

Balanced Productivity Metrics (BPM) incorporates the use of both quantitative

and qualitative data for measuring performance and productivityimprovement. The primary goals of a Software Engineering Institute (SEI)

Capability Maturity Model (CMM) based process improvement program are to

improve productivity, improve quality, and reduce risk via consistent

processes. BPM focuses on the SEI CMM core measures (size, time, effort, and

defects), as well as other data collected to measure process improvement. A

multi-dimensional approach is needed to measure productivity improvement

in a way that also provides an understanding of the environmental factors

(and other significant factors) that influence project productivity.

The BPM approach is used to measure productivity with regard to software

development and maintenance. BPM is based on the principle that the

management of productivity improvement should focus on achieving a

balance of time (schedule), cost (effort), and quality (defect rate) [TCQ] 1

improvement. It is consistent with a balanced scorecard philosophy, since

these fundamental metrics components cannot be observed independently

with regard to providing a valid productivity assessment. Balancing TCQ is

much like filling a balloon. If it is filled or compressed too much, it bursts. If it

doesnt burst and it is compressed in one area, it expands in another.Likewise, if a projects schedule (time) is compressed too much, the project

will either fail or the project effort (cost), project related defects and

production defects (quality) will be disproportionably increased, which

increases maintenance effort (cost). The reverse is also true. Project

schedules can be increased too much, which does reduce effort and defects,

but may not provide the desired business value. Therefore, a balanced

measurement approach is needed.

To quote Karl Wiegers:

A risk with any metrics activity is dysfunctional measurement, in which

participants alter their behavior to optimize something that is being measured,

rather than focusing on the real organizational goal. For example, if we are


3/22

measuring output productivity but not quality, expect some developers to

change their coding style to expand the volume of material they produce, or to

code quickly without regard for bugs. I can write code very fast if it doesnt

actually have to run correctly. The balanced set of measurements helps

prevent dysfunctional behavior by monitoring the groups performance in

several complementary aspects of their work that lead to project success.2

The two primary BPM reporting components for providing a balanced set of

metrics, are the:

Productivity Metrics Report (PMR), and the

Process Performance Report (PPR).

PMR Quantitative Measures

The quantitative measures associated with the Productivity Metrics Report

(PMR) are used to assess productivity improvement. The initial assessment

period is considered the baseline year. Improvement is normally expected at

the end of the second year over the baseline year. Every year thereafter,improvement is expected over the previous year.

The PMR uses a combination of metrics for calculating the BPM Score, which

measures success in achieving productivity improvement goals. The project

metrics are calculated for each software enhancement and new development

project, and then averaged for each productivity metrics component. The

maintenance metric is calculated with regard to the yearly hours required to

provide ongoing support for the total inventory of supported applications. Themetrics components to be included in calculating the productivity score are

shown in Table 1.

Metrics Component Calculation


4/22

Project Schedule Duration = Total Project Months / Total Projects Units of

Product Measurement (UPM) = Months per UPM

(Function Points, Lines of Code, etc.)

Project Output Productivity = Total Project Hours / Total Projects Unit of

Measure

= Effort Hours per UPM (Could also calculate as FP per

hour, etc.)

Project Quality = Mean Time To Defect (MTTD), for each project is

calculated by dividing the number of days exposure

during the first 30 days after deployment by thenumber of defects found

= SUM (Each Projects MTTD)/ Total Number of

Projects

= Weighted Average MTTD

Maintenance Output Productivity = Total Maintenance Hours / Total

Maintenance UPM

= Effort Hours per UPM

Table 1 BPM Components and Calculations

PPR Qualitative Measures

The qualitative measures associated with the Process Performance Report

(PPR), when analyzed along with the quantitative measures, provide insightinto the overall process productivity and direction toward achieving process

improvement goals. Looking at the BPM Score which is a letter grade or

number only provides the current status or contractual measure. Unless

there is an underlying analysis behind that score, there is little information to

use for identifying the next steps. The PPR provides information that can be

used to identify problems and areas that have improved, for managing


5/22

process productivity improvement activities. This report provides the

following analyses, trending, and comparisons to previous period:

Demographic data analysis related to organization profiles, project

types, size, complexity, and languages

Statistical analysis of metrics categories against previous performance

trends

Analysis of delivery performance, including requirements volatility

Significant factors impacting project results

Analysis of customer satisfaction survey results in relation to project

data

Analysis of environmental factors related to tools/methodology,technical difficulty, and personnel

Process Overview

Administration

The BPM process involves joint responsibilities. The client performancemetrics manager (client PMM) manages the process and the database and

serves as team lead for a productivity metrics team. The client PMM will

prepare the PPR and the IT metrics group will prepare the PMR, which includes

calculating the BPM Score. Change management with regard to this process

will be the responsibility of the productivity metrics team. (These roles may

vary depending on organizational structure. Additional roles are identified in

Table 2.)

Frequency

The PMR should be calculated annually for productivity scoring related to

annual software process improvement goals, or contractual agreements in the

case of outsourcing. For purposes of monitoring progress and enabling the

parties to promptly address any service or process deficiencies, quarterly BPM

Reports containing cumulative data should be created and reviewed.


6/22

If, at any time the BPM Score is to be calculated, and one or more metrics or

component metrics cannot be calculated, either because the relevant data is

not available, or the applicable scoring scale has not been adopted or is not

yet in effect, then the BPM Score should be calculated disregarding such

metrics. The weights of the other metrics or component metrics within a given

metric shall be increased proportionately such that other metrics or

component metrics maintain their respective relative weights.

Data Collection and Dictionary

Data consistency is the key to having a good decision support database;

therefore, to insure consistency, the definitions must be documented. Also,

when building an internal historical project repository, the project data points

must be categorized. This will provide stratification such that apples-to-apples

project comparisons can be made based upon project type, environment, user

organization, development organization, language, etc. The data collectionrequirements should also be defined and documented. The collection of

software project data during the project is designed to track the status of

projects, enable predictive analysis, and facilitate the collection of project

closeout data required for the BPM reports. Upon completion of the project,

the data should be summarized in a detailed data collection form.

Tools/Database

BPM project data should be stored in a metrics database. The project-level

information in this database can be used for planning new projects and

evaluating risks. At a minimum, this can be done by using data from similar

historical projects for providing an analogy to the project being planned. This

is associated with the CMM Level 2 Software Project Planning key process area

and supports the software project estimation and risk management processes.

(For more information on this, see my December 2000 ITMS article, Saving

the World One Project at a Time: Planning by the Numbers.)8

Process Steps

The BPM process-flow schematic is illustrated in Figure 1. The BPM schematic

description and responsibilities are provided in Table 2. The BPM process

begins with collecting baseline project data and the development of a


7/22

productivity baseline. The schematic illustrates the next steps after the

baseline has been established.


8/22

Figure 1 BPM Process Flow Schematic

Balanced Productivity Metrics Process- Schematic Description

Activity Performed By1. Software Enhancement or NewDevelopment Project LaunchedWhenever new projects are launched,it must be determined if they meetthe criteria for being included in theproductivity metrics process. Theremay be some projects that werelaunched prior to the implementation

of this process that should be includedas well. Also, projects should beincluded that were not initially inscope, but changed such that theysubsequently fit the criteria.

2. Track Project Phase Effort,Schedule, and Defects The actualeffort hours, start date, and end datefor each project phase should betracked. The project phases include

planning, requirements analysis, andmain build (detailed design deployment). Defects are tracked byseverity (1critical, 2 serious, 3moderate, and 4 tolerable/cosmetic)from the beginning of system testuntil deployment. Also, defects mustbe tracked by severity, for the first 30days after deployment.

Activity

Performed By

IT project manager


9/22

2. Track Project Phase Effort, Schedule, and Defects

The actual effort hours, start date, and end date for each project phase should

be tracked. The project phases include planning, requirements analysis, and

main build (detailed design deployment). Defects are tracked by severity

(1critical, 2 serious, 3 moderate, and 4 tolerable/cosmetic) from the beginning

of system test until deployment. Also, defects must be tracked by severity, for

the first 30 days after deployment.

IT project manager

3. Schedule and Perform Software (UPM) SizingFunction points, source lines of code or some other method should be used to

size the project and application.

IT project manager and technician

4. Enter Project Data into Detailed Data Collection Form

On project completion, project data is entered in the detailed data collection

form or spreadsheet. This form is sent to the IT program manager prior to the

project closeout meeting.

IT project manager

5. Review Project Data

The project information provided in the detailed data collection form should bereviewed and validated during the project closeout meeting. Agreement

should be reached on the ratings and results provided. This can provide an

opportunity for joint problem solving related to project impacts and process

improvements needed. The validated detailed data collection form should

then be forwarded to the Client performance metrics manager.


10/22

Client program manager

6. Enter Detailed Project Data

The data contained in the detailed data collection form is entered into the

metrics database.

Client performance metrics manager

7. Track Ongoing Maintenance Effort Hours Actual effort

hours charged to ongoing maintenance support must be tracked.

IT project manager

8. Schedule and Perform Application (UPM) Baseline Sizing

Function points, source lines of code, or some other method should be used to

size the applications being maintained.

IT project manager and sizing specialist

9. Update Maintenance Effort and Size Spreadsheet

A data repository should be maintained containing maintenance productivity

data.

IT metrics group

10. Prepare Quarterly or Yearly BPM ReportThe PMR containing the BPM score, is created by the IT metrics group, and the

Performance Manager creates the PPR.

IT metrics group and client performance metrics manager


11/22

Table 2 BPM Process Schematic Description

Critical Success Factors

The following factors are critical to the success of a Balanced Productivity

Metrics Program: Client and IT partnering relationship with open

communication paths

Senior management support and a documented policy statement

Client program management, client performance metrics management,

IT project management, and IT metrics group support, and synchronized

processes

Accurate data collection and sizing on a timely basis

Using project-level quantitative and qualitative metrics data for software

project planning and process improvement (i.e., dont just accumulate it for

contractual purposes)

Communication, training, and mentoring related to the BPM process

Scoring

Metrics Component Scores

Each BPM Metric Component will be given a letter score determined in

accordance with the scoring scale shown in Table 3, assuming a 5% yearly

improvement for each component is desired. This letter grade will be

converted to a numerical score using the scale also shown in Table 3. (The

percentages and letter grades can be adjusted based upon the yearlyimprovement desired for each TCQ category.)

Grade


12/22

Numerical Score

Component Grade (calculated for each Metrics Component and then converted

to numerical score)

A

5.0

A for = 5.5% or better (exceeded)

B

4.0

B for = 4.5% to 5.4% (expected)

C

3.0

C for = 0 to 4.4%

D

2.0

D for =


13/22

Metrics Component Weighting and Balanced Scoring

Business value should determine the weightings with regard to Project TCQ

and Maintenance, since this is a business decision as to which component hasthe highest priority, or if they are all equally important. Of course this will

vary from project to project, depending on the circumstances, therefore the

BPM weightings should reflect the overall philosophy for the organization.

Component/Category weightings should be established once per year, at the

beginning of each annual measurement cycle. If everything were equally

weighted, the total BPM composite numerical score would be calculated as

shown it Table 4, and the final composite BPM grade would be calculated as

shown in Table 5. Examples of BPM Score Calculations are provided in Tables

6, 7, and 8.

Component

Weight

Composite Numerical Score Calculation

Project schedule duration

25%

Project schedule numerical score (T3) x 0.25 (T4)

Project output productivity

25%

+ Project output numerical score (T3) x 0.25 (T4)

Project quality


14/22

25%

+ Project quality numerical score (T3) x 0.25 (T4)

Maintenance output productivity

25%

+ Maintenance output numerical score (T3) x 0.25(T4)

Total

100%

= Composite numerical score (convert to grade in T5)

Table 4 (T4) Numerical Component Weighting and Composite Calculation

Grade

Numerical Score

Assessment

A

4.5 5.0

Exceeds standards

B

4.0 4.4

Meets standards


15/22

C

3.0 3.9

Does not meet standards

D

2.0 2.9


F

0.0 1.9


Table 5 (T5) Final BPM Composite Grading

Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade (T3)

Numerical Score (T3) x Weight (T4)

Composite Score and Grade (T5)


16/22

Project Schedule Duration

1.1

1.045

5% better

B

4 x .25

1

Project Output Productivity

3.6

3.42

5% better

B

4 x .25

1

Project Quality

4.5

4.73

5% better

B

4 x .25

1

Maintenance Output Productivity


17/22

1.8

1.71

5% better

B

4 x .25

1

Total

4 = B (meets standards)

Table 6 Example 1: BPM Score Calculation

Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade

Component Score x Weight


18/22

Composite Scoring


1.1

1.045

5% better

B

4 x .25

1


3.6

3.49

3% better

C

3 x .25

0.75

Project Quality

4.5

4.73

5% better

B

4 x .25

1


19/22


1.8

1.71

5% better

B

4 x .25

1

Total

3.75 = C (does not meet standards)


Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade


20/22

Component Score x Weight

Composite Scoring


1.1

1.045

5% better

B

4 x .25

1


3.6

3.39

6% better

A

5 x .25

1.25

Project Quality

4.5

4.73

5% better

B

4 x .25


21/22

1


1.8

1.75

3% better

C

3 x .25

0.75

Total

4 = B (Meets Standards)


Conclusion

The measurement of productivity and performance is just as important for

inhouse IT as it is for an outsourced IT environment. A balanced approach, as

provided by the BPM process, is needed in order to ensure that all aspects of

productivity are assessed and that the correct behavior with regard to

performance is achieved. Some outsourcing vendors will argue that only one

measure should be used, such as effort per unit of output, but to most clients,


22/22

schedule and quality are equally important. Therefore, measures should be in

place such that schedule or quality cannot be sacrificed, and to prevent

dysfunctional measurement behavior from occurring. It is beneficial to IT that

all factors be measured, due to the clients ever changing TCQ priorities. In

addition to balancing the TCQ metrics and quantitatively tracking projects, it is

also important to understand the relationship between the quantitative and

qualitative measures that influence productivity. When we trend the

quantitative and qualitative data and compare the results to previous internal

benchmarks, our corporate knowledge will be increased such that we can

make intelligent business decisions. The BPM process can help us to

understand the factors that influence our productivity, know where we are on

the map, effectively predict where we are headed, and recognize when we get

there.

estimator_featured article ii

Documents