What is your PMI?A Model for Assessing the Maturity of Performance Management in Organizations

Dr. Mika Aho

mika.aho@rongo.fi

PMA 2012 Conference: From Strategy to Delivery

Fitzwilliam College, University of Cambridge, UK

11.07.2012

Presentation Outline

Rationale for Performance Management

Maturity Models

Background for the study

Performance Management Index (PM-Index, PMI)ComponentsMaturity LevelsKey CapabilitiesExample Scores and Results

Summary

Rationale for Performance Management (PM)

Central part of business management and the mechanisms that support decision-making

It broadly encompasses all those measures that an organization needs to take in order to achieve specific aims and objectives

The range of methods and procedures that an organisation uses to manage its personnel effectively

The actions people take are closely related to the organization’s overall strategy

Effective PM aims to integrate all these procedures, methods and measures into a single, organization-wide management system by leveraging modern ICT

Rationale for Performance Management(from Information and Knowledge point of view)

Organizations need reliable, accurate and timely data; these being the fundamental tools of performance management

Sometimes, the concept of PM means dealing only with the data for certain pre-defined standards and key performance indicators (KPI)

This perception of PM leaves the rest of the data and information in the organization untouched

To implement a really comprehensive PM strategy organizations should leverage their data by adding analytical functions to their traditional measurement and monitoring processes

The use of analytics helps organizations to proactively seek out and identify trends, and to create alternative scenarios for what might happen in the business

Performance Management and Information Technology

Information technology acts as an enabler and integrator to support the overall PM initiative in an organisation

It integrates all of an organisation’s processes, measures, information and intangible assets so that they support the different phases involved in implementing a strategy, e.g.

Visualize and monitor the strategy in a form of dashboards, reports, and analytical tools

Create rolling sales forecasts and use those results in the budgeting process

In mature organizations, the performance data is often stored in an organization-wide data warehouse

PM furnish the decision-makers with all the information they need to make really informed decisions

Background for Maturity Models

There is nothing new in the idea that the evolution of an organization, or any one part of it, can be described by using maturity models

For example the Capability Maturity Model (CMM), have already been in use for several years to demonstrate the evolution of different areas such as

Corporate Governance (Weill & Ross, 2004)

IT and Strategy Alignment (Luftman & Kempaiah, 2007)

Service Oriented Architectures (Perko, 2008)

Performance Management (Aho, 2009; 2011)

Maturity Models

Over the years, organizations have adopted models by which they can improve their own maturity and capability, and that of their procedures

Maturity = development from some initial stage to an improved, target stage

Capability = what an organization can do and is capable of doing

When moving from a stage to another, the target area, or the whole organization, improves in terms of its qualitative and quantitative capabilities over time

The higher the maturity level, the more efficiently the organization is performing

At the higher maturity levels any procedure is controlled, foreseeable and the targets that have been set for it are achieved

At the lower maturity levels, however, there might be considerable variation; the process might be unstable and there are risks associated with it

Maturity Models and Performance Management

PM and maturity models have not been studied much academically, although some attempts had been made in related areas, such as

business intelligence (e.g. Eckerson, 2007; Hostmann, 2006; van Roekel et al., 2009)

analytical competition (Davenport & Harris, 2010)

performance culture (Dresner, 2010)

information technology growth (Nolan, 1979)

IT-business alignment (Luftman & Kempaiah, 2007)

There are some drawbacks in the existing maturity models

They are very seldom based on empirical evidence

There are few indications as to why certain components and maturity levels were selected

They only concentrate on particular aspects of PM, such as business intelligence or analytical capability, but fail to examine the concept as a whole

How to create a Maturity Model for Performance Management?

In order to create a maturity model for performance management

1. One first needs to identify the components that comprise PM

2. Secondly, to define a set of levels that denotes the evolution of PM in organisations

3. Thirdly, to identify the key capabilities for an each component and maturity level combination

4. And finally, to build a metric that can be used to assess the PM maturity, and to present the results in a convenient and understandable way

1

2

3

4

”A Construct for Performance Management Maturity Assessment”

The study was conducted during 2009-2011

A Constructive Research Approach

Empirical evidence was collected from five large Finnish manufacturing companies

Methods: interviews, questionnaire, participant observation, literature study

Doctoral dissertation published in November 2011

Tampere University of Technology, Faculty of Business and Technology Management

Maturity levels

Overview of the model (PMI, PM-Index, Performance Management Index)

Questions

Calculation engine

Maturity Model Metric

User interface

Web-based toolMain components

Supporting components

Sub components

The focus of this paper

Main components

Information

Performance

Strategy and business

Technology

Business value

Decision support

Strategy and objectives

Data and information analysis

Performance management processes

Measurement and monitoring

Common structures and master data

Data and information quality

Information production, distribution and availability

Data warehouse architecture

Business and technology alignment

Application architecture

Sta

keho

lder

s

Org

aniz

ation

al c

ultu

re

Com

pete

ncie

s

Com

mun

icati

on

Man

agem

ent a

nd R

espo

nsib

ility

Methods and Tools

Scale and Scope

Inta

ngib

le

asse

ts

Sub components

Information

Performance

Strategy and business

Technology

Business value

Decision support

Strategy and objectives

Data and information analysis

Performance management processes

Measurement and monitoring

Common structures and master data

Data and information quality

Information production, distribution and availability

Data warehouse architecture

Business and technology alignment

Application architecture

Sta

keho

lder

s

Org

aniz

ation

al c

ultu

re

Com

pete

ncie

s

Com

mun

icati

on

Man

agem

ent a

nd R

espo

nsib

ility

Methods and Tools

Scale and Scope

Inta

ngib

le

asse

ts

Supporting components

Information

Performance

Strategy and business

Technology

Business value

Decision support

Strategy and objectives

Data and information analysis

Performance management processes

Measurement and monitoring

Common structures and master data

Data and information quality

Information production, distribution and availability

Data warehouse architecture

Business and technology alignment

Application architecture

Sta

keho

lder

s

Org

aniz

ation

al c

ultu

re

Com

pete

ncie

s

Com

mun

icati

on

Man

agem

ent a

nd R

espo

nsib

ility

Methods and Tools

Scale and Scope

Inta

ngib

le

asse

ts

Maturity Levels

In CMM PMI Maturity Level PMI Maturity Level Description5 – Optimized Level 5

Strategic toolPerformance management has become a strategic tool for management, and it is a central part of the organization’s control and management system. Performance management is also strongly connected to the different phases of strategy implementation in the organization.

4 – Managed Level 4Analytical business

Business becomes more analytical in key business areas. The metrics and scorecards are closely aligned to the organization’s strategy.

3 – Defined Level 3Fact-based decision-making

Decisions are more often made based on facts, rather than management instinct. The organization’s data is stored in a centralized data warehouse.

2 – Repeatable Level 2Understanding the value

The organization understands the value of performance management for its business. Organizational goals and objectives are defined.

1 – Initial Level 1Information silos

The solutions are local; there are no common standards, no shared resources or management. As a result the management does not get a clear and consistent picture of the organization as a whole.

Maturity Levels

In CMM PMI Maturity Level PMI Maturity Level Description5 – Optimized Level 5

Strategic toolPerformance management has become a strategic tool for management, and it is a central part of the organization’s control and management system. Performance management is also strongly connected to the different phases of strategy implementation in the organization.

4 – Managed Level 4Analytical business

Business becomes more analytical in key business areas. The metrics and scorecards are closely aligned to the organization’s strategy.

3 – Defined Level 3Fact-based decision-making

Decisions are more often made based on facts, rather than management instinct. The organization’s data is stored in a centralized data warehouse.

2 – Repeatable Level 2Understanding the value

The organization understands the value of performance management for its business. Organizational goals and objectives are defined.

1 – Initial Level 1Information silos

The solutions are local; there are no common standards, no shared resources or management. As a result the management does not get a clear and consistent picture of the organization as a whole.

Create understanding

Build foundation

Leverage data and information

Align with strategy process

Key Capabilities

LEVEL 1 Information silos

• Poor quality data• No defined measures• Temporary ad-hoc

solutions• Solutions made for

single business cases• Reports built on

operational information systems

LEVEL 2Understanding the

value

• Operations and development plan

• Data marts• Budgeting• Creation of strategic

goals• Financially focused

solution• Interactive reporting

systems• Formal PM method in

use

LEVEL 3Fact-based

decison-making

• Top management support

• Management dashboards

• Common business vocabulary and master data

• Strategy execution and monitoring

• Planning and forecasting

• Key Performance Indicators

• Data warehouses

LEVEL 4Analytical business

• Competence centre• Balanced metrics• Individual and external

stakeholder metrics• Strategy planning and

analysis• Rolling forecasts• Enterprise data

warehouse (EDW)

LEVEL 5Strategic tool

• High quality data• Strategy feedback and

refinement• Continuous strategic

planning• Strategy scorecard• External data sources• Integration between

architectures and processes

• Service-oriented architecture (SOA)

For a complete matrix, please visit http://pm-index.com/matrix/

3,22 3,53

2,61 2,91 3,07

0,0

1,0

2,0

3,0

4,0

5,0

Case 1 Case 2 Case 3 Case 4 Case 5

case companies

Descriptions for each maturity level and component combination

” At the third maturity level a common organizational culture starts to evolve. The management, in particular, begins to appreciate the benefits to be gained from performance management. People in the organization start to adapt fact-based decision-making, and their decisions and actions are based on their understanding of performance management systems. People are starting to understand better the factors that have an impact on the business.

The performance management initiative gets support from top management. The organization begins to form a performance management competence centre where specialists from both business and IT are joined together to meet the needs of the end users.

The organization has a common set of standards and vocabulary. The shared terms and metrics are standardized across the organization. From the technology point of view, one characteristic of this level is that there are data warehouses that have more actionable and better quality data… ”

Structured results

Structured results

Structured results

Structured results

Questionnaireresults

Questionnaireresults

Questionnaireresults

Questionnaireresults

Summary by Case company

Summary by Case company

Summary by Case company

Summary by Case company

Structured interview

(1)

Transcribed interview

Structured interview

(2)

Summary by Case company

Field journal

Other sources

Literature study

Analysis of the empirical evidenceStructured

interview (2)

Structured results

Questionnaireresults

Process for identifying the components

Interview (15)

Participant observation(~125 days)

Project plans, minutesof meetings, internal documentation etc.

Questionnaireand interviews

Process for identifying the components

Literature

Empiricalevidence

Methods:• Quantifying• Theming• Typing• Content analysis

Analytical unit (n=475) Grouped analytical units (n=56)

Category

• Timeliness of information helps decision-making

• SAP forces to go towards common vocabulary

• MDM system in place• Master data has accountable• Common business vocabulary

from ERP project• Data quality is challenging• Data producer is responsible

for its validity• …

• Common business vocabulary

• Master Data• Data quality

and validity

Data and information

Process for identifying the components, cont.

Data andInformation

Data input Data quality and validity

Information

Information distribution

and availability

Information production

Sourcesystem

Infrastructure

Process for identifying the maturity levels and the key capabilities

E.g. Brudan, 2009; Balanced Scorecard Institute, 2011; Moncla, 2004; Hostmann, 2007; Luftman & Kempaiah, 2007; Eckerson, 2007; Hagerty, 2006; Davenport & Harris, 2007; Davenport & Harris, 2010; Wettstein & Kueng, 2002; Curtis et al., 2009

Empirical evidence

N=374

N=385

N=759

Literature and existing maturity models in the area

Level 1 Level 2 Level 3 Level 4 …Technology Application

architectureSpreadmarts(Eckerson, 2007a)

Traditional back-office applications(Luftman & Kempaiah, 2007)

Spreadsheetapplications arefrequently used(Hostmann, 2007)

Data Marts(Eckerson, 2007a)

Applications aresiloed(Hostmann, 2007)

DataWarehouses(Eckerson, 2007a)

Hybridtechnologies(Hostmann, 2007)

Improvedreportingsystems (Brudan, 2009)

Organization-widedata warehouse(Eckerson, 2007a)

Focus on applications thatimprove business process for a long-term benefit(Luftman & Kempaiah, 2007)

…

… … … … …

Validating the Model

Building the maturity model

Building the metric Questionnaire Analysis

Maturity model validation Metric validation

Group interviews

Construct validation

PrestudyLiterature

ObservationsInterviews

Changes to the maturity model

Metric calibration

Maturity model validation• No statistical methods were used• Construct validity increased when the construct was

incrementally developed• The components and their suitability were examined based

on literature, direct observations and interviews• The selection of the components are justified well

Metric validation• Definition of the 283 statements• Questionnaire was sent to key employees in the case-

companies in four parts• Analysis of the data, selection of proper calculation model• Interview with the case companies• Later the index was further refined and calibrated

Validating the Construct, cont.The dimensions of a weak market test

One person Team or department

Strategicbusiness

unit

Divisionor country

Entireorganization

Regular usereplacing old

system(s)

Regular use in parallel with old

system(s)

Ad hocusage

Used once

The intensityof usage

The extent of usage

Weakmarket test

passed

Weakmarket testnot passed

Rejected before considering implementation

Rejected after considering implementation

Reject after unsuccessful implementation trial

Tried once but not actually used

Case companies

Case companiesLogica

Rongo

(Labro & Tuomela, 2003)

A web-based tool for assessing the PM maturity and to present the results

A web-based tool for assessing the PM maturity and to present the results

Statement 1 2 3 4 5 0

Performance Management solution supports strategy monitoring x

The performance metrics used in our organization are in balance x

Performance data is used to devise alternative scenarios x

The technological architecture and infrastructure are flexible x

Data and information are high quality x

Data quality problems are openly communicated in our organization x

Top management supports performance management initiative x

The owner of Performance Management comes from the business, not IT x

Through scorecards, the employees participate to strategy execution every day x

Example scores and results

1

2

3

4

5Strategy and business

Performance

Intangible assets

Information

TechnologyCommunication

Methods and tools

Scale and scope

Management andresponsibility 3,0

3,84,1

0

1

2

3

4

5

Business value Strategy and objectives Decision support

Strategy and business

2,53,0

3,43,7 3,8 3,9 4,1

0

1

2

3

4

5

Managementand

responsibility

Business value Methods andtools

Scale and scope Strategy andobjectives

Communication Decisionsupport

Strategy and business

Example scores and results

1

2

3

4

5Strategy and business

Performance

Intangible assets

Information

TechnologyCommunication

Methods and tools

Scale and scope

Management andresponsibility

1,7

3,2 3,4 3,7 3,9

0

1

2

3

4

5

Technology Information Intangibleassets

Performance Strategy andbusiness

Communication

Communication

Summary

The study

presents a novel model for assessing the maturity of PM

further extend PM research by providing a deeper understanding of the underlying components of PM

attempts to strengthen the existing theoretical framework in this field

increases the understanding of what is a relatively new area of PM research

In the model

the identified nine components are used to describe the key areas of PM

the five maturity levels identify the key characteristics of each development stage

In contrast to existing models, in which PM is usually seen only in conjunction with strategic management, this model examines PM particularly from an information and knowledge management point of view

For more information, please visitwww.pm-index.com

What is Your PMI?http://your.pm-index.com

(available in the autumn 2012)

Good at Finnish? Check out my dissertationhttp://bit.ly/PMDissertation