s06 - chiesa et al, 2009
Post on 02-Jun-2018
218 Views
Preview:
TRANSCRIPT
-
8/10/2019 S06 - Chiesa Et Al, 2009
1/32
Performance measurement inR&D: exploring the interplaybetween measurement objectives,dimensions of performance and
contextual factorsVittorio Chiesa1, Federico Frattini2, ValentinaLazzarotti3 and Raffaella Manzini4
1Politecnico di Milano - Department of Management, Economics and Industrial Engineering,Piazza L. da Vinci 32, 20133 Milano, Italy. vittorio.chiesa@polimi.it2Politecnico di Milano - Department of Management, Economics and Industrial Engineering,Piazza L. da Vinci 32, 20133 Milano, Italy. federico.frattini@polimi.it3Universita` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy.
vlazzarotti@liuc.it4Universita` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy.rmanzini@liuc.it
Measuring research and development (R&D) performance has become a fundamental concern
for R&D managers and executives in the last decades. As a result, the issue has been
extensively debated in innovation and R&D management literature. The paper contributes to
this growing body of knowledge, adopting a systemic and contextual perspective to look into
the problem of measuring R&D performance. In particular, it explores the interplay between
measurement objectives, performance dimensions and contextual factors in the design of a
performance measurement system (PMS) for R&D activities. The paper relies on a multiple
case study analysis that involved 15 Italian technology-intensive firms. The results indicate
that firms measure R&D performance with different purposes, i.e. motivate researchers and
engineers, monitor the progress of activities, evaluate the profitability of R&D projects, favour
coordination and communication and stimulate organisational learning. These objectives are
pursued in clusters, and the importance firms attach to each cluster is influenced by the context
(type of R&D, industry belonging, size) in which measurement takes place. Furthermore, a
firms choice to measure R&D performance along a particular perspective (i.e. financial,
customer, business processes or innovation and learning) is influenced by the classes of
objectives (diagnostic, motivational or interactive) that are given higher priority. The
implications of these results for R&D managers and scholars are discussed in the paper.
1. Introduction
M easuring performance and contribution tovalue research and development (R&D)
has become a fundamental concern for R&D
managers and executives in the last de-cades (Kerssen-van Drongelen and Bilderbeek,
1999). Since the 1990s, several phenomena have
R&D Management 39, 5, 2009. r 2009 The Authors. Journal compilation r 2009 Blackwell Publishing Ltd. 2009,4889600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
mailto:vittorio.chiesa@polimi.itmailto:vittorio.chiesa@polimi.itmailto:vittorio.chiesa@polimi.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:rmanzini@liuc.itmailto:rmanzini@liuc.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:vlazzarotti@liuc.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:federico.frattini@polimi.itmailto:vittorio.chiesa@polimi.itmailto:vittorio.chiesa@polimi.itmailto:vittorio.chiesa@polimi.it -
8/10/2019 S06 - Chiesa Et Al, 2009
2/32
encouraged the development and adoption of
specific approaches for assessing the performance
of R&D: increasing turbulent dynamics in com-
petitive arenas, shortened life cycles, globalisa-
tion, reduced time to market, increased R&D
costs and risks (Nevens et al., 1990; Bayus,
1994; Wind and Mahajan, 1997; Wolf, 2006).
Although it is still acknowledged that the
measurement of R&D performance is a challen-
ging task that might also hinder the creative and
innovative capacity of the firm (Brown and Sven-
son, 1998), today the issue is being extensively
debated in the innovation and R&D management
literature and it raises the interest of practitioners
as well (Pappas and Remer, 1985; Brown and
Svenson, 1988; Sivathanu and Srinivasa, 1996;Werner and Souder, 1997; Hauser, 1998; Driva
and Pawar, 1999; Driva et al., 2000; Poh et al.,
2001; Loch and Tapper, 2002; Godener and
Soderquist, 2004; Ojanen and Vuola, 2006).
This paper aims at contributing to this growing
body of knowledge, adopting a systemic and
contextual perspective to look into the problem
of measuring R&D performance. In particular, it
explores the interplay between measurement ob-
jectives, performance dimensions and contextual
factors in the design of a performance measure-
ment system (PMS) for R&D activities. Morespecifically, it aims to understand: (i) which objec-
tives companies pursue when they measure R&D
activities performance and whether they can be
categorized in some ways; (ii) which approaches to
R&D performance measurement are used to pur-
sue different classes of objectives; and (iii) how the
importance attached to different classes of objec-
tives, and the approaches used to pursue them, are
affected by the context in which measurement
takes place. Although the literature on manage-
ment accounting and control has acknowledged
the importance of these topics (e.g., Simons, 2000;
Azzone, 2006), they have not been properly in-vestigated in R&D settings so far. What is more, a
better understanding of these issues would be
highly beneficial for R&D managers. The rich
empirical data presented and discussed in this
paper will provide R&D managers with a number
of insights that represent a valuable starting point
to design a PMS for R&D that is adequate to the
objectives they have in mind, and is appropriate as
well to the context in which their firm operates.
Moreover, some practical suggestions about how
to improve managers satisfaction with the PMS
are discussed in the paper.In order to pursue its objectives, the paper first
develops a reference framework, which is used as a
guide for the subsequent multiple case study
analysis. The empirical investigation involved a
number of Italian firms operating in technology-
intensive industries, for which technological inno-
vation and, therefore, the results of their R&D
efforts, are a major source of competitive advan-
tage. Because of the significant contribution of
R&D to the companys overall success, these firms
will be far more likely to systematically measure
their innovative activities performance. There-
fore, they represent an ideal empirical setting for
investigating the issues we are interested in.
The remainder of the paper is organised as
follows: Section 2 reviews the relevant literature
on R&D performance measurement, whereas
Section 3 describes the reference framework un-derlying the research. Section 4 illustrates the
methodology used for the empirical analysis,
and Section 5 discusses the result of the multiple
case study. Finally, Section 6 concludes and out-
lines some avenues for future research.
2. Literature review
Existing research into the measurement of R&D
performance in industrial firms has investigated
the topic from four different perspectives, asshown in Figure 1.
At a first level, research has basically focused on
the choice of the indicators or metrics that are best
suited to the characteristics of R&D. Brown and
Svenson (1998) find that an effective PMS for
R&D is built around a limited number of indica-
tors that measure results rather than behaviour,
and privileges objective and external metrics to
subjective and internal ones. Nixon (1998) ad-
vances that performance indicators for R&D
should have a strategic orientation and reflect the
firms critical success factors, they should be simple,
able to encourage change and to balance financialand non-financial perspectives. Werner and Souder
(1997) state that the most effective measurement
approaches for R&D are those that balance both
quantitative and qualitative metrics, as also under-
lined by Pawar and Driva (1999) and Bremser and
Barsky (2004). Hauser (1998) shows that the choice
of the most appropriate metrics should be based on
the type of R&D, whether it is applied research,
core technological development or basic research.
Other scholars in this stream of research investigate
the opportunity to use financial indicators in R&D
departments (Rockness and Shields, 1988) and tobuild a synthetic indicator of R&D productivity or
efficiency (Tipping et al., 1995).
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 489
-
8/10/2019 S06 - Chiesa Et Al, 2009
3/32
At a second level, research has looked into the
choice of the performance dimensions, or perspec-
tives, along which the measurement of R&D
should be undertaken. Pawar and Driva (1999)
advance that the measurement of R&D perfor-
mance needs to be articulated into time, costs,
quality and flexibility dimensions. Kim and Oh
(2002) identify the following types of R&D per-
formance variables: market-oriented, R&D pro-ject-specific and R&D researcher-specific. Davila
(2000) analyses the use of cost, time and customer
(or market) information in the measurement of
new product development performance. Many
scholars in this stream of research have attempted
to apply the Balanced Scorecard (BSC) approach
(Kaplan and Norton, 1992) to R&D. Kerssen-van
Drongelen and Cook (1997), e.g., show how to
develop a measurement approach for R&D per-
formance that, integrating financial, client, inter-
nal business, innovation and learning perspectives,
allows to implement the firms R&D and compe-
titive strategy. Bremser and Barsky (2004) illus-trate how the BSC approach should be integrated
with the stage-gate system (Cooper, 1993) for the
organisation of innovation development activities.
At a third level, research has adopted a sys-
temic perspective in the study of R&D perfor-
mance measurement, assuming the whole PMS
for R&D as the unit of analysis. Kerssen-van
Drongelen et al. (2000), e.g., conceive the PMS
for R&D as comprising the following elements:
metrics organised into a consistent structure,
standards to measure performance against, fre-
quency and timing of measurement and formatfor information reporting. Similarly, Ojanen and
Vuola (2006) suggest that an effective PMS for
R&D should be an internally consistent set of
measurement perspectives, objectives, control ob-
jects and measurement process. In other words,
adopting a systemic perspective means looking at
R&D performance measurement in terms of a
system, which should be made of a set on
integrated and internally consistent elements
(Chiesa and Frattini, 2007), i.e. PMS objectives,
performance dimensions, metrics or indicators,control objects and measurement process.
Finally, a more strategy-oriented stream of
research has adopted a contextual perspective,
emphasising that a PMS for R&D should be
studied within the context in which it is used,
which is both internal and external to the firm.
This body of literature is consistent with the
largest part of the extant management accounting
and control research (e.g., Gordon and Miller,
1976; Gordon and Narayanan, 1984) and basi-
cally reminds that the PMS is used in a specific
R&D setting, being basic and applied research or
NPD (Pappas and Remer, 1985; Chiesa andFrattini, 2007), with a given amount and quality
of available resources (Godener and Soderquist,
2004), within the scope of a firms specific busi-
ness strategy, mission, values and management
style (Nixon, 1998; Kim and Oh, 2002; Loch and
Tapper, 2002) and, finally, in a broader competi-
tive, economic, social, cultural and political con-
text (Loch et al., 1996; Pillai et al., 2001).
3. Reference framework
Investigating the interplay between measurement
objectives, performance dimensions and contextual
Indicators andmetrics for R&D
performancemeasurement
Dimensions for R&D performancemeasurement
Performance MeasurementSystems for R&D
Performance MeasurementSystems for R&D within the firms internal
and external context
Figure1. Taxonomy of research streams on performance measurement in R&D.
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
490 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
4/32
factors in the design of a PMS for R&D, the paper
contributes to the fourth level of analysis described
in the last section. Although the literature adopting
a contextual perspective has studied the impact of
several endogenous and exogenous variables on
the design of specific components of the R&D
PMS, an integrated view on how the objectives for
which the PMS is introduced affect the design of its
constitutive elements, and how they are in turn
influenced by the measurement context, has not
been advanced yet.
A company that seeks to measure the perfor-
mance of its R&D can do this with many different
purposes. According to Kerssen-van Drongelen
and Cook (1997), there are two main classes of
underlying reasons for R&D performance mea-surement, i.e. to motivate scientists and research-
ers and to diagnose activities and processes. Loch
and Tapper (2002) identify the following foremost
objectives for which firms control their R&D
performance: align behaviour and set up priori-
ties, evaluate and reward researchers, establish an
operative control and stimulate learning and im-
provement. Kerssen-van Drongelen et al. (2000)
add that performance measurement, especially in
complex new product development projects, can
serve the purpose of favouring communication
and coordination among top managers, middlemanagers and researchers.
Management accounting and control research
has repeatedly shown that the set of objectives for
which a firm measures its business performance
represents a driving force that heavily affects the
design of the other PMS constitutive elements,
e.g., dimensions of performance, indicators, con-
trol objects and measurement process (Neely,
1999; Bititici et al., 2000; Tuomela, 2005). Most
of all, Robert Simons (1994, 1995) explains that
companies pursue different objectives through
different control systems (called levers of con-
trol). Specifically, belief systems are used tocommunicate and reinforce the firms value and
the paths to be followed to identify and exploit
value creation opportunities; boundaries systems
are needed to encourage individual creativity,
within well-defined frontiers; diagnostic control
systems serve the purpose of coordinating and
motivating the implementation of the firms strat-
egy; and interactive control systems are used to
stimulate organisational learning, communication
and the emergence of ideas related to new business
opportunities. Each of these management control
systems has specific characteristics in terms ofperformance dimensions and perspectives of ana-
lysis, measurement frequencies and standards.
In our research, we wanted to understand
whether these general concepts (in particular,
the influence of measurement objectives over the
characteristics of the PMS) hold true in R&D
settings as well. Moreover, we were interested in
the role of the internal and external contextual
factors. In other words, considering that the PMS
for R&D is embedded in the firms internal and
external context, we wanted to understand
whether and how these factors are able to affect
both the importance the firm attaches to different
classes of objectives and the design of its consti-
tutive elements. This assumption is consistent
with the largest part of extant management con-
trol research (Chenhall and Morris, 1986).
Considering that the number of relevant con-textual factors is potentially countless and that a
PMS for R&D is comprised of a high number of
interrelated parts, we decided to confer a more
specific and definite scope to our research. As far
as the characteristics of the PMS are concerned,
we focused on the following elements: (i) the
dimensions along which R&D performance is
assessed and (ii) the indicators (or metrics) that
are used to measure performance along the above-
mentioned dimensions (see, e.g., Chiesa and Frat-
tini, 2007). As far as the context in which mea-
surement takes place is concerned, we focused onthe following factors that have been identified by
extant research as influential over the design of the
PMS for R&D (Pappas and Remer, 1985; Ker-
ssen-van Drongelen and Bilderbeek, 1999; Davila,
2000; Bremser and Barsky, 2004): (i) the type of
R&D activity that is measured (being it basic and
applied research or new product development); (ii)
the size of the firm and its R&D unit; and (iii) the
industrial sector the firm belongs to. The reference
framework that served as a guide for our empirical
analysis is summarised in Figure 2. It hypothesises
PMS
OBJECTIVES
PMS
CHARACTERISTICS:
-Performance dimensions
-Indicators
MEASUREMENT
CONTEXT:
-Type of R&D activity
-Size of the firm and the R&D unit
-Firms sector of activity
Figure 2. The reference framework.
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 491
-
8/10/2019 S06 - Chiesa Et Al, 2009
5/32
that the objectives for which a firm adopts the
PMS influence the design of its constitutive ele-
ments. Furthermore, it advances that the context
in which measurement takes place has a potential
effect over the choice of the PMS objectives as well
as the design of its constitutive elements. The
empirical analysis will help shed light on the
strength and nature of these hypothesised rela-
tionships.
4. Research methodology
We decided to use case study research as an
overall methodological approach for our empiri-
cal investigation. As suggested by a number ofscholars, this is in fact a very powerful method for
building a rich understanding of complex phe-
nomena (Eisenhardt and Graebner, 2007) that
requires the capability to answer to how and
why questions (Yin, 2003). In particular, we
used a multiple case study approach, which was
chosen because it allows both an in-depth exam-
ination of each case and the identification of
contingency variables that distinguish each case
from the other. Furthermore, multiple case stu-
dies are appropriate when attempting to exter-
nally validate the findings from a single casestudy, through cross-case comparisons (Eisen-
hardt, 1989). Therefore, they typically yield
more robust, generalisable and testable interpre-
tations of a phenomenon than single case study
research (Eisenhardt and Graebner, 2007).
The study involved 15 Italian firms from dif-
ferent industries (e.g., aerospace, pharma-biotech,
pharmaceuticals, machining centres, chemicals)
that were studied during the last 2 years (see
Table 1, where real names have been blinded for
confidentiality reasons). As a unit of analysis for
our case study, we considered the PMS used in the
firms R&D unit. In a number of cases, thestudied company had more than a single organi-
sational unit devoted to R&D activities; unfortu-
nately, in these instances, we had the opportunity
to study only one of them. This impeded us from
undertaking an embedded analysis that could
have provided richer information. We adopt the
largely applied and broad distinction between
Research and Development, including in the for-
mer basic and applied research activities and in
the latter the development of both incremental
and radical new products. We were able to
classify the activities undertaken in the R&Dunits that we considered as either Research and
Development, this indicating a widespread orga-
nisational separation between Research and
Development activities (Chiesa, 1996).
We gathered information basically through
direct interviews; in particular, we followed these
steps:
At the outset of each case, a relationship was
established with a senior manager from the
selected firm. This person was informed about
the research project through a written sum-
mary and a telephone meeting. During this
meeting, we asked the respondent to introduce
ourselves to the head of the firms R&D
function or to another R&D manager who
was responsible for the performance of the
R&D unit and the operation of the PMS; Then we personally interviewed the selected
R&D managers; we undertook two semi-
structured interviews for each respondent
(each interview lasted on average one and a
half hour) in order to gather the information
required to pursue the papers research objec-
tives. Direct interviews followed a semi-struc-
tured replicable guide (see Appendix A),
which comprised a set of open questions for
each of the relevant constructs in our reference
framework (e.g., objectives of the PMS and
dimensions of performance);
Secondary information was collected in theform of company reports and project docu-
mentation. In particular, we gathered and
analysed all the reporting documents that
were generated in support to the functioning
of the PMSs. These informed the researchers
with background information about the se-
lected firms, the type of R&D activity they
undertake and the approaches they use for
measuring R&D performance. Above all,
these secondary information sources were in-
tegrated, in a triangulation process, with data
drawn from the direct interviews, in order toavoid post hoc rationalisation and to ensure
construct validity (Yin, 2003);
All interviews were tape-recorded and tran-
scribed; generally, at this stage a telephone
follow-up with the respondents was conducted
in order to gather some important missing data.
Data and information gathered through the
case studies were manipulated before being ana-
lysed. In particular, we applied the following
techniques (Miles and Huberman, 1984): (i) data
categorisation, which requires the decomposition
and aggregation of data in order to highlightsome characteristics (e.g., objectives pursued
with the PMS or context in which measurement
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
492 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
6/32
takes place) and to facilitate comparisons; (ii)
data contextualisation, which implies the analysis
of contextual factors, not included in the concep-
tual model, that may reveal unforeseen relation-
ships between events and circumstances. Then, a
preliminary within-case analysis was performed;
the purpose was to consider each case study as aseparate one and to systematically document the
variables of interest defined in the reference fra-
mework. Then, explanation-building procedures
were applied so that the relationships between the
PMS objectives, characteristics of the PMS and
context in which measurement takes place were
identified. Finally, a cross-case analysis was un-
dertaken for comparing the patterns that emerged
in each case study in order to arrive at a general
explanation of the observed phenomenon. These
structured procedures for data collection and
analysis, as well as the use of the semi-structuredinterview guide, helped enhance the reliability of
the research (Yin, 2003).
The following section reports and discusses the
empirical evidence we gathered for the 15 cases
included in our sample. It is used to illustrate the
interplay between measurement objectives, per-
formance dimensions and contextual factors in
the design of a PMS for R&D.
5. Results and discussion
The empirical evidence that was gathered for the
cases in our sample is synthesised and mapped
along the dimensions of the reference framework
in Appendix B, where some information about
the difficulties encountered in the adoption and
use of the PMS, as well as the satisfaction of
the firms executives with this tool, is reported.
Table 2 provides a synoptic view of these data to
allow a more straightforward comparison andanalysis.
An in-depth discussion of this empirical evi-
dence is reported in the following paragraphs.
5.1. PMS objectives and measurementcontext
The empirical analysis indicates that firms decide
to measure the performance of their R&D activ-
ities with multiple purposes. In particular, based
on our study of the 15 firms, it is possible to
identify the following list of major objectives a
company might aim at when it comes to measur-
ing R&D performance:
(1) Motivate researchers and engineers and im-
prove their performance in R&D activities;
(2) Monitor the progress of R&D activities with
respect to resource consumption targets, tem-
poral milestones and technical requirements;
(3) Evaluate the profitability of R&D activities
and their contribution to the firms economic
value;(4) Support the selection of the projects to be
initiated, continued or discontinued;
Table 1. The studied firms
Firm Sector of activity No. of
employees
Role of people interviewed
Company A Semiconductors 50,000 R&D projects managerCompany B Electronics for industrial
applications500 Director of R&D and quality manager
Company C Machining centres 160 Director of the technical departmentCompany D Aerospace 1,800 Planning and cost control managerCompany E Pharmaceuticals 500 Director of the development departmentCompany F Pharma-Biotech 60 Chief operating officerCompany G Chemicals 19,300 Director of innovation & technology in plastic
additivesCompany H Aerospace 9,000 Program managerCompany I Pharma-Biotech 700 Director of oncology divisionCompany L Pharmaceuticals 70 General director of research laboratoriesCompany M Household electrical appliances
and home automation
60,000 R&D platform manager
Company N Power generation technologies 2,200 Technology and business development managerCompany O Medical imaging diagnostic 1,000 Vice President for research and developmentCompany P Pharmaceuticals 3,000 Vice President for corporate drug developmentCompany Q Energy conversion 2,600 R&D director
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 493
-
8/10/2019 S06 - Chiesa Et Al, 2009
7/32
Table 2. Synoptic representation of the case study evidence
Firm Type of
R&Dactivity
Size of the
R&D unit(number ofemployees)
Sector PMS objectives Performance perspectives
associated with eachPMS objective
Company A Basic andappliedresearch
700 High-tech (1) Motivate scientistsand engineers(2) Favour coordinationand communication(3) Stimulateorganisational learning
(1) Innovation and learning(23) Business process
Company B New productdevelopment
100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities
(1a) Business process(1b) Customer(2) Financial
Company C New productdevelopment 15 High-tech (1) Motivate scientistsand engineers (1a) Innovation and learning(1b) Business processCompany D New product
development300 High-tech (1) Monitor the progress
of R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication
(1-23) Business process
Company E New productdevelopment
50 Science-based (1) Monitor the progressof R&D activities(2) Motivate scientistsand engineers
(1) Business process(2) Innovation and learning
Company F Basic andapplied
research
60 Science-based (1) Motivate scientistsand engineers
(1a) Innovation and learning(1b) Business process
Company G Basic andappliedresearch
300 High-tech (1) Motivate scientistsand engineers(2) Favour coordinationand communication3) Stimulateorganisational learning
(1a) Innovation and learning(1b) Business process(23) Business process
Company H New productdevelopment
200 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication4) Reduce uncertainty
(1a) Business process(1b) Customer(2a) Business process(2 b) Financial(2c) Customer(34) Business process
Company I Basic andappliedresearch
280 Science-based (1) Motivate scientistsand engineers(2) Select R&D projects(3) Monitor the progressof R&D activities
(1a) Innovation and learning(1b) Business process(23) Business process
Company L Basic andappliedresearch
70 Science-based (1) Motivate scientistsand engineers
(1a) Innovation and learning(1b) Business process
Company M New productdevelopment
200 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication
(4) Stimulateorganisational learning(5) Reduce uncertainty
(12a) Financial(12b) Customer(12c) Business process(345) Business process
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
494 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
8/32
(5) Favour coordination and communication
among the different people and organisa-
tional units taking part in R&D activities;
(6) Reduce the level of uncertainty that sur-
rounds R&D activities; and
(7) Stimulate and support individual and organi-
sational learning.
Moreover, our analysis reveals that these objec-
tives are pursued in clusters by firms. A first group
of companies can be identified that use perfor-
mance measurement in R&D with the main pur-pose of improving the degree of control they
exercise on R&D activities, and to have a support
for taking more effective management decisions.
Examples of these firms are Companies B, D, N,
O and Q, which share the following as main
objectives for R&D performance measurement:
monitor the progress of activities, select R&D
projects and evaluate the profitability of R&D
activities. These are labelled in the paper as
diagnostic objectives, as they correspond to the
reasons underlying the use of diagnostic control
systems in Robert Simonss theory (Simons,1995). A second group of firms can be identified
that use performance measurement mainly as a
tool for motivating scientists and engineers, direct-
ing their efforts toward the long-term innovation
targets of the firm and overcoming the lack of
commitment that R&Ds largely intangible results
often determine. This is the case of Companies A,
C, F, G, I and L. They believe that motivating
researchers and technicians is a pre-requisite for
improving their performance in R&D activities,
and often establish to reward them on the basis of
the performance that are estimated by the PMS
(as it happens, e.g., in Companies F and I, where
performance measurement provides the data ne-cessary to operate a management by objectives
MBO rewarding system). We consider these firms
as mainly interested in pursuing motivational
objectives through R&D performance measure-
ment, which remind us of the reasons for which
managers use belief and boundaries control
systems (Simons, 1995). Finally, our analysis
unravels the existence of a number of companies
for which performance measurement also serves a
second purpose, besides the diagnostic or the
motivational objectives mentioned above. In
particular, they believe it is a very useful meansto improve and streamline the execution of R&D
activities and processes, overcoming some of the
Table 2. (Contd.)
Firm Type of
R&Dactivity
Size of the
R&D unit(number ofemployees)
Sector PMS objectives Performance perspectives
associated with eachPMS objective
Company N New productdevelopment
100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Select R&D projects
(123a) Customer(123b) Financial(123c) Business process
Company O Basic andappliedresearch
100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities
(3) Select R&D projects
(123a) Financial(123b) Customer
Company P Basic andappliedresearch
300 Science-based (1) Favour coordinationand communication(2) Stimulateorganisational learning(3) Motivate scientistsand engineers
(1) Business process(23a) Innovation andlearning(23b) Financial
Company Q New productdevelopment
150 High-tech (1) Monitor the progressof R&D activities(2) Select R&D projects(3) Evaluate theprofitability of R&Dactivities
(1) Business process(23a) Business process(23b) Financial
PMS objectives and performance perspectives are ordered on the basis of the importance attached by each firm.
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 495
-
8/10/2019 S06 - Chiesa Et Al, 2009
9/32
organisational barriers that characterise them. In
particular, performance measurement is also used
to favour coordination and communication, sti-
mulate quick and effective organisational learning
and reduce the level of uncertainty surrounding
critical decisions in R&D. These objectives are
acknowledged as relevant by Companies A, D, G,
H, M and P, and they are called interactive in
this paper, as they closely mirror the use of
interactive control systems described by Simons
(1995). Figure 3 shows the position of the 15 cases
discussed in this paper with respect to the three
clusters of objectives. It is interesting to note that
these results are consistent with the sparse empiri-
cal evidence available in the literature (Kerssen-
van Drongelen and Cook, 1997; Kerssen-vanDrongelen et al., 2000; Loch and Tapper, 2002).
Interestingly, our analysis suggests that the
importance a firm attaches to each cluster of
objectives is influenced by some characteristics
of the context in which measurement takes place.
In particular, diagnostic objectives are predomi-
nant in those firms that decide to measure the
performance of new product development activ-
ities. The cases of Companies B, D, H, M, N and
Q suggest that the need for control is stronger in
NPD than in basic and applied research. The
main reason is that the output of NPD activities issold directly on the market; therefore, the respect
of deadlines, quality requirements and target
costs in these activities has a more direct impact
on the firms market competitiveness than in the
case of basic and applied research, whose clients
are basically internal. Moreover, the amount of
financial and human resources involved in NPD is
very large (especially if compared with basic and
applied research), this making a proper evalua-
tion of R&D profitability, and an accurate prior-
itisation of projects, critical challenges for R&D
managers. Our analysis reveals that the need for
diagnostic control in R&D is also particularly
strong in large R&D units. The larger a firms
R&D unit, the higher the number of different
(and often interrelated) projects that are contem-
porarily undertaken, the higher the number of
researchers and engineers (often belonging to
different departments or functional areas) taking
part in these projects and the larger the amount of
resources devoted to R&D activities. These con-
ditions make the need for a tight diagnostic
control particularly evident, as it is clear forinstance in the case of Companies D and H.
Nevertheless, exerting this type of control over
R&D activities requires that the latter are, at least
to some extent, predictable, that standards to
measure performance against can be easily iden-
tified and that the progression of project activities
along a sequence of stages can be a priori identi-
fied. It is clear that new product development is
more foreseeable than basic and applied research,
but predictability also depends on the character-
istics of the industry in which a firm operates.
Kodama (1995) suggests that it is possible toclassify industrial sectors on the basis of the
probability with which an R&D project is frozen,
which is a measure of the predictability of R&D
activities.1 High-tech industries are characterised
by a freezing rate that decreases throughout the
R&D process, whereas science-based industries
are those in which the freezing rate always
Company A
Company B
Company C
Company D
Company E
Company F
Company GCompany H
DIAGNOSTIC
1) Monitor the progress of activities
2) Evaluate the profitability
of R&D activities
3) Select R&D projects
MOTIVATIONAL
1) Motivate scientists and engineers
INTERACTIVE
1) Favour coordination and communication
2) Stimulate organisational learning
3) Reduce uncertainty
Company N
Company O
Company M
Company L
Company I
Company P
Company Q
Figure3. The studied companies and the emerging clusters of objectives. Within each cluster, objectives are ordered on the basis oftheir relative importance.
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
496 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
10/32
remains high. Our analysis suggests that firms use
performance measurement in NPD with the main
purpose of controlling activities in those indus-
tries (high-tech, in the definition of Kodama)
where the failure rate of projects and activities,
and hence their unpredictability, is smaller. This is
evident if we consider, e.g., that Companies B, D
and H operate in the electronics and aerospace
industries, Company M in the household appli-
ance and home automation sectors and Compa-
nies N and Q in the power generation and energy
conversion industries. More interestingly, Com-
pany O mainly pursues diagnostic objectives
although engaged in basic and applied research,
and this appears to be linked to the lower degree
of uncertainty characterising the industry inwhich it operates.
On the other hand, firms tend to pursue mo-
tivational objectives through R&D performance
measurement in basic and applied research. This
is clear if we consider the cases of Companies A,
F, G, I and L. The managers we interviewed
acknowledged that the motivational aspect of
performance measurement is stressed here be-
cause the activity is very much uncertain, mostly
unforeseeable and with distant time outcomes,
which makes it difficult to align researchers
efforts with the firms strategic goals. In theseinstances, improving the performance of research-
ers is a matter of stimulating their creativity but,
at the same time, directing their efforts to the
aspects that are relevant for the firm as a whole.
The need to motivate researchers through perfor-
mance measurement seems to be influenced not
only by the type of R&D activity. Company E
exemplifies for instance a situation where the
motivational purpose of performance measure-
ment is felt as particularly critical in new product
development. This is due to the fact that Com-
pany E operates in a science-based industry
(Kodama, 1995), where failure rates and degreesof uncertainties are significantly higher than zero
also in the downward phases of the R&D process,
i.e. development and testing, that turn out to be
highly unpredictable. Finally, our empirical ana-
lysis suggests that motivation of researchers might
become the main objective for R&D performance
measurement in small organisations. In these
cases, in fact, control is very often exerted on a
personal (or clan) basis (Ouchi, 1979), and hence
its very essence lies in the capability to align
employees efforts to the firms priorities. This is
very clear for instance in the case of Company C.Whereas diagnostic and motivational objec-
tives are mutually exclusive, interactive ones are
pursued by the firms in our sample along with
another class of objectives. Moreover, it emerges
that companies that conceive performance mea-
surement in R&D as a means to streamline
communication and coordination, to stimulate
organisational learning and to overcome deci-
sion-making inertia (i.e. to reduce uncertainty in
R&D) share as a common feature the fact of
having a very large R&D unit. Our case studies
clearly indicate that, in order to pursue both
diagnostic and motivational objectives, a criti-
cal aspect is to prevent researchers and engineers
from perceiving their creativity and autonomy as
being too much constrained by the PMS. There-
fore, it is important to adopt an enabling ap-
proach in the management of their performance(Wouters and Wilderom, 2008), where researchers
and engineers are continuously involved in the
measurement process, a double-loop flow of in-
formation keeps them informed about the pro-
gress of R&D activities and coordination and
collective learning allow researchers and engi-
neers to take more autonomous and empowered
decisions. The firms in our sample acknowledge
that this fundamental objective can be actually
pursued through the PMS, and that it becomes
more critical in large R&D units, characterised by
a higher degree of organisational complexity,hierarchy, fragmentation and vertical specialisa-
tion, which raises the need for a better coordina-
tion and more effective organisational learning
processes. Figure 4 shows the conditions under
which each cluster of objectives for R&D perfor-
mance measurement grows in importance.
5.2. PMS characteristics
Our analysis indicates first of all that the dimen-
sions along which R&D performance is evaluated
can be brought back to the BSC perspectives, assuggested by a number of scholars (Kerssen-van
Drongelen and Cook, 1997; Bremser and Barsky,
2004). In fact, the companies that we studied
measure R&D performance taking into account:
The economic and financial aspects associated
with R&D (financial perspective);
The extent to which R&D identifies and
satisfies the needs of its internal and external
customers (customer perspective);
The efficiency with which specific tasks and
processes are carried out (business process
perspective); The extent to which R&D contributes to
generate new knowledge and innovation
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 497
-
8/10/2019 S06 - Chiesa Et Al, 2009
11/32
opportunities (innovation and learning per-
spective).
There are firms that combine several different
perspectives in the measurement of R&D perfor-mance, although they do not knowingly report
using a BSC system. This is the case for instance
of Companies A and C, which take into account
both the innovation and learning and the business
process perspectives, or Company B, which com-
bines the financial, business process and customer
perspectives. Other companies measure perfor-
mance along a single most important dimension.
For instance, Company D uses a PMS very much
focused on internal processes efficiency.
More interestingly, our analysis suggests that
companies tend to use different performance
dimensions to pursue different classes of objec-tives. It noticeably emerges that financial and
customer perspectives are privileged by firms
pursuing diagnostic objectives, as it is clear
from the cases of Companies B, H, N and O.
This is obvious if we consider that selection and
prioritisation of R&D projects is carried out on
the basis of a projects contribution to the firms
competitive advantage, which depends on its
economic/financial outcome and appealing in
the eyes of the customers. On the other hand,
the innovation and learning perspective is wide-
spread among firms pursuing motivational ob-jectives, like Companies A, C, F, I and L,
suggesting that researchers and engineers need
to be motivated mainly on the basis of their
capacity to contribute to the firms innovation
potential. The business process perspective is
instead used by both firms pursuing diagnostic
and motivational objectives. In the latter case,using this perspective besides the innovation and
learning one serves the purpose of introducing a
dimension of performance that can be more
directly controlled by the researcher, which is
critical for motivational purposes as also indi-
cated by theories of action, design and expecta-
tion (e.g., Pritchard, 1990; Moizer, 1991). For
instance, Company A evaluates with this aim the
efficiency (subjectively assessed by peers) with
which researchers perform specific tasks or ac-
quire specific competencies. Firms pursuing di-
agnostic objectives use instead the business
process perspective with the main purpose ofintroducing an operative form of control that
financial and customer-oriented measures do not
allow to perform. As far as interactive objectives
are concerned, it emerges that they are associated
mainly with the business process perspective.
What is interesting to note in this case is that
the efficiency in undertaking business processes is
evaluated in an interactive manner, through a
continuous involvement of engineers and re-
searchers in the measurement process, as it
emerges from the cases of Companies D, H and
M. This is in fact a pre-requisite for introducingthe enabling approach in performance manage-
ment that stimulates the individual autonomy and
DIAGNOSTIC
OBJECTIVES
- New Product Development- Large firms and R&D units
- High Tech industries
MOTIVATIONAL
OBJECTIVES
- Basic and Applied Research- Small firms and R&D units
- Science Based Industries
INTERACTIVE
OBJECTIVES
- Large firms and R&D units
Figure4. Clusters of objectives and measurement context.
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
498 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
12/32
creativity mentioned above. This evidence is con-
sistent with the results reported by Simons (2000)
on the use and role of interactive management
control systems.
Figure 5 suggests therefore that three arche-
types for R&D performance measurement
emerge, each characterised by an internally con-
sistent set of objectives, performance dimensions
and characteristics of the measurement context.
As is clear from Figure 5, our analysis suggests
that the choice of the dimensions along which
R&D performance measurement is carried out is
influenced mainly by the objectives that are pur-
sued, rather than the context in which measure-
ment takes place. In other words, contextual
factors do not appear to affect the design of the
PMS constitutive elements, as instead hypothe-
sised in Section 3. This is clear from the analysis
of Table 2, which shows that companies operating
in different contexts use the same performance
dimensions to pursue identical objectives through
the PMS.
Another interesting aspect unearthed by our
analysis is that firms with large R&D units seem
to be more inclined to use the PMS for diagnos-
tic purposes rather than for motivational ones,
DIAGNOSTIC
OBJECTIVES
- New Product Development
- Large firms and R&D units
- High Tech industries
- Financial perspective
- Customer perspective
- Business process perspective
MOTIVATIONAL
OBJECTIVES
- Basic and Applied Research
- Small firms and R&D units
- Science Based Industries
-Innovation and learning perspective
- Business process perspective
INTERACTIVE
OBJECTIVES
- Large firms and R&D units
- Business process perspective
Figure5. Emerging archetypes for performance measurement in R&D.
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 499
-
8/10/2019 S06 - Chiesa Et Al, 2009
13/32
in comparison with firms with small R&D units.
This is true unless large firms are engaged in basic
and applied research or operate in very uncertain
(science-based) industries, where the motiva-
tional use of the PMS also grows in importance
in large R&D units (see, e.g., the cases of Com-
panies A, I and P).
Furthermore, our analysis shows that each
performance dimension requires specific indica-
tors to be properly estimated. Table 3 provides an
overview of the indicators, associated with the
different performance dimensions, that were iden-
tified in our empirical analysis. It should be noted
that the choice of the indicators is not affected
by the specific class of objectives pursued by the
firm. The same indicators are used by the firmsin our sample, e.g., to measure the business
processes perspective for diagnostic or motiva-
tional purposes.
As is clear, firms try to measure each perfor-
mance dimension combining input, process and
output indicators (Brown and Svenson, 1988;
Hauser, 1998), without any discernible correla-
tion between the type of indicator and perfor-
mance dimension. It is also interesting to
underline the predominance of quantitative ob-
jective indicators (Werner and Souder, 1997). The
managers we interviewed explained their choice toprivilege objective indicators (also at the costs of
leaving out some important intangible facets of
R&D performance) with the need to ensure the
measurability of these metrics, which is funda-
mental for both diagnostic and motivational
purposes, as the theory of task motivation and
incentives (Locke, 1968) indicates. The scarcity of
subjective metrics (both quantitative and qualita-
tive) used by the firms in our sample is, however,
partially in contrast with the evidence gathered in
previous research (e.g., Chiesa et al., 2008), and
this dissimilarity deserves special attention in
future research.
5.3. Critical aspects associated with theuse of the PMS
Analysing the problems associated with the in-
troduction and use of the PMS in the firms in our
sample, it clearly emerges that pursuing motiva-
tional objectives, especially in basic and applied
research units, is far more challenging than using
a PMS with diagnostic purposes in an NPD
organisation. In the former case, the largest partof the managers we interviewed have struggled to
make scientists and researchers positively accept
the PMS, with sporadic cases in which one or two
researchers left the organisation after the intro-
duction of the PMS. It is interesting to note that,
even in these cases, managers have not abandoned
the idea of bringing in the PMS (apart from
Company I). Rather, they have adopted a more
incremental approach to establish the PMS in the
organisation, have often re-designed its charac-
teristics (e.g., measurement frequency, definition
of targets or performance metrics) to take into
account the complaints or suggestions for im-
provement coming from researchers and have
more deeply involved them in the measurement
process. Although tangible results are difficult to
observe, and the costs to run the system are
particularly high, these managers are on averagesatisfied with the PMS. On the other hand, no
particular problems have come across in those
firms that have introduced and used a PMS
with diagnostic purposes in their NPD units,
where tangible results (in terms, e.g., of improved
timeliness of development projects and time-
to-market) can often be observed and the satisfac-
tion of managers is particularly high. It should be
noted that the design of the PMS is often con-
tinuous, with the characteristics of the system
(e.g., the monitored performance dimensions)
that are modified over time to mirror the changesin the competitive strategy and the environment
in which the firm operates (see, e.g., Companies
F, N and M). Although costly, this approach
seems to be particularly useful to improve the
effectiveness of the PMS and managers satisfac-
tion with it.
6. Conclusions
This paper adopts a systemic and contextual
perspective to look into the problem of measuring
R&D performance. In particular, it explores theinterplay between measurement objectives, per-
formance dimensions and contextual factors in
the design of a PMS for R&D activities. With this
aim, we first developed a reference framework
that identifies: (i) the main contextual factors that
might affect the importance a firm attaches to
different objectives for R&D performance mea-
surement and the approaches it uses to measure
R&D performance (i.e. type of R&D activity,
industry belonging and size); (ii) the main aspects
that should be looked at when designing a PMS
for R&D (i.e. dimensions of performance andindicators). This framework was used as a refer-
ence model for the subsequent empirical analysis,
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
500 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
14/32
Table 3. Performance perspectives and indicators
Performance perspective Type of
indicator
Indicators
Financial perspective Input Total cost of each R&D project (4)R&D annual spending (3)R&D annual investment (2)
Process Cost for acquiring a new technology (2)Present Value of R&D accomplishments/R&D expenditures (1)
Output IRR or NPV due to R&D projects (5)Profits due to R&D (3)ROI due to R&D projects (3)Sales (or % of sales) from new products (2)Cost (or % of cost) reduction from new projects (2)Market share due to R&D and innovations (2)
Customer perspective Input No. of interactions with customers during the project (6)% of budget dedicated to customer analysis or verification (3)
% of customer driven projects (1)No. of customers included in the project team (1)Process Time to market (4)
Engineering hours on projects/engineering hours on projects andtroubleshooting (2)No. of training sessions signed off by customer and delivered (1)No. of problem analysis reports requested and delivered (1)
Output No. of customer complaints (4)Customer satisfaction (2)% of support requests fulfilled (2)No. of new customers (1)Response time to customer requests for specials (1)
Innovation and learningperspective
Input No. or % of people with management experience (2)No. of employees in R&D (1)
Process No. of hours of staff training (5)
% of suggestions implemented (2)No. of meeting or time dedicated to the analysis of reasons for failure ofprevious projects (2)Capability to acquire new bodies of competencies (2)
Output No. of new ideas per year (4)No. of innovations delivered to production and commercialization (4)No. of citations of the researchers publications (4)No. of publications (3)No. of patents registered/pending (3)% of patent applications that resulted in registered patents (3)Average product life-cycle length (3)No. of improvements suggestions per employee (3)Scientific excellence of the new ideas identified per year (2)Market attractiveness of the new ideas identified per year (2)International relevance of the competencies acquired during 1 year (2)
No. of products in development or projects in course (2)No. of new processes and significant enhancements per year (1)Business process perspective Input Experience of R&D employees (2)
No. or % of employees involved in goal setting (2)Availability (knowledge) of advanced managerial tools, e.g., projectmanagement techniques (2)Availability (knowledge) of advanced IT support tools, e.g., rapidprototyping and design support tools (1)
Process % of projects respecting costs and budget (6)Agreed milestones/objectives met (6)Quality of documentation to development (5)Average annual improvement in process parameters, e.g., quality cost,lead time, WIP, reliability, capability, down time (4)% of collaboration objectives fully satisfied (3)% of projects that lead to new or enhanced products or processes (2)
% of R&D expenditures that lead to new or enhanced products orprocesses (2)Time spent on changes to original product/project specification (2)
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 501
-
8/10/2019 S06 - Chiesa Et Al, 2009
15/32
which involved 15 technology-intensive firms ac-
tively engaged in R&D activities.
The results of the empirical investigation,
which were discussed at length in the previoussection, indicate the existence of three archetypal
models that companies adopt to measure the
performance of their R&D, which represent in-
ternally consistent sets of contextual variables,
objectives for performance measurement and
performance dimensions. They are schematically
represented in Figure 5. The existence of these
models suggests that firms use performance mea-
surement in R&D to pursue different types of
objectives. In particular, two distinct clusters of
firms emerge: one that uses performance measure-
ment with the main purpose of exerting control
over R&D activities and support critical manage-ment decisions (diagnostic objectives) and the
other that conceives performance measurement
mainly as a means to improve the motivation of
researchers (motivational objectives). Another
set of objectives is concerned with the capability
of performance measurement to improve coordi-
nation and communication, streamline the execu-
tion of complex interrelated tasks and favour
organisational learning (interactive objectives).
They seem to be pursued by firms in combination
with diagnostic or motivational objectives. The
analysis also reveals that the importance firmsattach to each class of objectives is significantly
influenced by the context in which measurement
takes place (see Figures 4 and 5). It also emerges a
specialisation in the performance dimensions used
to measure the different classes of objectives
(see Figure 5). In particular, diagnostic objec-tives are pursued mainly through the use of
financial- and customer-related measures,
whereas indicators associated with the innovation
and learning perspective are the most widespread
among companies pursuing motivational objec-
tives. Contextual factors do not seem to directly
affect the choice of the performance dimensions
used in the PMS. Their impact is in fact mediated
by the classes of objectives the firm decides to
pursue. The paper also provides a synoptic view
of the indicators (or metrics) that the firms
investigated in the scope of our research use to
measure each dimension of performance (seeTable 3). No relevant specialisation in the use of
a given type of indicators emerges along the
different perspectives for performance measure-
ment or classes of objectives.
6.1. Implications for managers
Although the results of the paper should be better
conceived in an exploratory fashion, we believe
they hold valuable implications for R&D man-
agers and, especially, for the heads of R&D unitsand departments who are interested in designing a
PMS for the organisation they are responsible for.
Table 3. (Contd.)
Performance perspective Type of
indicator
Indicators
% of projects using a common design platform (2)Rate of re-use of standard designs/proven technology (2)Average product/service cost variance (2)No. of collaborations stipulated/no. of collaboration opportunitiesidentified (2)Sum of revised project durations/sum of planned durations (1)
Output Average project delay (5)% of projects delayed or cancelled due to lack of funding (5)% of projects delayed or cancelled due to lack of human resources (4)Rate of successful projects, i.e. project achieving the assigned time, cost,quality (4)% of project milestones completed (4)Product quality, measured through indicators specific for each industry/
product (3)No. or % of products/projects completed (2)Degree of project completion (2)Total product development time (2)% of on time deliveries of specification to manufacturing (2)Average time of re-design (1)No. of customer detected design faults (1)
In parentheses the number of firms that used the specific metric.
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
502 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
16/32
First, they are given a number of examples about
how a relevant sample of technology-intensive
firms have designed and used a PMS in their
R&D units, with the aim of pursuing different
classes of objectives under the influence of dis-
similar contextual variables. This rich body of
empirical evidence will provide R&D managers
with a number of insights useful to design a PMS
that is appropriate to the context in which they
operate. The synoptic view of the indicators used
by the firms in the sample (see Table 3) can be a
particularly helpful starting point for the defini-
tion of the set of metrics to be used in their PMSs.
Second, our analysis suggests that introducing
and using a PMS in a basic and applied research
unit to pursue motivational objectives can beparticularly challenging and costly. Some ap-
proaches that are likely to improve R&D man-
agers satisfaction with this type of PMS appear
to be: use of an incremental approach to establish
the PMS into the organisation (e.g., using it first
to measure the most repetitive and predictable
tasks); continuous re-design of the PMSs char-
acteristics to incorporate the suggestions for im-
provement coming from researchers; and a deeper
involvement of researchers in the measurement
process. Finally, the cases in our sample suggest
that a continuous re-design of the PMS to mirrorthe evolution of the firms competitive and R&D
strategy is an important ingredient of success,
notwithstanding the main purposes for which
the PMS is designed and used.
6.2. Implications for research
The paper adds to our understanding of perfor-
mance measurement in R&D because it is one of
the first contributions, to our best knowledge,
that systematically studies the objectives forwhich a firm decides to measure its R&D activ-
ities performance. Moreover, it explores whether
established concepts in management accounting
research (e.g., the relationship between objectives
for performance measurement and characteristics
of the PMS) can be applied in R&D settings as
well. Finally, the approach adopted in the paper
can encourage researchers in the field of R&D
performance measurement to investigate whether
and how the other dimensions of the performance
measurement context (e.g., the firms R&D strat-
egy or the R&D organisational structure) influ-ence the importance a firm attaches to different
classes of objectives.
6.3. Limitations and future research
The study obviously has some limitations. First,
because of the adopted research methodology,results cannot be statistically generalised; they
can only be analytically extended to other indus-
trial firms operating in technology-intensive in-
dustries. Even if the internal validity of the
empirical results is ensured by the cross-case,
explanation-building and pattern-matching ana-
lyses, the study does not explicitly take into
account the effects that other contextual factors
are likely to have on the choice of the objectives
and the characteristics of the PMS elements.
Therefore, further research should be aimed at
exploring the joint effects of other contextualfactors (e.g., the firms R&D strategy) on the
design of the PMS. Second, our analysis is ex-
ploratory in intent. Although we provide some
evidence on the satisfaction of R&D managers
and executives with their measurement system, we
do not systematically assess the capability of a
PMS with specific characteristics to accomplish,
in a given context, the objectives for which it has
been designed. This represents an interesting
avenue for future research, which would require
the development of an appropriate measure of
effectiveness for an R&D PMS and a statistical
analysis of a representative sample of firms.Finally, we believe that adopting a longitudinal
perspective to study the organisation-wide im-
pacts associated with the adoption of the R&D
PMS is another interesting avenue for future
investigation.
References
Azzone, G. (2006) Sistemi di controllo di gestione.
Metodi, strumenti e applicazioni. Milano: Etas.
Bayus, B.L. (1994) Are product life cycles really getting
shorter? Journal of Product Innovation Management,
11, 300308.
Bititici, U.S., Turner, T. and Begemann, C. (2000)
Dynamics of performance measurement systems.
International Journal of Operations and Production
Management, 20, 692704.
Bremser, W.G. and Barsky, N.P. (2004) Utilizing the
balanced scorecard for R&D performance measure-
ment. R&D Management, 34, 229238.
Brown, M.G. and Svenson, R.A. (1988) Measuring
R&D productivity. Research-Technology Manage-
ment, 31, 1115.
Brown, M.G. and Svenson, R.A. (1998) MeasuringR&D productivity. Research-Technology Manage-
ment, 41, 3035.
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 503
-
8/10/2019 S06 - Chiesa Et Al, 2009
17/32
Chenhall, R.H. and Morris, D. (1986) The impact of
structure, environment, and interdependence on the
perceived usefulness of management accounting sys-
tems. The Accounting Review,61, 1635.
Chiesa, V. (1996) Separating research from develop-
ment: evidence from the pharmaceutical industry.
European Management Journal, 14, 6, 638647.
Chiesa, V. and Frattini, F. (2007) Exploring the differ-
ences in performance measurement between research
and development: evidence from a multiple case
study. R&D Management, 37, 283301.
Chiesa, V., Frattini, F., Lazzarotti, V. and Manzini, R.
(2008) Designing a performance measurement system
for the research activities: a reference framework and
an empirical study. Journal of Engineering and Tech-
nology Management, 25, 213226.
Cooper, R. (1993) Winning at New Products: Acceler-ating the Process from Idea to Launch. Reading:
Addison-Wesley.
Davila, T. (2000) An empirical study on the drivers of
management control systems design in new product
development. Accounting, Organisations and Society,
25, 383409.
Driva, H. and Pawar, K.S. (1999) Performance mea-
surement for product design and development in a
manufacturing environment.International Journal of
Production Economics, 60, 6168.
Driva, H., Pawar, K.S. and Menon, U. (2000) Measur-
ing product development performance in manufac-
turing organizations. International Journal ofProduction Economics, 63, 147159.
Eisenhardt, K.M. (1989) Building theories from case
study research. Academy of Management review, 14,
4, 532550.
Eisenhardt, K.M. and Graebner, M.E. (2007) Theory
building from cases: opportunities and challenges.
Academy of Management Journal, 50, 2532.
Godener, A. and Soderquist, K.E. (2004) Use and
impact of performance measurement results in
R&D and NPD: an exploratory study. R&D Man-
agement, 32, 191220.
Gordon, L.A. and Miller, D.A. (1976) A contingency
framework for the design of accounting information
systems. Accounting, Organizations and Society, 1,
5969.
Gordon, L.A. and Narayanan, V.K. (1984) Manage-
ment accounting systems, perceived environmental
uncertainty and organizational structure: an empiri-
cal investigation. Accounting, Organizations an So-
ciety, 9, 3347.
Hauser, J.R. (1998) Research, development and engi-
neering metrics.Management Science,44, 16701689.
Kaplan, R.S. and Norton, D.P. (1992) The balance
scorecard measures that drive performance. Har-
vard Business Review, 70, 7179.
Kerssen-van Drongelen, I.C. and Bilderbeek, J. (1999)
R&D performance measurement: more than choos-ing a set of metrics. R&D Management, 29, 35
46.
Kerssen-van Drongelen, I.C. and Cook, A. (1997) De-
sign principles for the development of measurement
systems for research and development processes.
R&D Management, 27, 345357.
Kerssen-van Drongelen, I.C., Nixon, B. and Pearson,
A. (2000) Performance measurement in industrial
R&D.International Journal of Management Reviews,
2, 111143.
Kim, B. and Oh, H. (2002) An effective R&D PMS:
survey of Korean R&D researchers. International
Journal of Management Science, 30, 1931.
Kodama, F. (1995) Emerging Patterns of Innovation.
Boston: Harvard Business School Press.
Loch, C.H., Stein, L. and Terwiesch, C. (1996) Mea-
suring development performance in electronics in-
dustry. Journal of Product Innovation Management,
13, 320.Loch, C.H. and Tapper, S. (2002) Implementing a
strategy-driven performance measurement system
for an applied research group. Journal of Product
Innovation Management, 19, 185198.
Locke, E.A. (1968) Toward a theory of task motivation
and incentives.Organizational Behaviour and Human
Performance, 3, 157189.
Miles, M.B. and Huberman, A.M. (1984) Qualitative
Data Analysis. Newbury Park: Sage.
Moizer, P. (1991) Performance appraisal and rewards.
In Ashton, D., Hopper, T. and Scapens, R.W. (eds.),
Issues in Management Accounting. Englewood Cliffs:
Prentice Hall, pp. 126144.Neely, A. (1999) The performance measurement revo-
lution: why now and what next?International Journal
of Operations and Production Management, 19, 205
228.
Nevens, T.M., Summe, G.L. and Uttal, B. (1990)
Commercializing technology: what the best Compa-
nies do. Harvard Business Review, 68, 154163.
Nixon, B. (1998) Research and development perfor-
mance measurement: a case study. Management
Accounting Research, 9, 329355.
Ojanen, V. and Vuola, O. (2006) Coping with the
multiple dimensions of R&D performance analysis.
International Journal of Technology Management,33,
279290.
Ouchi, W. (1979) A conceptual framework for the
design of organizational control mechanisms. Man-
agement Science, 25, 9, 833848.
Pappas, R.A. and Remer, D.S. (1985) Measuring R&D
productivity. Research Technology Management, 28,
1522.
Pawar, K.S. and Driva, H. (1999) Performance mea-
surement for product design and development in a
manufacturing environment.International Journal of
Production Economics, 60-61, 6168.
Pillai, A.S., Joshi, A. and Rao, K.S. (2001)
Performance measurement of R&D projects in a
multi-project, concurrent engineering environment.International Journal of Project Management, 20,
165177.
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
504 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
18/32
Poh, K.L., Ang, B.W. and Bai, F. (2001) A compara-
tive analysis of R&D project evaluation methods.
R&D Management, 31, 6376.
Pritchard, R.D. (1990) Measuring and Improving Orga-
nizational Productivity: a Practical Guide. New York:
Praeger.
Rockness, H.O. and Shields, M.D. (1988) An empirical
analysis of the expenditure budget in research and
development. Contemporary Accounting Research,4,
568581.
Simons, R. (1994) How new top managers use control
system as levers of strategic renewal. Strategic Man-
agement Journal, 15, 169189.
Simons, R. (1995)Levers of control How Managers Use
Innovative Control System to Drive Strategic Re-
newal. Boston: Harvard Business School Press.
Simons, R. (2000)Performance Measurement & ControlSystems for Implementing Strategy. Upper Saddle
River: Prentice Hall.
Sivathanu, P.A. and Srinivasa, R.K. (1996) Perfor-
mance monitoring in R&D projects. R&D Manage-
ment, 26 , 5765.
Tipping, J.W., Zeffren, E. and Fusfeld, A.R. (1995)
Assessing the value of your technology. Research-
Technology Management, 38, 2239.
Tuomela, T.S. (2005) The interplay of different levers
of control: a case study of introducing a new perfor-
mance measurement system. Management Account-
ing Research, 16, 293320.
Werner, B.M. and Souder, W.E. (1997) Measuring
R&D performance State of the art. Research-Technology Management, 40, 3442.
Wind, J. and Mahajan, V. (1997) Issues and opportu-
nities in new product development: an introduction
to the special issue. Journal of Marketing Research,
34, 112.
Wolf, M. (2006) The world must get to rips with seismic
economic shifts. Financial Express, February, 7.
(available at:http://www.financialexpress-bd.com).
Wouters, M. and Wilderom, C. (2008) Developing
performance-measurement systems as enabling for-
malization: a longitudinal field study of a logistics
department.Accounting, Organizations and Society,
33, 488516.Yin, R.K. (2003) Case study Research: Design and
Methods. London: Sage.
Notes
1. Kodama (1995) classifies industrial sectors into
dominant design, high-tech and science-based in-
dustries. These three clusters differ with respect to
their level of risk, defined as the probability that an
R&D program expenditure is frozen in an inter-
mediate phase of the R&D process (freezing rate).
Dominant design industries are characterised by a
freezing rate that dramatically decreases from basicthrough applied research, coming to zero in the
development phase. The dominant design industries
are: food, textile, pulp and paper, printing and
publishing, oil and paints, petroleum and coal,
rubber, ceramics, iron and steel, transportation,
energy. High-tech industries are characterised by a
freezing rate that decreases throughout the R&D
process, but remains 40 even in the late develop-
ment. High-Tech industries are: ordinary machin-
ery, electrical machinery, communications and
electronics, precision equipment and aerospace.
Science-based industries are those in which the
freezing rate always remains high throughout the
entire R&D process. Science-based industries are
basically pharmaceuticals and industrial chemicals.
Vittorio Chiesais Professor of R&D Strategy andOrganisation at Politecnico di Milano. He is
member of the Management Committee of MIP
the Business School of Politecnico di Milano,
where he is responsible for the Technology Strat-
egy area. His main research interests concern
R&D management and organisation, technology
strategy and international R&D. He has pub-
lished 6 books and more than 100 papers, includ-
ing 40 articles in leading international journals
such as the Journal of Product Innovation Man-
agement, IEEE Transactions on Engineering Man-
agement and International Journal of Operationsand Production Management.
Federico Frattini (Corresponding author) is Assis-
tant Professor at the Department of Manage-
ment, Economics and Industrial Engineering of
Politecnico di Milano. His research interests
concern R&D performance measurement, the
organisation of R&D activities and the commer-
cialisation of innovation in high-tech markets. He
has published more than 30 papers, including
articles in R&D Management, Journal of Engi-
neering and Technology Management, and Inter-national Journal of Technology Management.
Valentina Lazzarotti is Assistant Professor at
Universita` Carlo Cattaneo LIUC (Castellanza,
Varese). She teaches Economics and Business
Organization and Management Control Systems
at LIUC. She obtained her Master Degree in
Economics at Universita` Bocconi (Milano). Her
research interests concern R&D performance
measurement and organization. She has pub-
lished more that 30 papers including articles in
Journal of Engineering and Technology Manage-ment and International Journal of Technology
Management.
Performance measurement in R&D
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd
R&D Management 39, 5, 2009 505
http://www.financialexpress-bd.com/http://www.financialexpress-bd.com/ -
8/10/2019 S06 - Chiesa Et Al, 2009
19/32
Raffaella Manzini is Associate Professor at
Universita` Carlo Cattaneo LIUC (Castellanza,
Varese). She teaches Economics and Business
Organization and Technology Strategy at LIUC
and Politecnico di Milano. Her research interests
concern technology strategy and planning, R&D
management and organization. She has published
more than 60 papers including articles in leading
journals such as Long Range Planning and R&D
Management.
Appendix A. Interview protocol
What kind of products or services does yourcompany supply? Which are the distinctive
characteristics of the industry in which it
operates? How high is the probability that
an R&D expenditure is frozen in an inter-
mediate phase of R&D process? How long
does this failure rate remain higher than zero?
Is it higher than zero also in the downward
phases of the R&D process, i.e. development
and testing? How do these characteristics
affect the challenges your firm is confronted
with in R&D?
Which are your companys core competences?
Which are the most critical for your competi-tive advantage?
What percentage of your companys annual
turnover is invested in research and/or devel-
opment activities?
What kind of R&D activity does your com-
pany carry out? Is it basic and applied re-
search or new product development? Is it
undertaken in distinct organisational units?
Which are the main characteristics of your
R&D organisation? Which is the scope of the
activities undertaken in the R&D unit we are
focusing on?
How many people does your company em-
ploy? How many scientists and researchers are
employed in R&D? And how many in the
units we are focusing on?
How long have you been measuring the per-
formance of your R&D unit? Which are the
main characteristics of the performance mea-
surement system you have been employing?
Have they evolved over time?
Why have you decided to measure R&D
performance? Which are the main objectives
that you pursue through R&D performance
measurement? How would you rate them on
the basis of their importance for your R&D or
competitive strategy? Are measurement objec-
tives mutually exclusive to some extent? What kind of performance dimensions does
your company measure in order to pursue the
established objectives? Are these performance
dimensions completely controllable by re-
searchers and engineers? Is the choice of the
performance dimensions influenced by the
objectives you wish to pursue? Are there any
other reasons behind the choice of these
dimensions?
Which indicators are employed to operatively
measure performance? Do you use different
indicators to measure different performancedimensions? Is the choice of the indicators
constrained by any reasons (e.g., incompat-
ibility with the corporate-level PMS)?
Did your company suffer from any con-
straints of human and financial resources in
designing and implementing the performance
measurement system? Is the set of objectives
that your firms pursue, or the performance
dimensions that it employs, somehow limited
by a lack of resources? And what about the
degree of measurement formalization?
Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini
506 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd
-
8/10/2019 S06 - Chiesa Et Al, 2009
20/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
21/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
22/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
23/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
24/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
25/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
26/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
27/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
28/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
29/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
30/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
31/32
-
8/10/2019 S06 - Chiesa Et Al, 2009
32/32
top related