What is the Effect of Performance Measurement on Perceived Accountability Effectiveness

in State and Local Government Contracts

Anna Amirkhanyan

ABSTRACT

Designing and implementing performance measurement systems in public contracts is not an

easy task. Little guidance has been available on which specific measures work better in

producing certain managerial benefits. The objective of this study is to evaluate the effect of

different performance measurement practices on accountability effectiveness in government

contracts. The findings suggest that the overall scope of performance measurement has a

positive impact on the government’s ability to effectively manage contracts. More specifically,

measuring costs, client impact, service timeliness and disruptions, as well as specifying the

detailed processes for service delivery are associated with higher accountability effectiveness.

On the other hand, evaluating quality, client satisfaction, and using informal monitoring

techniques has a negative impact on perceived accountability effectiveness. The results of this

study provide motivation for the contract managers to optimize performance monitoring and

reduce transaction costs by relying on the measures that are more likely to improve contract

implementation.

1

INTRODUCTION

In his inaugural address President Barack Obama noted that the question Americans

should be asking today is not whether the government is too big or too small, but whether it

works. This once again renewed attention to performance in the environment of financial crisis

is especially relevant to the large number of privatized services delivered by nonprofit and for-

profit organizations. While performance monitoring and measurement have a long history in the

United States, these functions traditionally receive little attention by government managers (Van

Slyke 2003). Little is known about the effectiveness and relative efficacy of measures used by

the public managers overseeing contract implementation (Ho 2006; Yang, Hsieh, and Li 2009).

The value of performance monitoring and measurement is questionable even in localities known

for contracting out virtually all of their services (Prager 2008). Nonetheless performance

measurement involves critical management decisions that “operationalize policies” and “provide

real-world specificity to abstract ideas and policy and are therefore of great consequence”

(Cohen and Eimicke 2008, 151).

Designing and implementing performance measurement systems in government agencies

and private contracted organizations is not an easy task both theoretically and practically

(Heinrich 2002). The complex nature of performance necessitates the use of multiple measures

to capture all aspects of organizational well-being. Importantly, these oversight systems are not

static: some scholars recommend using detailed performance metrics initially and later

simplifying them, abandoning some measures and focusing on others in order to build trust

between parties (Linder 2004). To date, little guidance is available on which specific measures

work better in producing certain managerial benefits. In this connection, Cohen and Eimicke

(2008, 155) note: “[t]he challenge for the managers is how to create a set of measures that is

comprehensive and still limited enough to focus the organization on what is most important.”

2

Understanding how to build the optimal performance measurement practices is especially

important since these activities are almost always associated with higher administrative costs

(Melkers and Willoughby, 2005; Van Slyke 2003, 306; Zimmerman and Stevens 2006). This

goal is particularly relevant in the context of public-private partnerships that are subject to

informational asymmetry and opportunistic behavior of private actors.

This research examines performance measurement practices with the purpose of

understanding their benefits. While the long term programmatic outcomes of contracted services

are often hard to verify, managerial outcomes can become useful proxies for organizational

performance. Of the many managerial impacts of performance measurement this study focuses

on accountability effectiveness1 (Romzek and Johnston 2002). Both “accountability” and

“effectiveness” have been widely used in a variety of contexts and defined somewhat differently.

Our approach here is more narrow and specific to the context of government contracting. In this

field, the concept of accountability effectiveness has been proposed by Romzek and Johnston

(2002, 2005) to describe the capacity of a government agency “to design, implement, manage,

and achieve accountability for its social service contracts. This includes the state’s ability to

obtain timely and accurate reporting from the contractor and to use that information to evaluate

performance and correct deficiencies” (Romzek and Johnston, 2005: 437). Accountability

effectiveness is different from the overall program results; instead, it refers to the managerial

effectiveness in contract implementation. The first objective of this study is to evaluate the

influence of the scope of performance measurement on accountability effectiveness in a sample

of state and local government contracts. The second objective is to examine the effect of fifteen

distinct performance measurement practices on contract accountability effectiveness. Past

research suggests that some performance monitoring systems are developed unilaterally by the

government agency, while others result from a collaborative dialogue between the government

3

and the contractor (Amirkhanyan 2009). Hence, the third objective of this study is to determine

whether these collaborative measurement efforts affect accountability effectiveness. The results

of this study may help contract managers to optimize the process of performance monitoring and

reduce transaction costs by focusing on a shorter list of measures that are likely to improve the

governments’ capacity to implement contracts while maintaining high levels of accountability.

PERFORMANCE MEASUREMENT AND ACCOUNTABILITY

Performance measurement is the collection, reporting, and review of data reflecting

various aspects of organizational performance, including service quality, cost-effectiveness and

others (Blasi 2002, 531; Cohen and Eimicke 2008). As the classical upward accountability and

compliance with the formal rules in the public sector gradually gives way to the new concept of

managerial answerability to a variety of actors operating in the “hollow state,”2 the demands for

optimizing the performance measurement process increase (De Vries 2007; Holzer and Kloby

2005). The measures currently used to evaluate publically delivered as well as privatized

programs are diverse and complex (Behn 2003; Blasi 2002; Boyne 1998; Byrnes, Freeman, and

Kauffman 1997; Callahan and Kloby 2007; Dilger, Moffett and Struyk 1997). They include

organizational inputs, processes, outputs and outcomes; reflect internal capacities and external

perceptions, and provide both quantitative and qualitative accounts of organizational activities

(Blasi 2002; Cohen and Emicke 2008; Dalehite 2008; Dilger, Moffett and Struyk 1997; Kelly

and Swindell 2002a).

As the U.S. government continues to rely on contractors in the delivery of public services

(Hefetz and Warner 2007), measuring and monitoring the contractors’ performance is as critical

as in the public sector. While, ideally, competitive private markets replace the government

monopolies and create pressures to improve performance (Savas 2000, 2003), the government

4

agencies need to stay involved in contract management since they are ultimately responsible for

the outcomes (Brown and Potoski 2006; Brown, Potoski, and Van Slyke 2006; Goldsmith and

Eggers 2004; Johnston and Romzek 1999; Prager 1994; Rubin 2006). Close monitoring is

necessary because (a) market competition is limited in many areas (Johnston and Girth 2010) and

contractors in such markets can behave opportunistically; (b) market forces may fail to pressure

the contractors to achieve some publicly important outcomes, such as equal access or legal

compliance (Chan and Rosenbloom 2010), and (c) even in competitive markets the information

about the contractor and the services is often incomplete, which makes it difficult to choose the

right contractor (Arrow 1984; Eisenhardt 1989). Empirical studies of performance monitoring

confirm that contractors are in fact monitored at least as intensively and closely as the programs

delivered in-house (Marvel and Marvel 2007).

Contract management involves four important policy choices, including the make-or-buy

decision (or agenda setting), contracting formulation, implementation, and evaluation (Brown

and Potoski 2003; Yang, Hsieh, and Li 2009). The specific managerial activities pursued during

these phases include evaluation of provider markets, political feasibility assessments, initial

examination of service characteristics, pre-award conferences, ongoing contract specification,

communication, data collection, reporting, inspections, sanctioning, terminations, and renewals.

Most of these activities have the goal of improving the contractor’s performance.

Elements of performance measurement are in fact present in each of the four domains of

contract management. During the agenda setting phase, governments consider service

measurability – the ease of developing a clear set of quantifiable and reliable measures – and,

accordingly, determine the feasibility of privatization (Amirkhanyan, Kim and Lambright 2007).

The contract formulation phase involves planning for performance measurement and the

preliminary measurement of the contractors’ past performance (Yang, Hsieh, Li 2009). The latter

5

includes investigating the contractor’s reputation, reviewing its past service quality, management

capacity, regulatory compliance and professional certifications. At this stage, the partners discuss

the basic approaches to service delivery, standards and expectations. The contract

implementation and evaluation phases often run in parallel, with the service delivery being

observed, inspected, recorded, and reported upon, while the partners clarify and, sometimes,

modify performance standards in a collaborative fashion (Amirkhanyan 2009).

The specific measures used within a contracting arrangement may be similar to those

used in-house such as cost-effectiveness, service scope, quality, regulatory compliance and equal

access. The measurement process, however, may be organized in different ways. First, several

parties may be responsible for contract evaluation. It may be outsourced or performed directly by

the government (Brown and Potoski 2006), and may utilize data that are self-reported, observed

by the government agency, or collected by a third-party (Cohen and Eimicke 2008, 151).

Second, different contract monitoring theories may underline these efforts. One approach

aspires to achieve complete contract specification and involves defining all contingencies,

expectations, standards, inputs, processes, outputs, and outcomes at the onset of the contract and

tracking these data through a pre-determined monitoring and reporting procedure (Milgrom and

Roberts 1992). Due to a high degree of goal ambiguity and environmental uncertainty within

many service fields, achieving complete contract specification is regarded impossible (Brown,

Potoski and Van Slyke 2006; Tirole 1999), and its practical implementation raises concerns

about micromanagement and excessive hierarchical control which undermined the idea of

market-based alternatives. As an alternative approach, the New Public Management movement

suggests focusing on the end results in performance measurement. Performance-based

contracting focuses the government’s attention on a set of outcomes, allowing the contractors to

determine the inputs and the process, and linking the outcomes to monetary rewards (Goldsmith

6

and Eggers 2004; Heinrich 1999). While this mode of contracting may save costs (Straub 2009)

and generate data for resource allocation decisions (Heinrich 1999), several problems have been

noted. The performance standards are often not well-designed (Heinrich 1999), the sanctions are

rarely executed in response to inadequate performance (Cohen and Eimicke 2008; Van Slyke

2007), and the democratic process of service delivery may be undermined (Chan and

Rosenbloom 2010). The third contract management philosophy – relational or cooperation

contracting – expands its focus back to inputs and processes and a broader set of democratic

values, such as fairness and equal treatment. Rather than specifying all measures and standards

in advance, relational contracts are long-term open-ended trust-based partnerships that allow

performance criteria to evolve in response to the changing conditions; here, informal and

professional pressures replace the rigid sanctions; and the parties work collaboratively on

overcoming the obstacles (Allen 2002; Amirkhanyan 2009; Beinecke and DeFillippi 1999;

Campbell and Harris 1993; Davis 2007; DeHoog 1990; Sclar 2000; Smith 2005). Amirkhanyan

(2009) cites numerous examples of the more informal performance measurement practices: e.g.,

a contracting officer voluntarily attends the theater performances involving incarcerated youth –

a service provided by a for-profit arts company – and informally seeks the participants’ and their

parents’ feedback which she does not have to formally record or report. While relational model

attempts to address the limitations of the previous approaches, the contractors’ close involvement

in performance evaluation raises objectivity related concerns by biasing the agency towards a

particular contractor (DeHoog 1990; Van Slyke 2003, 306).

Irrespective of the theoretical underpinnings of contractor performance measurement, the

latter remains among “the most serious management challenges” facing public managers

(Kelman 2002, 312). Similar to the publicly administered programs, contract outcomes are

difficult to quantify, and the pursued goals may be diverging and unclear (Meyers, Riccucci, and

7

Lurie, 2001; Riccucci, 2005). Cost concerns may supersede the quality considerations, and the

performance measures may not reflect the initial goals (Heinrich 1999). These problems not

only increase the costs of contract management (Sclar 2000), but also raise the central issue of

government contracting that has to do with accountability (Hodge 2000). As Heinrich writes:

“[u]seful performance management systems will improve programs by assisting public managers

to identify poor performers, to follow-up with corrective actions, and to reward good performers

and replicate their approaches” (Heinrich 1999, 367). The question of the main features and

components of such useful performance measurement systems remains largely unanswered.

Despite the long history and scope of evaluation efforts, both organizational performance

and contract management literatures continue to ask whether performance measurement matters

(Ho 2006). While performance measurement is eventually expected to be translated into better

programmatic outcomes and have some political or symbolic effects, the more immediate impact

of these practices is expected to be on management (Moynihan 2005; Yang and Hsieh 2007).

Specifically, performance measurement has been viewed as “the newest method of ensuring

accountability” (Zimmerman and Stevens 2006, 315). While it is often assumed that

performance measurement leads to better accountability, few studies actually explored this

question. In the broader public sector literature, Ho (2006) and Berman and Wang (2000) suggest

that performance measurement leads to improved perceived accountability of government

agencies. This question has not been examined in the government contracting literature.

Moreover, no data exist on whether the scope and type of measures may improve accountability

effectiveness in government contracts.

The first objective of this study is to determine whether the scope of performance

measurement in government contracts can positively influence accountability effectiveness in

government contracts. Here, the term “scope” refers to the aggregate of all types of performance

8

measurement activities used to monitor and evaluate a contract. Using multiple measures of

contractor performance can help generate more performance data and reveal service delivery

problems. Multiple measures are likely to supply different kind of information – qualitative and

quantitative, perceptual and more objective – drawing a more comprehensive picture of the

contractor’s performance. If one measure helps identify a case of substandard performance

associated with one specific aspect of service delivery, other measures may help verify and

clarify the extent of the problem and determine its impact on other aspects of performance or

other stakeholders involved in the implementation process. For example, increasing costs of

production may be interpreted differently depending on the associated changes in client welfare,

service quality, and compliance with the regulatory requirements. Thus, we expect the scope of

performance measurement to improve perceived accountability effectiveness.

The second objective of this study is exploratory: to examine fifteen distinct performance

measurement and monitoring techniques and to evaluate their individual effects on the overall

accountability effectiveness in government contracts. As mentioned earlier, government

agencies use a variety of performance measurement techniques focusing on costs, quality, impact

on service recipients, timeliness, compliance with the laws, fairness, reputation, and customer

satisfaction; they collect qualitative and quantitative data, and rely on formal and informal ways

of collecting and handling information. Past research found that the so-called higher-order

measures, such as efficiency, are more likely to influence management and operation of public

organizations rather than the lower-order measures, such as workload and outputs (Ammons and

Rivenbark 2008). Thus, our goal is to determine which performance measures (detailed in the

methodology section) are more likely to be associated with the more effective accountability

relationships in government contracts. On one hand, the data on contractor performance

outcomes, such as the impact of services on the clients, or service quality, should be critical for

9

the agency’s ability to evaluate performance, detect deficiencies and correct them. On the other

hand, quality and impact data may be hard to obtain and interpret, while using the more straight-

forward data on costs, compliance with the industry laws, service timeliness and disruptions may

be more feasible in order to detect and promptly correct performance problems.

The third objective of this study has to do with how these measures are developed. The

literature on performance measurement found evidence of both the government agencies and the

private contractors participating in the formation of monitoring systems. Amirkhanyan (2009)

found that a variety of collaborative activities are employed by both parties: monitoring officers

seek their contractors’ input on performance evaluation, meanwhile the contractors develop and

propose new measures and actively negotiate the existing monitoring arrangements. Several

other studies also provide evidence of multiple parties participating in the development and

implementation of performance measures (Heikkila and Isett 2007; Holzer and Kloby 2005;

Romzek and Johnston 2002, 428). Such participatory mechanisms, though complex and lengthy,

ensure that all decision-makers understand the background and the advantages of the

measurement process and perceive the monitoring systems as credible (Kravchuk and Schack

1996). Thus, collaboration between the agency and the contractor in the process of developing

performance measurement mechanisms might affect the timeliness and the accuracy of

performance data and influence the likelihood of imposing the sanctions and eventually

correcting the deficiencies. In this connection, the third objective of this study is to determine

whether collaborative practices used to develop performance measurement and monitoring

systems positively influence the governments’ ability to manage their contracts effectively.

Accountability in contract implementation is a function of many other factors, such as the

degree of trust between the agency and the contractor. As detailed in the next section, in this

analysis we will control for the effect of several organizational and environmental factors.

10

METHODS

Data3

Sixty-nine interviews4 were administered with government contract managers in state and local

government agencies as well as the managers of nonprofit and for-profit organizations contracted

by these governments.5 Jurisdictions under consideration included the District of Columbia, three

adjacent counties, and one adjacent state.6 Searchable open-ended online listings of contracted

services were accessed on the procurement office web sites of each jurisdiction.7 While the

stratified random sampling would strengthen the design of this study, the lack of jurisdiction-

level data on the proportional distribution of service fields made this strategy impractical.

Purposive sampling, which involves identification of cases that appear to represent the

population with the purpose of capturing a broad range of its characteristics, was appropriate and

practical considering the objectives of the study and the data sources.

The sampling procedure started with an extensive review of service contract listings on

the procurement web sites of the five examined jurisdictions. I sought to maximize the

representation of service fields, service measurability, award amounts, vendor ownership status

and other factors. After reviewing over two hundred contracts as well as the organizational

structures of each jurisdiction, a preliminary list of service areas was created (shown in Table 1).

Contracts for long term and medical care, construction and maintenance, management consulting

services, and IT were very prevalent in the listings, and were also prominently represented in the

sample. Respondents associated with the relatively infrequent contracts such as translation,

parking meter maintenance or animal care were also included, which helped improve the

variation of award amounts. This study avoided two-sided representation of contracts, i.e.,

public and private respondents in this study were not associated with the same contract. While

this does not allow us to examine the same contractual arrangements from the two sides,

11

independence of observations in the sample allows us to run a single regression model without

biasing the result. A control variable indicating each respondent’s public or private status is

included in all regressions (coded as one for public respondents and zero for the contractors).

This study uses a non-probability (purposive) sample, and its limitations (e.g., the

possibility of over-representing certain service areas) may apply to this analysis. However, by

diversifying the service fields, this study makes a contribution to the contracting literature by

capturing the richness of performance measurement strategies used by state and local

governments. Specifically, it was critical to incorporate contracts falling on the wide spectrum of

measurement efforts considering the 15 identified performance measurement tactics detailed

below. This would be difficult to achieve in a sample of contracts in the same field. Also, while

the sample appears quite heterogeneous, the listings reviewed by the author were also

characterized by high levels of diversity. Thus, the sample appears to be representative; however,

it is most representative of the five studied jurisdictions8.

After two pre-tests, the final sample of thirty-nine public employees and thirty private

managers was interviewed by the investigator. Program officers were interviewed in 96% of all

public-sector interviews, meanwhile, procurement officers knowledgeable about monitoring

procedures were interviewed in the remaining 4% of cases. Government officers who oversaw

multiple contracts were asked to discuss the most typical contract they monitored. Each person

participated in one interview, which took approximately one hour. The percentage of contracts

with for-profit vendors was higher among both public and private respondents: sixty-seven

percent of public contract managers discussed a contract with a for-profit organization, with the

remaining thirty-three percent discussing a contract with a nonprofit organization. Among

private respondents, sixty-three percent were for-profit, while thirty seven percent were

nonprofit. As shown in table 1, numerous service fields have been examined, including health

12

and psychological care (e.g., nursing home care or special therapy for incarcerated youth), as

well as management consulting (e.g., public program design and evaluation), construction and

maintenance (e.g., waste management equipment maintenance) and many others. Appendix A

presents six short vignettes describing six typical contracts included in the sample and

characterized by high, medium, and low levels of performance measurement and monitoring.

Dependent Variable

Six items, administered to both public and private respondents in the sample, were used to

measure the perceived levels of accountability effectiveness. These items are based on the

operational definition provided by Romzek and Johnston (2002, 2005). Respondents were asked

if they (a) strongly agreed, (b) somewhat agreed, (c) somewhat disagreed, and (d) strongly

disagreed with each of the following six statements:

1. The contractor accurately complies with our performance measurement requirements.9

2. We receive all the necessary information in a timely manner.

3. The information we receive is accurate.

4. We use various sanctions in cases when the contractor fails to provide timely and

accurate information on their performance.

5. Performance measures that we use help us reveal inadequate performance of our

contractor.

6. In general, we are very effective in terms of our ability to manage and implement this

contract.

While originally the sum of these six items was intended to be used as the dependent

variable in regression analysis, the obtained scale had an overall raw Cronbach alpha of only 0.6.

Exploratory factors analysis indentified two underlying factors: the first one strongly correlated

with items 1, 2, and 3, and the second one with items 5 and 610

. Based on this analysis, three

dependent variables were created:

a). Contractor complies by providing timely and accurate information, an ordinal

12-point scale, representing the sum of items 1 through 3 listed above.

b). Government reveals problems and effectively manages the contract, an ordinal 8

point scale, representing the sum of items 5 and 6 listed above.

c). Government uses sanctions, representing item 4 above.

13

Importantly, this research focuses on the managers’ perceived accountability effectiveness.

Certainly, accountability effectiveness is independent of the agencies’ and vendors’ perceptions

of thereof, but this study focuses on the respondents’ subjective evaluation of this phenomenon.

Independent Variables

The central independent variables of interest are the performance measurement practices used in

the course of contract monitoring as well as the collaborative practices used to develop these

measures. As shown in Appendix B, respondents were asked to recall if the government agency

“collected, monitored, or evaluated information” on fifteen distinct aspects of contractor

performance such as costs, quality, workload, impact on clients, customer satisfaction and

several others. Past studies have effectively used dichotomous (i.e., “yes” and “no”) response

categories to study performance measurement capacity in the contracting setting (Brown and

Potoski 2003). In this study, positive responses to each of the fifteen items, indicating that a

particular aspect of performance was indeed evaluated by the government, were coded as one.

Negative and the “don’t know/don’t recall” answers were coded as zero. In addition to the

fifteen variables measuring each of these performance measurement practices, the sum of all

fifteen items was calculated for each respondent. This variable – the number of performance

measurement techniques used – reflects the scope of performance measurement in each contact.

Six questions helped identify the collaborative practices used by the government and the

contractor in performance measurement. Questions 2 through 7, shown in Appendix B, were

used to create six dummy variables (asking for input, contractor negotiation, input incorporated,

communication affects performance, contractor seeking clarification, and government seeking

clarification), each coded as one for affirmative responses. Variable collaborative performance

measurement index, the sum of these six items, was used in the regression analysis.

Analysis

14

Since the three dependent variables were ordinal, ordered logit was used to examine the effect of

performance measurement and their collaborative development on the accountability

effectiveness measures. A set of other controls measuring important organizational and

environmental factors was included in each model (described in Appendix C). For each of the

three dependent variables, sixteen regression models were obtained, using the following process:

In the first model, the number of performance measurement techniques used was used as

the central explanatory variable. In subsequent fifteen models, the number of performance

measurement techniques used was replaced with each of the fifteen performance measures. 11

Regressions obtained with one of the three dependent variables - government uses

sanctions – had low overall statistical power and this model was excluded from the analysis.

The two remaining measures of accountability effectiveness – contractor complies by providing

timely and accurate information and government reveals problems and effectively manages the

contract – were statistically significant and produced interesting results.12

FINDINGS

Table 2 shows the prevalence of each measure in the sample. Consistent with the

reviewed literature, the numbers vary considerably. Measurement of quality, timeliness,

continuity, vendors’ workload, and using informal monitoring is quite common. Less common

are the contracts focusing on client satisfaction and the impact of services on clients, as well as

those using quantitative and qualitative performance indicators. Finally, very few respondents

reported monitoring cost-effectiveness,13

reputation, ability to provide services without

discrimination, specify detailed procedures for service delivery, or tailor performance

measurement to the contracted organization. Notably, government respondents are more likely

to report the use of each measure than the contractors. Several arguments have been proposed to

explain these findings (Amirkhanyan 2009). First, government monitors may experience

15

evaluation apprehension and tend to over-report their activities. Second, the contractors may not

be aware of the extent of government agency’s evaluation, especially if performance data is

collected by the government. Third, contractors may be skeptical of government’s efforts. Thus,

while government employees reported on their propensity to monitor performance, contractors

may have been commenting on the governments’ propensity to monitor performance effectively.

When asked if the government agency evaluated service quality, one private respondent

answered negatively and then elaborated: “They think they do! …But I really don’t think they do

it…” This finding was reiterated in other interviews.

Tables 3 and 4 provide descriptive statistics for the dependent variables which are

measured ordinally. Regression analysis pertaining to the first dependent variable – contractor

complies by providing timely and accurate information – is presented in table 5. Sixteen ordered

logit models were obtained by regressing the dependent variable on the total number of

performance measures (along with the control variables), and consequently, upon each of the

fifteen individual performance measures and the same set of controls. The total number of

measures and eleven out of fifteen individual performance measures had no significant influence

on this first measure of accountability effectiveness (results not shown). Meanwhile, four

measures produced significant results (see table 3). First, respondents who reported that their

contracts involved monitoring service costs reported significantly higher perceived levels of

contractor compliance and information timeliness and accuracy (Model 1). Similarly,

monitoring service disruptions had a positive effect on the dependent variable (Model 2). On the

other hand, monitoring service quality as well as monitoring contractors informally had a

negative association with contractor compliance and information timeliness and accuracy

(Models 3 and 4). Importantly, collaborative performance measurement index was insignificant

in all models. Among the control variables, “hard” services were associated with reduced

16

contractor compliance and information quality. Working with a contractor that has a unique

expertise and was selected using a competitive bidding process improved perceived compliance

and timely/accurate information, while dynamic environments and use of performance

information “self-reported” by the contractors had a negative effect on the dependent variable.

Finally, public rather than private respondents and those with more extensive contract

management experience were less likely to perceive higher levels of perceived contract

accountability effectiveness. This is a key finding suggesting that government program managers

and contractors give systematically different accounts to a third party about the degree to which

they felt the contractors were being held accountable by government agencies.

Regression analysis pertaining to the second dependent variable – government reveals

problems and effectively manages the contract – is presented in tables 6a and 6b. As explained

above, the dependent variable was regressed, consecutively, on the total number of performance

measures and each of the fifteen individual performance measures (along with the control

variables). This analysis also suggests that while some measures have a positive association with

the dependent variable, others have a negative effect. The overall scope of performance

measurement – the sum of all measures – had a positive effect on the government’s ability to

reveal problems and effectively manage contracts (Model 5). Measuring impact of services on

clients and service equitability, measuring service timeliness and disruptions, as well as

specifying the details of service provision had a positive association with the dependent variable

(models 6, 8, 9, 10, and 11). Meanwhile, measuring client satisfaction and monitoring informally

had a negative effect on the government’s ability to reveals problems and effectively manage

contracts (models 7 and 12). Similar to models 1 through 4, collaborative development of

performance measurement practices had no effect on the second measure of perceived

accountability effectiveness. Nonprofit status of contracted organizations reduced government’s

17

perceived ability to reveal problems and effectively manage contracts. The contractors’ financial

dependency was associated with a lower perceived accountability effectiveness. Monitoring done

directly by government agencies through inspections, observations, and other methods appeared

to improve perceived accountability. Government’s in-house professional capacity also

significantly improved accountability effectiveness. In two regressions, relationship length had a

positive association with the dependent variable. Finally, in two models (9 and 10), public sector

respondents had a significantly lower perception of accountability effectiveness than their private

counterparts.

DISCUSSION

Performance measurement activities are complex tasks that require specialized

knowledge and significant resources. Therefore, understanding the benefits of these activities is

useful to ensure a more efficient use of government funds. This study focuses on identifying

specific performance measurement approaches that impact government agency’s ability to keep

its contractors accountable. It attempts to address a question raised in the literature: “Is

monitoring always good and the more the better?” (Yang, Hsieh, and Li 2009 681). Our findings

suggest that the answer to this question is not simple. Some measures appear to have no

significant impact on government’s ability to collect good data and effectively manage contacts.

Furthermore, while some measures have a positive significant effect on perceived accountability

effectiveness, others have a negative impact. Notably, the overall scope of performance

measurement does in fact improve public agency’s ability to reveal problems and effectively

manage contracts. More diverse performance measurement activities – those involving multiple

and complementary ways of evaluating the contractors’ work – appear to improve the agency’s

ability to detect performance problems and have a perception of effective contract management.

While this does not necessarily suggest that “complete” contracting is the answer, it certainly

18

means that by diligently investing in multi-dimensional, diverse, and complex performance

measurement systems the agencies will receive a payoff in terms of improved perceived

accountability. This, perhaps, is the most important finding of this study relevant for the

practitioners: it suggests the importance of investing more time and effort in the design of multi-

faceted performance measurement systems. Nonetheless, some performance measures can

evidently be more effective than others.

When examining the timeliness and accuracy of information provided by the contractor

as well as the contractor’s propensity to comply with the requirements, monitoring service costs

has a positive impact. Privatization in the U.S. has been promoted primarily as a cost-cutting

tool, and it is not surprising that the examination of costs plays a central role in the oversight

process. While many government agencies may lack the capacity to effectively evaluate quality

and other non-financial aspects of performance, the review of financial data is a more traditional

and straightforward contract management procedure. Qualitative data provided by the

respondents in this study supports this assumption – several respondents described receiving

regular financial reporting from the contractor as a key element of contract evaluation - one that

essentially ensures that the contract continues as intended.

Notably, monitoring quality has a negative effect of the respondent’s perception of the

contractor’s compliance and timely and accurate information sharing. This may suggest a

number of things: a lack of capacity to collect data and evaluate service quality (Gianakis 2002),

poorly designed quality monitoring tools, as well as ambiguous, unsatisfying or contradictory

information on service quality (Frederickson and Frederickson 2006; Kravchuk and Schack

1996; Nicholson-Crotty, Theobald, Nicholson-Crotty 2006; Radin 2006). The general

performance measurement literature suggests that broad and ambiguous objectives of public

programs often make it difficult to measure success, and they introduce political tradeoffs

19

between multiple measures of quality, costs and others (Amirkhanyan, Kim and Lambright 2008;

Blasi 2002; Callahan and Kloby 2007; Frederickson and Frederickson 2006; Kravchuk and

Schack 1996). Obviously, it is easier to receive timely and accurate data on costs, than on

quality. In this connection, several authors note that performance measurement is often

inaccurate due to the public managers’ failure to invest into long-term and continuous evaluation

that can provide the “helicopter view” of performance (Blasi 2002; Bouckaert and Peters 2002;

Courty and Marschke 2007).

Similarly, this study finds that the use of informal monitoring techniques in the oversight

process has a negative effect on both measures of perceived accountability effectiveness:

receiving timely and accurate performance information and using it to reveal problems. These

findings are important for the discourse on trust-based relational contract management. The use

of informal monitoring may suggest a lack of formally designed oversight systems. Our findings

may also reflect respondents’ frustration with the inadequacy of informal contract monitoring

tools. DeHoog (1990) warns that cooperative contract management models that are based on

trust and informal ties may become more important than programmatic success, limit objectivity

and lead to collusion. Our findings may also suggest a missing variable in the model. Some

unobserved and potentially problematic characteristics of the contractor may prompt the public

agency to supplement the formal monitoring tools with the informal ones. This study suggests

that these strategies do not appear to improve perceived accountability effectiveness.

Notably, the monitoring of service timeliness and disruptions is also important for both

measures of effective contract accountability. Both of these items pertain to the process of

service delivery. They are relatively easy to track and may also be easy to interpret. A failure to

fix a parking meter undermines the timeliness of several other work orders, and clearly suggests

the contractor’s failure to perform well. Many contracts in our sample involved monitoring

20

systems (some computerized) that would promptly inform the government agency of any service

disruptions. This type of data serves as a first signal of a problem and is therefore critical to

effectively understand the status of service delivery and to correct deficiencies. Importantly,

these findings suggest that monitoring some aspects of service delivery process rather than the

outcomes can provide valuable information for accountability effectiveness which would not

necessarily be achieved in the typical performance-based contracts.

Regression analysis suggests that assessing the impact of service delivery on clients is

positively associated with the perception of effective contract monitoring. This measure is

designed to detect the outcomes or the end result of the contractors’ activities. This information

is what many external and internal stakeholders ultimately care about in contract

implementation. For instance, the parents’ dissatisfaction with the quality of care their children

receive at a child care center will have important implications for the contracting Head Start

Agency’s contract management practices. A nursing facility’s failure to ensure high-quality long

term care (e.g., follow practices geared towards minimizing the prevalence of bed sores or the

use of psychotropic drugs) will be the primary factors to be considered by the government

agency while managing its contractor. Thus, it is not surprising that understanding the impact of

services on the clients is helpful in effective management and implementation of these contracts.

In the light of these findings, however, it is interesting that evaluation of client satisfaction

actually has a negative effect on the dependent variable. On possible interpretation of these

findings may be that while the data on client impact may be generated by the public agency and

reflects some broader impacts and implications of service delivery (e.g., general trends in public

health improvement, environmental conditions, and others), client satisfaction surveys may

reflect some discontent and complaints related to service delivery. Receiving data directly from

21

the clients (i.e., from a separate stakeholder) may certainly complicate the overall perception of

contract management effectiveness.

Importantly, evaluating whether or not services were delivered in a fair and equitable

manner is also positively associated with the perception of government’s ability to reveal

performance problems and effectively manage contracts. The use of this measure may reflect

some attention to the core public sector values of fair distribution of resources and may suggest

higher levels of commitment to good oversight among these public agencies. Assessing fairness,

access and equitability of service delivery may be a signal of mission-orientation and better

articulation of contract priorities which in turn improves accountability.

Finally, specifying the detailed process for service delivery – providing directions on

precisely how, when and by whom services should be provided – is a performance monitoring

practice associated with better perceived accountability. In this connection, Behn (2003, 594)

writes about performance measures: “Do not be fooled. These guidelines are really requirements,

and these requirements are designed to control. The measurement of compliance with these

requirements is the mechanism of control.” This is especially true when an agency attempts to

maximize the specification of service delivery parameters. Thus, government agency’s

propensity to “control” its contractors by developing a more complete specification of how,

when and by whom the service should be delivered improves the contractor’s compliance and the

agency’s ability to obtain clear and informative data, detect problems, and act upon them.

Process specification also undoubtedly reflects a higher level of in-house service delivery

proficiency which allows the agency to spell out the expectations and effectively monitor their

attainment.

This study finds that the way performance measures are created (i.e., whether they are

created unilaterally by a government agency and imposed upon a contractor, or are a result of

22

negotiations and clarifications) has no effect on accountability effectiveness. Thus, performance

measures, no matter how they are created, can serve as useful tools that can help generate useful

information on performance and prompt the agency to use sanctions. While not undermining the

value of collaborative practices in the course of contracting, this study suggests that an agency

may unilaterally develop and enforce a set of measurement techniques that would help achieve

higher levels of contractor perceived accountability.

This paper does not argue that only seven measures are useful in improving

accountability or that other measures are of no value. In fact, the findings pertaining to the scope

of performance measurement suggest that the more measurement is done the more accountability

can be achieved. Rather, this analysis shows that different measures can have a different effect

on the managerial outcome considered here. This analysis also suggests that the process of

performance measurement should not be static. It is hard to develop a perfect system at the onset

of the contracting relationship. At that point, a greater scope of performance measurement may

be advisable. This will ensure higher accountability while the contracting agency develops a

better understanding of the most optimal performance measurement “package.” The latter may

be different for different service fields and levels of government. Thus, the findings of this study

provide further motivation for contract managers to work continuously on improving their

performance measurement and monitoring practices.

Results of this analysis also suggest that, aside from the scope and the type of

performance measurement activities, the perceived level of accountability may be determined by

a range of other factors. First, government agencies involved in direct monitoring of their

contractors - inspections, observations, or site visits – are able to achieve greater accountability

even after controlling for service measurability. On the other hand, data that are self-reported by

the contractor has the opposite effect. This is hardly surprising: the delegation of monitoring to a

23

vendor or a third party can produce informational silos or exacerbated principal-agent issues.

Meanwhile, direct monitoring often increases the likelihood of developing informal ties, having

first-hand knowledge of the implementation process (often, by virtue of being “on site” with the

contractor), and eventually results in more accurate and timely reporting and correction of

problems. Similarly, relationship length allows both parties to develop a common language and

to create a shared understanding of the contractor’s capabilities and the government’s propensity

to impose sanctions – these factors can facilitate a more cooperative contract evaluation process.

Potentially, this provides some evidence on the effect of stability and relationships (though, not

necessarily informal) on the effectiveness of oversight.

This research also suggests that government respondents are less likely to have a

perception of high accountability effectiveness. While the contractors, who are often inundated

with government’s monitoring activities, may be satisfied with the extent of evaluation,

government employees may feel that their efforts are not sufficient. Public managers may be

dissatisfied with the accuracy of data and their overall ability to keep the contractors

accountable. This is consistent with the contracting literature that finds inadequate monitoring

capacity among public managers overseeing state and local contracts (Amirkhanyan 2009; Van

Slyke 2003). This interpretation is reinforced by the findings of this study: as respondents’

contract management experience goes up their perception of accountability effectiveness goes

down. Possibly, as someone’s contract management experience increases so does their

understanding of service ambiguities and complexities, and appreciation of the limitations of

performance measurement. This undermines the public respondents’ perception of accountability

in government contracts. These results are important in the light of the earlier findings about the

vendors’ sentiments suggesting high degree of informational asymmetry (“They think they

[measure service quality]… But I really don’t think they do it”). The contract managers’

24

skepticism about accountability effectiveness undermines our earlier assertion that the public

managers may have an inaccurately optimistic view of their monitoring efforts. Instead, these

findings support the results of several past studies highlighting the contract managers’ frustration

with their inadequate monitoring capacity, despite a wide scope of responsibilities and actions

(Amirkhanyan 2009; Van Slyke 2003). The governments’ skepticism about their monitoring

capacity found in this paper suggests that this is an important area for the future research,

especially in the context of politically motivated contracts.

Supporting some of the basic tenets of contract management theory, this study suggests

that competitive procurement of contracts has a positive effect on the contractor’s compliance

with performance monitoring. Competitive private markets may pressure the contractors to

provide the information necessary to maintain the contract. Similarly, government’s in-house

expertise and professional capacity improves effective contract monitoring, while the lack of

thereof undermines public agency’s ability to understand the intricacies of service delivery and

reveal performance problems. This finding suggests that maintaining some degree of service

implementation capacity is important for effective contract monitoring.

Interestingly, we find that nonprofit status of the contractors is associated with a

significant decrease in perceived accountability effectiveness. On the one hand, nonprofit

organizations are often perceived are more trustworthy than their for-profit counterparts due to

their mission-driven nature and the fact that they re-invest profits towards their organizational

missions (Hansmann, 1987). These and other factors may decrease their opportunistic behavior

and increase compliance and cooperation in the oversight process. Amirkhanyan (2009) finds

that nonprofit service contractors are more likely to collaborate with public agencies in the

oversight process, and Van Slyke (2009) suggests that nonprofits may be initially perceived as

more trustworthy. On the other hand, contracting research suggests that public managers are

25

aware of the performance problems, lack of management capacity, financial abuses, and political

behavior prevalent in the nonprofit sector (Arenson 1995; Grimaldi and Trescott 2008; Reaves

2001; Hansmann 1986; Johnston and Romzek 1999; Prager 1994; Rose-Ackerman 1996). Our

findings may suggest lower capacity of nonprofit organizations to comply with the monitoring

process, and the nature of services delivered by nonprofit organizations may be characterized by

a higher degree of complexity and ambiguity that also undermines accountability effectiveness.

Finally, one of the most puzzling findings of this analysis pertains to the negative effect

of the contractors’ financial dependency on the perception of accountability effectiveness. One

would expect that the contractors’ dependency on a single contract would translate into their

willingness to cooperate in the evaluation process which would eventually improve the

likelihood of providing timely and accurate information and resolving performance problems.

What our findings may suggest is that the financial importance of a contract gives some vendors

an incentive to reduce transparency and evade government oversight. When more is at stake for a

contractor, the situation may create additional tensions and conflict, and the contractors may be

reluctant to cooperate out of fear of sanctions. Contractors that are dependent on a specific

government contract rather than those relying on diversified revenues sources, are also more

likely to be smaller and less professionalized which may affect the perception of accountability

effectiveness.

Another intriguing finding pertains to the negative effect of “hard” or easily measurable

services on the contractors’ compliance with the monitoring, as well as with the timely and

accurate information provided to the government agency. This finding may in fact imply some

procedural distinctions associated with the “hard” (i.e, tangible or more easily measurable)

services. In the qualitative portion of the interviews, several public managers discussed direct

supervision and inspections as important monitoring techniques in the fields of construction,

26

maintenance, waste removal and others. Potentially, quality of information reported by the

contractors in these fields may be inadequate and more direct inspections are used in such

contracts. Service type does not, however, affect the government’s ability to reveal problems and

effectively monitor these contracts.

While this study contributes to our understanding of which performance measurement

strategies matter for contract implementation, several unanswered questions remain. First, this

study focuses on the effect of performance measurement on one aspect of managerial

effectiveness – contractor accountability. The latter is important, as Cohen and Eimicke (2008,

155) argue: “innovation and customer needs may very well be less important than accountability

and transparency.” Nonetheless, public agencies pursue multiple goals when measuring

organizational performance (Smith and Larimer 2004). Therefore, future studies should assess

the impact of performance measurement on diverse programmatic and stakeholder outcomes

(including creativity and customer needs). In particular, this will allow us to see which measures

are more suitable for “motivating”, “celebrating”, “learning” and “improving”, and for achieving

other organizational goals. Measures found to be less important for contractor accountability

may in fact be critical for goal establishment, budgeting, motivation, promotion, and learning

(Behn 2003; Ho 2006). Agencies pursuing performance measurement should clarify the

objectives of their efforts and refine their contract monitoring systems depending on these

objectives. Additional and related research directions include comparing the vendors' and the

purchasers' perceptions of accountability effectiveness on the same contracts with a third

assessment done by a researcher. Understanding the relationship between these perceptual

measures of accountability effectiveness and actual program outcomes is important to adequately

assess the importance of performance monitoring efforts. More importantly, it is an open

question still not just whether perceived accountability effectiveness is related to "program

27

outcomes" but also whether actual accountability effectiveness is related to either of those

variables.

Furthermore, while this research approaches the use of performance measures as a

dichotomy, past literature suggests that such efforts may have different levels of intensity and

accuracy (Blasi 2002). Future studies should investigate not only the scope but also the intensity

and accuracy of various performance measurement approaches, and examine their effect on the

programmatic outcomes. Specifically, large-scope performance measurement may allow public

managers to effectively evaluate contractor performance, but it can also divert the contractor’s

resources and attention away from program implementation towards regulatory compliance, and

may eventually undermine service outcomes. In addition, while this manuscript tests the direct

effects of various performance measures, these practices may not only have different direct

effects, but also interact together to have interactive effects. Finally, different performance

measures (e.g., service disruptions) may have a different meaning and relevance in different

fields, and therefore is it important to investigate how performance measures are applied for

certain goods and services with similar production and consumption characteristics.

APPENDIX A. Mini-case studies illustrating specific services across sectors. NONPROFIT FOR-PROFIT

HIG

H s

cop

e o

f P

erfo

rma

nce

Mea

sure

men

t

(PM

) (1

1-1

5 m

easu

res)

Contract A is delivered by a homeless shelter

monitored by a county government using monthly

expenditures, client age, gender, ethnicity, length

of stay, employment, income, savings upon

arrival and departure, and data on applications

and admissions. All fifteen measures listed in the

questionnaire, except timeliness and customer

satisfaction, are used in the monitoring process.

The contractor submits reports and participates in

monthly sessions to discuss programmatic issues

and the state of homelessness in the county.

Recently, the reporting system has been enhanced

with new information technology. Since the

contractor feels the new system fails to

incorporate descriptive information about their

work and long-term client outcomes, they

voluntarily provide this information in their

informal communication with the county.

Contract D is with a for-profit organization

performing maintenance of solid waste

management equipment. The monitoring officer

prepares a detailed daily list of tasks for the

contractor, directly observes the contractor’s work,

daily selects a sample of trucks for a closer inspection

using a special evaluation checklist developed for

each type of repair, and seeks the truck drivers’

feedback to determine their satisfaction with the

quality of repairs and to illuminate any unaddressed

problems. With the exception of the contractor’s

reputation and equitable access to services, all

performance measures used in the interview guide are

utilized by the government agency. Recently, the

government conducted a longitudinal study comparing

the current and the previous contractors’ performance

and found some improvement in the incidence of

performance problems.

28

MO

DE

RA

TE

sc

op

e o

f P

M

(6-1

0 m

easu

res

)

Contract B is for nursing and medical services

for the juveniles referred by a local

correctional department. Through regular

reports and formal meetings, the monitoring

officer tracks the timeliness, the costs, the

workload, as well as multiple other quantitative

indicators and qualitative accounts of the clients’

health status and the types of services provided.

The officer does not use any informal channels of

communication to monitor care. He does not

examine the contractor’s reputation, client

satisfaction with services, or whether services are

delivered equitably, and according to the laws

and regulations in the field. The officer also

argues he is unable to detect the short-term or

long-term effects of care on the client health.

Contract E provides nursing services for

incarcerated children. The monitoring officer

maintains extensive communication with the

contractor, but is frustrated that most information

originates from self-reporting or whistle-blowing.

Timeliness, disruptions in service delivery, and the

contractor’s workload are among the monitored

aspects of performance. Indirect measures of care

quality are also used. No consistent assessment of

costs or correspondence of care provision to the

regulations in the field is conducted. Similarly, client

satisfaction, equitable access to care, and the

contractor’s reputation are not assessed. Monitoring

officer points to his inability to evaluate “how the

contractor does due diligence,” and cites the need to

reiterate contract requirements due to the contractor’s

staff turnover.

LO

W s

cop

e o

f P

M

(0-5

mea

sure

s re

port

ed)

Contract C is delivered by a nonprofit foundation

that facilitates quality improvement and

develops various population evaluation

frameworks for a local government health

department. The contractor is not aware of any

formal evaluation of quality or any other aspect

of their work, with the exception of timeliness of

submitted quality-improvement materials and

continuity in their work. Communication between

the agency and the contractor focuses on

eliminating the factors that hinder the contractor’s

ability to provide the expected deliverables on

time (e.g., those dealing with receiving the

necessary data from the government agency in

order to develop quality assessment and

improvement strategies and tools).

Contract F is a small for-profit therapy program

conducting substance abuse counseling. When asked

about government monitoring, the contractor notes

“[a]s hard as it is to believe, they do nothing.” The

contractor submits brief reports with a basic overview

of its work, however there is no feedback or

indication that these reports are reviewed. As a part of

internal evaluation the contractor conducts

professional client assessment the results of which are

voluntarily shared with the government agency (also,

with no feedback). Communication is generally

initiated by the contractor who raises concerns

regarding specific clients’ noncompliance. The

contractor indicated that only one of fifteen items is

used by the agency: the agency interacts with the

clients and obtains some informal feedback.

APPENDIX B

Interview Questions Utilized to Examine the Prevalence and the Process of Collaboration

1. Today I would like us to talk about performance evaluation and measurement, specifically, about any

kind of information that you might use to make sure that your contractor is complying with your

expectations and doing their job well. Some of these performance measures can be more formal and

quantitative (e.g., reporting the number of service units produced every week). Other measures can be

more informal (e.g., informally discussing service provision details). For the following questions

please choose one of the following answers: (1) yes, (2) no, (3) don’t know/don’t recall/refuse to

answer. In this contract, do you collect, monitor, or evaluate information on:

a. cost-effectiveness of contracted services?

b. quality of services provided by the contractor?

c. contractor’s workload (e.g., number of clients served, units of services provided, # hours of

work)?

d. the impact that services have on clients or service-recipients?

e. customer satisfaction?

f. contractors’ ability to provide equitable access to services without any discrimination (e.g.,

based on income, gender, or race)?

g. compliance of service provision with the law?

h. timeliness of service delivery?

29

i. service continuity or any disruptions in service delivery?

j. do you specify the detailed procedures for service delivery; in other words, precisely how

services should be delivered, and by whom?

k. do you use any quantitative measures of performance (for instance number of clients served,

# services provided, quantifiable impact on the clients’ status)?

l. do you use any qualitative, descriptive information on your contractor’s performance?

m. do you use any informal ways of obtaining performance information, such as through an

informal conversation with a client, contractor staff or a third party?

n. are the performance measures that you use tailored to this particular contractor (i.e., they

wouldn’t be used for another contractor)?

o. do you collect information on the reputation of your contractor formally or informally?

p. do you evaluate your contractor’s performance in any other ways that I have not listed?

2. Have you ever asked for contractor’s input on the performance measures that are used in this

contract? If yes, ask: Why was this done? Can you explain how this was done?

3. Has your contractor ever attempted to negotiate or discuss with you how their performance should be

measured or evaluated? If yes, ask: When did this happen? When this happened, how did your

agency respond?

4. Have you incorporated any performance measures that were modified in response to contractor’s

comments or proposed by your contractor?

5. Do you believe that your overall communication with the contractor influenced the type of measures

that you use? If yes, ask: In what way?

6. Has the contractor ever asked you for clarifications on the performance information that you

requested? If yes, ask: How did your agency respond?

7. Have you ever asked your contractor for clarifications on their performance information that was

gathered about them? If yes, ask: how did they respond?

APPENDIX C. Measurement of independent variables

Interview questions used to create each

variable Collaboration Determinants and Created

Measures

I would like to seek your input on the effects

of contractors’ nonprofit or forprofit status on

the overall contractual relationship. Is your

contractor a nonprofit or a forprofit

organization?

Nonprofit Status. Variable was created to indicate

nonprofit status of the vendor in the discussed

contracting arrangement (dummy variable).

N/A (respondent status recorded by the

primary investigator at the time of the

interview).

(A). Government Respondent. In the data, records

pertaining to government monitoring officers were

assigned the value of 1, while respondents

representing the vendors were assigned the value of 0.

We are here to discuss the contract with

________. Could you describe for me, very

briefly, what kind of services does this

contractor provide?

(B). Hard Services. Responses were categorized into

“hard” (easily measurable) services: IT, construction,

maintenance, public works, planting and plant control,

food supply and quality monitoring, animal care,

janitorial, translation, and recreational (camps, dance

lessons). “Soft” (hard-to-measure) services: long-term

care, medical, nursing care, health management,

mental health, psychological consultation, arts

therapy, programs for women and children,

consulting, evaluation and training, criminal justice,

animal care, substance abuse, and homelessness

(dummy variable). Using Wilson’s classification,

“hard” services also corresponded to those provided

30

by coping and procedural agencies, while “soft” ones

corresponded to s-called craft agencies (no production

agencies were involved in the analysis).

Some contractors’ financial health depends

solely on the government contract. Other

organizations are more fiscally independent, and

rely on other sources of revenues. Is your

contractor (a) financially very dependent on

your funding, (b) somewhat dependent, (c)

somewhat independent, (d) financially very

independent, (e) don’t know.

(C). Vendor’s financial dependency on the contract Variable “Contractor’s financial dependency” was

coded as one for responses (a) financially very

dependent on government funding and (b) somewhat

dependent, and zero for all other options (dummy

variable).

Do you believe that your contractor has its

own internal ways to evaluate its performance?

(C). Contractor uses internal performance

measures. Variable “internal measures” was created

and coded as one for affirmative responses to the

question (dummy variable).

Does your contractor have a unique expertise

that is difficult to find elsewhere? (Probe: Are

there any other organizations in this area that

provide similar services? Is this market very

competitive?)

(D). Contractor has unique expertise. Variable

“unique expertise” was created for affirmative

responses (dummy variable).

Did you go through the process of

competitive bidding for this contract? (D). Competitive bidding used. Variable

“competitive bidding” was coded as one for

affirmative responses (dummy variable).

Some contracts exist in the fields that

undergo rapid changes in service needs, service

technology, suppliers, or funding. Other

contracts exist in more stable, less uncertain

environments. Where would you place this

contract on this continuum between very

dynamic and very stable environments? (a)

Very dynamic, (b) Somewhat dynamic, (c)

Somewhat stable, (d) Very stable.

(D). Dynamic versus stable environment. Variable

“dynamic environment” was coded as one for

responses (a) Very dynamic and (b) Somewhat

dynamic (dummy variable).

When was this contract initiated? Have you

been working with this contractor before? If

yes, ask: In what capacity? For how long?

(E). Long-term vs. short-term relationship.

Variable “relationship length” was created reflecting

the number of years (interval-ratio variable). Eight

cases in the data had missing values and median

values were imputed in order to retain this variable

and maximize the size of the sample.

Some practitioners say that contractual

relationships often begin as more rigid (more

formal) and over time evolve into relationships

that are based on trust. Has this been the case

with this contract?

(E). Perceived goal congruence or trust Variable “trust” was coded as one for responses

confirming that the relationship was presently

characterized by trust between the government agency

and the vendor (dummy variable).

In some cases government agencies collect

and monitor all the information pertaining to

contractor’s performance directly. In other

cases, governments use information collected

and provided by the contractor (so-called self-

reported measures). There are also agencies

that use third parties to collect information and

(E). Monitoring: Self-reported measures used; (E). Monitoring: government collects performance

information; (E). Monitoring: third party monitoring is used Three dummy variables (“self-reporting”,

“government inspection”, “other monitors”) were

created based on the descriptive answer to the

31

do the monitoring (for instance the clients or 3rd

party inspectors). Which strategy do you use? question.

Do you publicize the information pertaining

to the performance of the contractor? If yes, ask:

In what way?

(E). Contractor performance publicized. As an

additional measure of third-party monitoring, variable

“publicized performance” was created and affirmative

responses to the second question were coded as one

(dummy variable).

Do you have professionals among your staff

who can thoroughly understand the nature of the

service delivered by your contractor (individuals

with similar education, degrees, professional

norms, etc.)?

(F). In-house capacity to deliver the service.

Variable “in-house professional capacity” was created

and coded as one for affirmative responses.

Do you think contractors should be engaged

in the development of performance measures to

oversee their own work? If yes or no, ask: Can

you explain, why?

(G). Contractor’s participation in performance

evaluation is perceived as desirable. Variable

“contractors should be engaged” was created and

coded as one for affirmative responses to question 17.

Note for the interviewer: verify respondent’s

employment status. Do you currently serve as

______________? How long have you been

working in this position? How long have you been involved in

managing contracts?

(G). Respondent’s work and contract management

experience. Two variables “work experience” and

“contract management experience” were created to

reflect the number of years. One case had a missing

value, and the average for the whole sample was

calculated and imputed in order to retain this variable

and maximize the sample. REFERENCES

Allen, P. 2002. “A Socio-legal and Economic Analysis of Contracting in the NHS Internal Market Using

a Case Study of Contracting for District Nursing.” Social Science & Medicine 54(2): 255-266.

Amirkhanyan, A.A., Kim, H.J., & Lambright, K.T. (2007). Putting the pieces together: comprehensive

framework for understanding the decision to contract out and contractor performance.

International Journal of Public Administration (IJPA), 30, 699-725.

Amirkhanyan, Anna A., Hyun J. Kim, and Kristina T. Lambright. 2008. Does the public sector outperform

the nonprofit and for-profit sectors? Evidence from a national panel study on nursing home

quality and access. Journal of Policy Analysis and Management (JPAM) 27 (2): 326-53.

Amirkhanyan, Anna A. 2009. Collaborative Performance Measurement: Examining and explaining the

prevalence of collaboration in state and local government performance monitoring. Journal of

Public Administration Research and Theory (JPART) 19 (3): 523-54.

Amirkhanyan, Anna A. 2010. Monitoring across sectors: Examining the effect of nonprofit and forprofit

contractor ownership on performance monitoring in state and local government contracts. Public

Administration Review, 70(5): 742-755.

Ammons, David N., and William C. Rivenbark. 2008. Factors Influencing the Use of Performance Data to

Improve Municipal Services. Public Administration Review 68 (2): 304-18.

Arenson, K. W. (1995). Ex-United Way leader gets seven years for embezzlement. New

York Times, June 23. Arrow, Kenneth J. The Economics of the Agency. In Principals and Agents: The Structure of Business;

Pratt, John W., Zeckhauser, Richard J., Eds.; Harvard Business School Press: Boston, 1984; 1–35.

Behn, Robert D. 2003. Why Measure Performance? Different Purposes Require Different Measures.

Public Administration Review 63 (5): 586-606.

Beinecke, R.H. and R. DeFillippi. 1999. “The Value of the Relationship Model of Contracting in Social

Services Reprocurements and Transitions: Lessons from Massachusetts.” Public Productivity and

Management 22(4): 490-501.

32

Berman, Evan M., and XiaoHu Wang. 2000. Performance Measurement in U.S. Countries: Capacity for

Reform. Public Administration Review 60 (5): 409-20.

Blasi, Gerald J. 2002. Government Contracting and Performance Measurement in Human Services.

International Journal of Public Admin 25 (4): 519-38.

Boukaert, Geert and B. Guy Peters. 2002. Performance Measurement and Management: The Achilles' Heel

in Administrative Modernization. Public Performance & Management Review (25) 4: 359-62.

Boyne, George A. 1998. Bureaucratic Theory Meets Reality: Public Choice and Service Contracting in

U.S. Local Government. Public Administration Review 58 (6): 474-84.

Brown, Trevor L., and Matthew Potoski. 2003. Contract-Management Capacity in Municipal and County

Governments. Public Administration Review 63 (2): 153-64.

------. 2006. Contracting for Management: Assessing Management Capacity Under Alternative Service

Delivery Arrangements. Journal of Policy Analysis and Management 25 (2): 323-46.

Brown, Trevor L., Matthew Potoski, and David M. Van Slyke. 2006. Managing Public Service Contracts:

Aligning Values, Institutions, and Markets. Public Administration Review 66 (3): 323-31.

Byrnes, Patricia, Mark Freeman, and Dean Kauffman. 1997. Performance Measurement and Financial

Incentives For Community Behavioral Health Services Provision. International Journal of Public

Administration 20 (8-9): 1555-78.

Callahan, Kathe, and Kathryn Kloby. 2007. Collaboration Meets the Performance Measurement Challenge.

Public Manager 36 (2): 9-24.

Campbell, D. and D. Harris. 1993. “Flexibility in Long-term Contractual Relationships: The Role of Co-

operation.” Journal of Law and Society 20(2): 166-191.

Chan, Hon S. and David H. Rosenbloom. (2010). Four challenges to accountability in contemporary public

administration: Lessons from the United States and China. Administration & Society,

forthcoming.

Cohen, Steven and William Eimicke. 2008. The Responsible Contract Manager: Protecting the Public

Interest in an Outsourced World. Washington, DC: Georgetown University Press.

Courty, Pascal, and Gerald Marschke. 2007. Making Government Accountable: Lessons from a Federal

Job Training Program. Public Administration Review 67 (5): 904-16.

Dalehite, Esteban. 2008. Determinants of Performance Measurement: An Investigation into the Decision to

Conduct Citizen Surveys. Public Administration Review 68 (5): 891-907.

Davis, P. (2007). The effectiveness of relational contracting in a temporary public organization: intensive

collaboration between an English local authority and private contractors. Public Administration,

85, 2, 383–404.

De Vries, Michiel. 2007. Accountability in the Netherlands: Exemplary in its Complexity. Public

Administration Quarterly 31 (3/4): 480-507.

DeHoog, R. 1990. “Competition, Negotiation, or Cooperation: Three Models for Service Contracting.”

Administration & Society 22(3): 317-340.

Dilger, Robert J., Randolph R. Moffett, and Linda Struyk. 1997. Privatization of Municipal Services in

America’s Largest Cities. Public Administration Review 57 (1): 21-6.

Eisenhardt, K. M. Agency Theory: An Assessment and Review. Academy of Management Review 1989,

14, 57–74.

Frederickson, David G. and George Frederickson. 2006. Measuring the Performance of the Hollow State.

Washington, DC: Georgetown University Press.

Gianakis, Gerasimos A. 2002. The Promise of Public Sector Performance Measurement: Anodyne or

Placebo? Public Administration Quarterly 26 (1/2): 35-64.

Goldsmith, Stephen and William D. Eggers. (2004). Governing by Network: The New Shape of the Public

Sector. Washington, DC: Brookings Institution Press.

Grimaldi, J.V., & Trescott, J. (2008). Smithsonian official resigned in wake of

ethics probe: Internal report cited Latino center leader for multiple violations.

Washington Post, April 15.

33

Hansmann, H.B. (1986). The role of nonprofit enterprise. In S. Rose-Ackerman (Ed.), The

Economics of Nonprofit Institutions: Studies in Structure and Policy (pp. 57-86). New

York: Oxford University Press

Hansman, H. (1987). Economic theories of nonprofit organizations. In W.W. Powell (Ed.), The

nonprofit sector: A research handbook (pp. 27-42). New Haven, CT: Yale University

Press. Hefetz, Amir, and Mildred Warner. 2007. Beyond the Market versus Planning Dichotomy: Understanding

Privatization and Its Reverse in U.S. Cities. Local Government Studies 33(4): 555–72.

Heikkila, Tanya, and Kimberley R. Isett. 2007. Citizen Involvement and Performance Management in

Special-Purpose Governments. Public Administration Review 67 (2): 238-48.

Heinrich, K.J. (1999). Do Govenrment Bureaucrats Make Effective Use of Performance Management

Information? JPART, 9,3, 363-393.

Heinrich, Carolyn J. 2002. Outcomes-Based Performance Management in the Public Sector: Implications

for Government Accountability and Effectiveness. Public Administration Review 62 (6): 712-25.

Ho, Alfred Tat-Kei. 2006. Accounting for the Value of Performance Measurement from the Perspective of

Midwestern Mayors. Journal of Public Administration Research and Theory 16 (2): 217-37.

Ho, Shih-Jen Kathy, and Yee-Ching Lilian Chan. 2002. Performance Measurement and the

Implementation of Balanced Scorecards in Municipal Governments. The Journal of Government

Financial Management 51 (4): 8-19.

Hodge, Graeme A. 2000. Privatization: An International Review of Performance. Boulder: Westview

Press.

Holzer, Marc, and Kathryn Kloby. 2005. Public Performance Measurement: An Assessment of the State-

of-the-Art and Models for Citizen Participation. International Journal of Productivity and

Performance Management 54 (7): 517-32.

Johnston, J. M. and Girth, A. M. "Public service contracting and weak competition: Assessing the costs

and impacts of strategies used to “manage" provider markets" Paper presented at the annual

meeting of the Midwest Political Science Association 67th Annual National Conference, The

Palmer House Hilton, Chicago, IL. 2010-06-05 from

http://www.allacademic.com/meta/p361722_index.html

Johnston, Jocelyn M. and Barbara S. Romzek. (1999). Contracting and accountability in state Medicaid

reform: Rhetoric, theories, and reality. Public Administration Review 59(5): 383-399.

Kelly, Janet M., and David Swindell. 2002a. A Multiple-Indicator Approach to Municipal Service

Evaluation: Correlating Performance Measurement and Citizen Satisfaction across Jurisdictions.

Public Administration Review 62 (5): 610-21.

------. 2002b. Service Quality Variation Across Urban Space: First Steps Toward a Model of Citizen

Satisfaction. Journal of Urban Affairs 24 (3): 271-88.

Kelman, Steven. (2002). Contracting. In Lester A. Salamon, ed. The Tools of Government: A Guide to the

New Governance. New York: Oxford University Press.

Kopczynski, Mary, and Michael Lombardo. 1999. Comparative Performance Measurement: Insights and

Lessons Learned from a Consortium Effort. Public Administration Review 59 (2): 124-24.

Kravchuk, Robert S., and Ronald W. Schack. 1996. Designing Effective Performance-Measurement

Systems Under the Government Performance and Results Act of 1993. Public Administration

Review 56 (4): 348-58.

Linder, Jane C. 2004. Outsourcing for Radical Change: A Bold Approach to Enterprise Transformation.

New York: American Management Association.

Marvel, Mary K. and Howard P. Marvel. (2007). Outsourcing oversight: A comparison for in-house and

contracted services. Public Administration Review 67(3): 521-30.

Melkers, Julia, and Katherine Willoughby. 2005. Models of Performance-Measurement Use in Local

Governments: Understanding Budgeting, Communication, and Lasting Effects. Public

Administration Review 65 (2): 180-90.

34

Meyers, Marcia K., Norma M. Riccucci, and Irene Lurie. (2001). Achieving goal congruence in complex

environments: The case of welfare reform. Journal of Public Administration Research and Theory

11:165–201.

Milgrom, Paul and J. Roberts. 1992. Economics, Organizations, and Management. Englewood Cliffs,

NJ: Prentice Hall.

Milward, H. Brinton, and Keith G. Provan. 2000. Governing the Hollow State. Journal of Public

Administration Research and Theory 10 (2): 359-79.

Moynihan, Donald P. 2005. Why and How Do State Governments Adopt and Implement “Managing for

Results” Reforms? The Journal of Public Administration Research and Theory 15 (2) 219-43.

Nicholson-Crotty, Sean, Theobald, Nick A., and Jill Nicholson-Crotty. 2006. Disparate Measures: Public

Managers and Performance-Measurement Strategies. Public Administration Review 66 (1): 101-

13.

Prager, J. (1994). Contracting out government services: Lessons from the private sector. Public

Administration Review, 54, 176-184. Prager, Jonas. 2008. Contract City Redux: Weston, Florida, as the Ultimate New Public Management

Model City. Public Administration Review 68 (1): 167-80.

Radin, Beryl A. 2006. “One Size fits All.” In Challenging the Performance Movement: Accountability, Complexity, and Democratic Values, 33-52. Washington, DC: Georgetown University Press.

Reaves, J. (2001). Red faces at the Red Cross. Time, November 14. Riccucci, Norma M. (2005). Street-level bureaucrats and intrastate variation in the implementation of

Temporary Assistance for Needy Families policies. Journal of Public Administration Research

and Theory 15:89–111.

Romzek, Barbara S., and Jocelyn M. Johnston. 2002. Effective Contract Implementation and Management:

A Preliminary Model. Journal of Public Administration Research and Theory 12 (3): 423-53.

------. 2005. State Social Services Contracting: Exploring the Determinants of Effective Contract

Accountability. Public Administration Review 65 (4): 436-49.

Rose-Ackerman, S. (1986). The economics of nonprofit institutions: Studies in structure and

policy. New York: Oxford University Press. Rubin, Edward. (2006). The myth of non-bureaucratic accountability. in Michale W. Dowdle, ed. Public

Accountability: Designs, Dilemmas, and Experiences. Cambridge: Cambridge University Press.

Savas, E.S. (2000). Privatization and public-private partnerships. New York: Chatham House Publishers.

Savas, E.S. (2003). Competition and Choice in New York City Social Services. Public Administration

Review, 62, 1, 82-91.

Sclar, E.D. 2000. You Don’t Always Get What You Pay for: The Economics of Privatization. Ithaca,

NY: Cornell University Press.

Smith, Kevin B., and Christopher W. Larimer. 2004. A Mixed Relationship: Bureaucracy and School

Performance. Public Administration Review 64 (6): 728-36.

Smith, S.R. 2005. “NGOs and Contracting.” Pp. 591-614 in E. Ferlie, L.E. Lynn, Jr., and P. Christopher,

eds., The Oxford Handbook of Public Management. Oxford: Oxford University Press.

Straub, A. (2009). Cost savings from performance-based maintenance contracting, International Journal of

Strategic Property Management (2009) 13, 205–217.

Tirole, J. 1999. “Incomplete Contracts: Where Do We Stand?” Econometrica 67(4): 741-781.

Van Slyke, David M. 2003. The Mythology of Privatization in Contracting for Social Services. Public

Administration Review 63 (3): 296-315.

Van Slyke, David M. (2007). Agents or stewards: Using theory to understand the government-nonprofit

social service contracting relationship. Journal of Public Administration Research and Theory.

17(1): 157-187.

Yang, Kaifeng, and Jun Yi Hsieh. 2007. Managerial Effectiveness of Government Performance

Measurement: Testing a Middle-Range Model. Public Administration Review 67 (5): 861-79.

Yang, K, Hsieh, J.Y., and Li, T.S. (2009). Contracting Capacity and perceived Contracting Performance:

Nonlinear Effects and the Role of Time. PAR, 69, 4, 681-696.

35

Zimmermann, Jo An M., and Bonnie W. Stevens. 2006. The Use of Performance Measurement in South

Carolina Nonprofits. Nonprofit Management & Leadership 16 (3): 315-27.

Table 1. Sampled respondents and contract areas (% and # shown)

Service Area

# % # %

Consulting, evaluation and management training 9 23.1 6 20

Long-term care 7 17.9 1 3.3

Construction, maintenance, public works 6 15.4 3 10

Medical, nursing care, health management 5 12.8 2 6.7

Mental health, psychological consultation, arts therapy 4 10.3 1 3.3

Information Technology 3 7.7 2 6.7

Programs for women and children 1 2.6 3 10

Criminal Justice 1 2.6 2 6.7

Environmental (planting, plant control) 2 5.1 2 6.7

Food supply (and quality) monitoring 1 2.6 0 0

Animal care 0 0 1 3.3

Substance abuse, homelessness 0 0 3 10

Janitorial 0 0 1 3.3

Translation 0 0 1 3.3

Recreational (camps, dance lessons) 0 0 2 6.7

TOTAL 39 100 30 100

Government

respondents

(N=39)

Contractors

(N=30)

36

Table 2. The prevalence of performance measures used (% "yes" shown)

Performance Measurement Public

respondents

Contractors

1. Costs/cost-effectiveness 64.1 46.7

2. Quality 94.9 66.7

3. Workload 87.2 63.3

4. Impact on clients 79.5 56.7

5. Client satisfaction 84.6 53.3

6. Equitable delivery of services 41 30

7. Compliance with laws/regulations 64.1 43.3

8. Timeliness 94.9 66.7

9. Disruptions 97.4 73.3

10. Process specified in details 69.2 43.3

11. Quantitative indicators 76.9 60

12. Qualitative indicators 82.1 66.7

13. Informal monitoring 94.9 70

14. Measures tailored to organizations 41 40

15. Reputation 64.1 56.7

N=69

Table 3. Descriptives. Contractor complies by providing timely and accurate information.

Values Frequency Percent

8 3 4.35

9 8 11.59

9.5 1 1.45

10 4 5.8

11 14 20.29

12 39 56.52

Table 4. Descriptives. Government reveals problems and effectively manages the contract.

Values Frequency Percent

2 1 1.45

3 1 1.45

4 2 2.9

5 1 1.45

6 12 17.39

7 18 26.09

8 34 49.28

37

Table 5. Dependent variable: Contractor complies by providing timely and accurate information

b OR sig. b OR sig. b OR sig. b OR sig.

PERFORMANCE MEASUREMENT

Costs 1.69 5.40 **

Service disruptions 2.27 9.70 *

Service quality -2.24 0.11 **

PM conducted through unformal means -2.36 0.09 *

CONTROLS

Nonprofit Status -0.53 0.59 0.31 1.36 0.16 1.17 -0.11 0.89

Hard services -2.31 0.10 *** -2.16 0.12 ** -2.48 0.08 *** -2.49 0.08 ***

Collaborative perf. measurement index -0.01 0.99 -0.16 0.85 -0.23 0.80 -0.08 0.92

Contractor’s financial dependency 0.00 1.00 0.18 1.20 0.43 1.53 -0.37 0.69

Contractor has unique expertise 2.50 12.14 *** 2.56 12.96 *** 2.46 11.76 *** 2.45 11.62 ***

Contract awarded competitively 3.36 28.86 *** 3.06 21.24 *** 3.53 34.08 *** 3.21 24.67 ***

Environment perceived as dynamic -2.47 0.08 *** -2.38 0.09 *** -2.48 0.08 *** -2.24 0.11 ***

Relationship length -0.02 0.98 -0.02 0.98 -0.05 0.96 -0.03 0.98

Perceived trust -0.22 0.81 0.21 1.23 -0.39 0.67 0.25 1.28

Monitoring: self-reporting -3.29 0.04 ** -3.02 0.05 ** -3.20 0.04 ** -3.07 0.05 **

Monitoring: direct gov. inspections 0.75 2.13 0.81 2.26 0.94 2.56 1.04 2.82

Monitoring: third party monitoring -1.43 0.24 * -0.72 0.49 -0.54 0.58 -0.80 0.45

Government’s in-house prof. capacity 0.08 1.08 0.07 1.07 0.41 1.50 0.29 1.34

Cntr's engagement in PM is desirable 0.69 2.00 0.26 1.30 0.62 1.86 0.85 2.35

Respondent’s contract mgmt experience -0.16 0.85 *** -0.15 0.86 *** -0.17 0.85 *** -0.14 0.87 ***

Gov./contractor respondent dummy -4.51 0.01 *** -4.62 0.01 *** -4.59 0.01 *** -3.85 0.02 ***

Likelihood Ratio (ChiSq) 65.44 *** 63.52 *** 67.41 *** 63.76 ***

Note: *** - p < 0.01; ** - p <0.05; * - p <0.1; N=69.

Model 1 Model 2 Model 3 Model 4

38

Table 6a. Dependent variable: Government reveals problems and effectively manages the contract

b OR sig. b OR sig. b OR sig. b OR sig.

PERFORMANCE MEASUREMENT

Sum of performance measures used 0.23 1.26 **

Impact of services on clients 1.26 3.54 *

Client satisfaction -1.60 0.20 **

Services delivered equitably 1.25 3.50 **

CONTROLS

Nonprofit Status -1.23 0.29 ** -0.90 0.41 -1.06 0.35 * -1.28 0.28 **

Hard services 0.17 1.18 0.06 1.06 0.43 1.54 0.59 1.81

Collaborative perf. measurement index 0.06 1.06 0.02 1.02 0.18 1.19 0.14 1.15

Contractor’s financial dependency -1.44 0.24 * -0.91 0.40 -0.96 0.38 -1.57 0.21 **

Contractor has unique expertise 0.27 1.31 0.24 1.27 0.18 1.20 0.04 1.04

Contract awarded competitively -0.31 0.74 -0.06 0.94 -0.39 0.68 -0.33 0.72

Environment perceived as dynamic -0.58 0.56 -0.58 0.56 -0.11 0.90 -0.29 0.75

Relationship length 0.08 1.08 * 0.06 1.07 0.04 1.04 0.07 1.07

Perceived trust -0.44 0.64 -0.29 0.75 -0.56 0.57 -0.39 0.68

Monitoring: self-reporting -0.31 0.74 -0.86 0.42 -0.31 0.73 -0.14 0.87

Monitoring: direct gov. inspections 1.54 4.68 ** 1.79 5.99 *** 2.44 11.52 *** 2.06 7.82 ***

Monitoring: third party monitoring -0.75 0.47 -0.53 0.59 -0.16 0.85 -0.71 0.49

Government’s in-house prof. capacity 1.33 3.78 ** 1.62 5.05 *** 1.87 6.48 *** 1.68 5.37 ***

Cntr's engagement in PM is desirable -0.95 0.39 -0.80 0.45 -0.59 0.55 -0.55 0.58

Respondent’s contract mgmt experience -0.02 0.98 -0.02 0.98 0.00 1.00 -0.02 0.98

Gov./contractor respondent dummy -1.21 0.30 -0.72 0.49 -0.37 0.69 -0.89 0.41

Likelihood Ratio (ChiSq) 28.84 ** 27.30 * 28.60 ** 28.48 **

Note: *** - p < 0.01; ** - p <0.05; * - p <0.1; N=69.

Model 5 Model 6 Model 7 Model 8

39

Table 6b. Dependent variable: Government reveals problems and effectively manages the contract

b OR sig. b OR sig. b OR sig. b OR sig.

PERFORMANCE MEASUREMENT

Service timeliness 2.70 14.88 ***

Service disruptions 3.63 37.59 ***

Details service specifications 1.01 2.74 *

PM conducted through unformal means -2.23 0.11 **

CONTROLS

Nonprofit Status -0.96 0.38 -1.14 0.32 * -1.06 0.35 * -1.52 0.22 **

Hard services 0.31 1.36 0.36 1.43 -0.03 0.97 -0.40 0.67

Collaborative perf. measurement index 0.23 1.26 0.13 1.14 0.08 1.09 0.10 1.11

Contractor’s financial dependency -1.81 0.16 ** -1.67 0.19 ** -1.48 0.23 * -1.67 0.19 **

Contractor has unique expertise 0.37 1.45 0.38 1.46 0.22 1.24 0.12 1.13

Contract awarded competitively -0.87 0.42 -0.99 0.37 -0.30 0.74 -0.22 0.81

Environment perceived as dynamic -0.95 0.39 -1.00 0.37 * -0.33 0.72 -0.20 0.82

Relationship length 0.10 1.11 ** 0.11 1.12 ** 0.06 1.06 0.07 1.08

Perceived trust -0.67 0.51 -0.35 0.71 -0.56 0.57 -0.35 0.71

Monitoring: self-reporting -0.62 0.54 -0.28 0.76 -0.34 0.71 -0.48 0.62

Monitoring: direct gov. inspections 1.82 6.14 *** 1.98 7.22 *** 1.88 6.56 *** 2.51 12.31 ***

Monitoring: third party monitoring -0.83 0.43 -0.87 0.42 -0.60 0.55 -0.57 0.56

Government’s in-house prof. capacity 1.82 6.20 *** 1.60 4.95 ** 1.58 4.87 ** 1.88 6.53 ***

Cntr's engagement in PM is desirable -1.13 0.32 -0.62 0.54 -1.00 0.37 0.02 1.03

Respondent’s contract mgmt experience -0.03 0.97 -0.01 0.99 -0.02 0.98 -0.03 0.97

Gov./contractor respondent dummy -1.64 0.19 ** -1.57 0.21 ** -1.05 0.35 -0.50 0.61

Likelihood Ratio (ChiSq) 33.16 ** 37.61 *** 26.74 * 29.52 **

Note: *** - p < 0.01; ** - p <0.05; * - p <0.1; N=69.

Model 11 Model 12Model 9 Model 10

40

FOOTNOTES

1 This study adopts Romzek and Johnston’s definition and operationalizations of this term. Importantly, in some

studies, Romzek and Johnston use the term contract implementation and management effectiveness. These two

terms have been defined similarly and used interchangeably. 2 This is a term coined by Milward and Provan (2000).

3 Most of the information pertaining to data and measurement appeared in earlier publications based on the same

data: Amirkhanyan (2009) and Amirkhanyan (2010). 4 The instrument included questions with nominal or ordinal response categories, as well as those requiring

descriptive explanation and, hence, producing data that were appropriate for both qualitative and quantitative

analyses. 5 Please, contact the author for a copy of the interview guide and the data collection protocol.

6 These jurisdictions were conveniently accessible to the primary investigator and allowed in-person interviews. The

unique location of Washington, DC metropolitan area allowed collecting data from several jurisdictions in Maryland

and Virginia (population ranging from 195,000 to 5,600,000; proportion of Whites ranging from 38% to 80%, and

the median household income ranging between 43,000 and 77,000). 7 Telephone book was used in one jurisdiction in the absence of a procurement office web site.

8 The results of this research can therefore be applied more readily to jurisdictions with higher median income and

located near large metropolitan areas. The reliability of data collection was improved by using a common data

collection protocol. However follow-up research performed in other locations is needed to enhance the external

validity of the findings. 9 The statement offered to the private respondents was: “We accurately comply with government agency’s

performance measurement requirements.” Other statements were also modified accordingly. 10

Hypothesis testing rejected the “no common factors” hypothesis (p<.0001) and failed to reject the “2 factors are

sufficient” hypothesis (p=.82). Eigen values of the weighted reduced correlation matrix were 4.19 and 2.04 for the

two factors and they explained 67% and 33% of the variance. 11

Since some performance measures strongly correlated with one another, including all fifteen measures in one

model suggested multi-collinearity. 12

The design of this study has its limitations. Most importantly, this study covers a limited number of neighboring

jurisdictions which limits its generalizability. Thus, our findings may be more readily applied to locations with

higher population income and those in large metropolitan areas. The reliability of data collection was enhanced by

using a common data collection protocol, however, the replications of this research in other locations may help

enhance its external validity. 13

Low prevalence of cost assessment may have been caused by a large proportion of fixed-cost contracts especially

in the human services area. Thus, in these cases, monitors reported reviewing the billing documentation, rather than

evaluating the cost-effectiveness of the contracted services.