quantile regression for binary performance indicators

APPLIED STOCHASTIC MODELS IN BUSINESS AND INDUSTRYAppl. Stochastic Models Bus. Ind. 2008; 24:401–418Published online 26 August 2008 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/asmb.732

Quantile regression for binary performance indicators

Paul Hewson1,∗,† and Keming Yu2

1School of Mathematics and Statistics, University of Plymouth, Drake Circus, Plymouth PL4 8AA, U.K.2Department of Mathematical Sciences, Brunel University, Uxbridge, Middlesex UB8 3PH, U.K.

SUMMARY

Quantile regression is an emerging modelling technique; we examine an approach allowing this techniqueto model binomial variables in a Bayesian framework and illustrate the value of this advanced techniqueon a set of local government performance indicators from England and Wales. In U.K. local government,there is currently particular interest in assessing performance relative to ‘top’ and ‘bottom’ quartiles; allauthorities are expected to match the current best quartile performance within 5 years, any authority inthe ‘bottom’ quartile is assumed to be significantly below par. By its very nature, quantile regressionlets us to explore relationships between various covariates and these particular levels of performance.Additionally, by examining a number of other percentiles, we demonstrate how quantile regression givesa much fuller insight into the apparent behaviour of the system we are modelling. Rather than relyingon asymptotic results, we use Bayesian methods that allow us to explore the uncertainty implicit in ourmodel building and predictions. We suggest that this is most important when analysing data that are usedto make managerial and administrative decisions. Copyright q 2008 John Wiley & Sons, Ltd.

Received 14 April 2008; Revised 1 May 2008; Accepted 25 June 2008

KEY WORDS: Bayesian quantile regression; MCMC; performance indicators

1. INTRODUCTION

In a statistical context, one possible definition for a performance indicator has been given as ‘asummary statistical measurement on an institution or system that is intended to be related to the‘quality’ of its functioning’ [1]. Such measures can either relate to the resources required to operatea service, be measures of the process or be measures of the qualitative and quantitative achievementof the service. Current administrative usage classifies these as ‘inputs’, ‘outputs’ and ‘outcomes’,respectively [2]. There is an ever increasing use of performance management throughout the publicsector both internally in terms of administration and management as well as externally in terms ofaccountability and political direction. There has been a corresponding interest from the statistical

∗Correspondence to: Paul Hewson, School of Mathematics and Statistics, University of Plymouth, Drake Circus,Plymouth PL4 8AA, U.K.

†E-mail: [email protected]

Copyright q 2008 John Wiley & Sons, Ltd.

402 P. HEWSON AND K. YU

community in relation to this high-profile application area, and a number of specific modellingtechniques have been demonstrated [3]. We make a further proposal in this paper, the use of quantileregression. To do this, however, we adapt the existing quantile regression techniques to modelbinary data; the particular indicator we consider here, as with many others, is essentially measuredas a percentage. While we confirm the general value of quantile regression, specific motivationto use quantile regression comes from U.K. practice of regarding ‘top’ quartile performance asbeing the benchmark to which all other bodies should aspire, and ‘bottom’ quartile performanceas delineating unacceptable levels of performance. Understanding relationships with covariates(‘inputs’ in performance indicator terminology) at these frontiers is therefore of particular interest.To our knowledge, work using Bayesian quantile regression has never been reported in this contextwith U.K. local government performance indicators.

The structure of the paper is as follows. A brief overview of quantile regression will be given inSection 2; along with our Bayesian approach to fitting such models. We will then indicate how wehave extended these models to deal with binomial responses. As stated, we will demonstrate thevalue of this technique by investigating a particular set of local government performance indicatorsfrom England and Wales, namely three indicators measuring the administration of housing relatedbenefits. Section 3 will therefore provide an overview of public sector performance indicators,paying particular attention to the U.K. and indicate how recent legislation has provided a strongimperative to consider the use of models for upper and lower quartile performance. The applicationand the data we are going to consider in detail in this paper will be described in Section 4. Resultsof our model fitting will be presented in Section 5 and discussed in Section 6. Finally, we willprovide some conclusions about the role of quantile regression in terms of assessing performanceindicators in Section 7.

2. QUANTILE REGRESSION

Quantile regression extends the least-squares estimate of conditional means for a range of modelsestimating conditional quantile functions [4]. A number of reviews have appeared recently [5].Perhaps most relevant for this application is a recent overview specifically considering economicapplications [6].

We give here a brief explanation of the key features of quantile regression in a Bayesian setting.Consider first a conventional regression, where we wish to model response yi as

yi =xTi b+�i

given xi , a vector of covariates with xi =(1, xi1, . . . , xip) with p possibly equal to one, b is acorresponding vector of coefficients such that b=(�0,�1, . . . ,�p) and �∼N(0,�2). Where we aimto model the conditional mean response, the Gaussian likelihood is given by

L(b)∝exp

{− 1

2�2n∑

i=1(yi −xTi b)

2}

and maximizing L(b) over b gives the familiar least-squares estimates.Our interest however is in terms of modelling conditional quantiles rather than the conditional

mean. Conventional models are fitted by linear programming methods, and inference relies onasymptotics, and often the bootstrap (itself relying on asymptotic properties). Bayesian inference

Copyright q 2008 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2008; 24:401–418DOI: 10.1002/asmb

QUANTILE REGRESSION FOR BINARY PERFORMANCE INDICATORS 403

is therefore attractive, where one does not wish to rely on asymptotic properties. In the Bayesiancontext, perhaps most work has been performed in terms of median regression modelling (L1regression), although there is a body of work considering regression on a range of quantiles andeven allowing the potential for the quantile of interest to be expressed as a prior. To illustrate ourapproach to Bayesian quantile regression, we make a general assumption about the errors �, andwith a loss function ��(·) to be defined later we would seek to maximize

L�(b)∝exp

{−

n∑i=1

��(yi −xTi b)}

where the loss function ��(·) for the 100�% quantile is given by a ‘check function’

��(·)=��I[0,∞)(�)−(1−�)�I(−∞,0](�) (1)

so maximizing the likelihood is equivalent to minimizing the ‘check function’. In this checkfunction, IA is the indicator function on the set A:

IA(�)={1, �∈ A

0, � /∈ A

and function can be viewed as an asymmetric Laplace density given by

f (�)=�(1−�)exp(−��/�) (2)

hence we can regard L�(b) as an asymmetric Laplace likelihood. It is convenient to simulate fromthis Laplace density using the linear combination

1

�U− 1

1−�V

where U and V are independent exponential random variables with mean 1.This fitting method is similar to another proposal based on scale mixtures of normals, which itself

leads to an asymmetric Laplace density [7]. There are other proposals in the Bayesian framework,mostly for median rather than full quantile regression such as methods based on Dirichlet ProcessPriors [8]. Proposals have also been made based on modelling the error structure with Polya treesconstrained to have median zero [9]. Approximate inference has also been developed based onthe substitution likelihood [10]. However, in using (2) as the basis for our model fitting, it hasbeen shown that many priors for b, including an improper uniform prior, always result in a properjoint posterior distribution [11]. Consequently, although a standard conjugate prior distribution isnot available for the quantile regression formulation, it is still relatively easy to use Markov chainMonte Carlo (MCMC) methods for obtaining the posterior distributions.

We provide more details on model fitting later, we next need to describe modifications to dealwith the particular response variables in our application.

2.1. Dealing with binomial response

As will be explained in Section 4, we will consider three binary variables reporting the administra-tion of three distinct but related benefits by local government in England and Wales. The indicators



report the number of new claims paid within 14 days for each of three benefits. For our purposeshere, we will therefore consider three separate sets of models

ri j ∼ B(�i j ,ni j ) (3)

where ri j denotes the number of claims paid within 14 days for authority i=1 to n where inthis case n=331 and benefit j =1 to p where p=3 benefits considered. ni j is the correspondingnumber of new claims submitted, and �i j is the proportion of claims paid within 14 days for eachbenefit for each authority.

We then wish to model �i j in terms of a linear predictor, such that

log

(�i j

1−�i j

)=xibj (4)

where xi is a vector of covariates for each authority i . We considered three covariates; the cost ofprocessing claims, the year (1993–1994, 1994–1995, 1995–1996) as a mean centered covariate,i.e. (−1,0,1) and the overall size of the claimant base. b j is a vector of parameters relating thecovariates to the particular quantile response p for one of the three benefits j . This obviously couldtake the form of a generalized linear model and certainly quantile regression has been developedfor count data.

However, one of the key asymptotic assumptions underlying classical quantile regression is thecontinuous nature of the response variables [12]. The Poisson (or variants such as the negativebinomial) do not have continuous support over the distribution function, and therefore the quantilesare not continuous. One approach is to ignore the discrete nature of the response variables, forexample taking log(1+ yi ) when dealing with count data as has been used for exampling whenusing Bayesian methods to investigate conditional quantiles for the count of patent applications [7].As an extension to this, Box–Cox transformations have been used with additive smooth functionsfitted to the quantiles by the LMS method [13]. Another promising proposal has been to introducejittering to the variables to create a pseudo-continuous variable with the same quantiles as the countdata [14]. Further proposals are made in the context of binary data in relation to willingness-to-paystudies, which relied on quantile information being preserved with monotone transformation inassociation with a normal kernel [15]. Fortunately, binomial data are easier as we can treat thediscrete response as a realization of an underlying continuous distribution [16].

In dealing with Binomial responses, we note that instead of assuming a linear relationshipbetween the response and covariates, we are assuming a linear relationship via a logistic linkfunction. As quantile regression has the property that the transformed quantile regression equalsto the quantile regression of the transformed variable, i.e.

Q(t (y))= t (Q(y)) (5)

where t (·) denotes a transformation and Q(·) denotes the quantile. We assume that there exists aparametric transformation such as the Box–Cox transformation, which means that the regressionfunction can be dealt with as though linear even if the linearity does not hold exactly. Therefore,to be able to use existing results on the propriety of Bayesian quantile regression, we report herean interim development in the modelling of binary indicators. To implement Bayesian quantileregression inference for binomial data effectively, we prefer to first create pseudo-normal variates[17]. In general, we wish to approximate the log-likelihood function for each observation ri j |ni j bya normal density in �i j . Taking a second order Taylor series expansion, with dispersion parameter



�, with gi j =xi b j denoting the point at which the approximation is centered; pseudo-normalvariables can be formed as follows:

y(∗)i j = �i j −

��

��(ri j |�i j , �i )

�2��2

(ri j |�i j , �i )

�(∗)i j = − 1

�2��2

(ri j |�i j , �i )

More specifically for the binomial model, where �=1 by definition, with a logistic link thelog-likelihood is given by

�(ri j |�i j )=ri j�i j −ni j log(1+e�i j )

and so the derivatives are given by

��

��i j= ri j −ni j

e�i j

1+e�i j

�2��2i j

= −ni je�i j

(1+e�i j )2

In our case, where we wish to consider separate models for j =1, . . . ,3 benefits, given ri j is thenumber of new benefits processed within 14 days and ni j is the number of new benefit applicationsfor authority i and benefit j , then �i j is the linear predictor given covariates xi and the current

estimate of parameters b j then our pseudo-observations are given by

y(∗)i j = �i j +

(1+e�i j )2

e�i j

(ri jni j

− e�i j

1+e�i j

)(6)

and the pseudo-variances are given by

�(∗)i j = 1

ni j

(1+e�i j )2

e�i j(7)

Given the pseudo-observations, y(∗) =(y(∗)1 , . . . , y(∗)

n )T, we therefore seek to model

y(∗)i =xib+�i (8)

where �i are based on the asymmetric Laplace density given in (2). We therefore construct anMetropolis–Hastings based sampler for the posterior density �(b|y(∗)) of b given y(∗), �(b) by



using

�(b|y(∗))∝ L�(y(∗)|b)�(b)

where �(b) is the prior distribution of b and L(y(∗)|b) is the likelihood based on the asymmetricLaplace density given earlier in Equation (2). Given y(∗)

i j and �(∗)i j , a Metropolis–Hastings sampler

can be used with some confidence [11]. While it had been shown that it is possible to use animproper uniform prior and retain propriety of the posterior, in this application we use normalpriors with zero mean and variance 10 for all regression coefficients. This is relatively small, butit should be noted that it puts reasonable probability mass over the range of values which couldbe anticipated given the binomial response.

Having discussed the models we will be using, we next consider performance managementin the U.K. public sector, and provide a particularly strong motivation for using these quantileregression models.

3. PUBLIC SECTOR PERFORMANCE MANAGEMENT

The U.K. Cabinet have recently emphasized the need for outcome focused delivery rather than orga-nization focused delivery and argue that performance indicators are needed to manage this activity[18]. HM Treasury, on releasing the recent ‘FABRIC’ document on Performance Managementstated:

‘Performance information is a cornerstone of our commitment to modernize government. Itprovides some of the tools needed to bolster improvements in public sector performance . . . . . .

Good quality information also enables people to participate in government and exert pressurefor continuous improvement. In addition to empowering citizens, this information equipsmanagers and staff within the public service to drive improvement. Performance informationis thus a catalyst for innovation, enterprise and adaptation.’ [19]

Performance management is central to public sector administration in the U.K., evidenced by thelarge number of governmental components that are strategically involved including the NationalAudit Office Directorate of Performance Management, the Audit Commission Centre for Perfor-mance Measurement, The Performance Information Panel (Her Majesty’s Treasury) and the CabinetOffice’s Modernizing Public Service Group’s Performance Measurement Team as well as depart-ment specific endeavour. For local government, the Office of the Deputy Prime Minister (ODPM)have a role in setting policy and indicators.

Passive reporting of raw data in U.K. Local Government has been required for over two decades[20]. Since then, in the U.K., a programme of transferring state-owned activity to the private sectorhas been carried out. Changes in private sector business practice, as well as political and businesslimits on the ability to transfer services to the private sector, have focused attention on the desireto introduce private sector management styles into the public sector. One early attempt in the U.K.was the introduction of the ‘Citizen’s Charter’ legislation, followed almost a decade later by theintroduction of the ‘Best Value’ Legislation. The Citizen’s Charter and Best Value Legislationaimed to build incrementally on this passive reporting in that they sought to simulate a businessenvironment where success could be acknowledged and failure could be dealt with. It was intendedthat this would provide an imperative for the public sector similar to the competitive stimulus



in the private sector. Performance indicators were no longer seen as numbers to be passivelyreported, but as currency for a particular method of management. As a result, it was envisagedthat mechanisms would be developed for performance management [2, 21]. We highlight here the‘Best Value’ legislation; in addition to publishing performance indicators and the establishmentof various inspectorate bodies that gave the Secretary of State a range of powers to interveneagainst authorities where there was deemed to be a failure to ‘secure continuous improvement’,this legislation has been interpreted with a particular interest in performance ‘quartiles’. It isanticipated that all authorities will take steps over five years to match their performance with thatof the current ‘top’ quartile, whereas ‘bottom’ quartile performance is suggestive of failure [22].This focus on quartiles therefore forms part of our motivation for considering quantile regression.

Before considering the particular application in detail, it is worth noting that the growth ofperformance management since the 1990s has recently been the focus of considerable attention inthe statistical world [23]. It has also been ‘officially’ reviewed, for example by the Public AccountsCommittee in the U.K. who concluded that measurement culture had much to offer, but highlighteda number of risks such as distorting behaviour to pursue apparently better results and even raisedthe possibility of it creating incentives to cheat [24]. Part of the drive within the U.K. has beenbased upon activity in the U.S., such as 1993 Government Performance and Results Act (GPRA)[25]. Performance management culture was regarded as highly successful in the United Statesin a programme initially called as National Performance Review and later renamed as NationalPartnership for Reinventing Government [26, 27]. Current developments in the U.K. include PublicService Agreements (PSA) whereby the Treasury is increasingly using performance measurementas a means of managing how money is spent [28].

4. LOCAL GOVERNMENT BENEFIT ADMINISTRATION

For this study, we will examine a particular local government indicator related to the administrationof benefit. Despite the interest in ‘outcome’ focused performance indicators mentioned above, weconsider here an ‘output’ indicator on the basis that we have a closer association between councilactivity and the measure and therefore modelling the relationship between the covariates and theindicator is less subject to confounding. Data relating to the administration of three benefits bylocal councils have been used. While some measure of performance in this area has been publishedsince 1993–1994, the potential value of the time series is severely limited by the reduction ofthe available data in 1996–1997 to a simple percentage without the component numerator anddenominator (and later a change to the average pay-out time). This kind of limitation on dataquality is not unusual in performance management systems, and we will therefore consider a fewgeneral points about data quality in performance management systems.

‘Public agencies are very keen on amassing statistics—they collect them, add them, raisethem to the nth power, take the cube root and prepare wonderful diagrams. But what youmust never forget is that every one of those figures comes in the first instance from thevillage watchman, who just puts down what he damn pleases.’ Sir Josiah Stamp 1880–1941(Governor of the Bank of England)

Many performance indicators have been defined as byproducts of data derived from pre-existing administrative systems. This suggests that potentially available information is valued



over potentially useful information. It could be claimed that information designed to capturecompliance with statute does not necessarily have the detail to understand variations in perfor-mance. In order to use information that is freely available rather than commit additional resourcesto further data capture, compromises can be made in the actual definition of the performanceindicator. Auditing of performance indicators has been considered [29]. In the U.K., while someof the raw data may fall within the remit of National Statistics, most of the effort in terms ofdefining, collecting, calculating and publishing performance indicators falls entirely outside theirremit. Data collection is controlled by financially trained auditors rather than by statisticians.When these auditors have serious doubts about data integrity, this is recorded and the indicatorsare qualified. This information has been omitted from the analysis we report here, one furtherdevelopment would be to incorporate this qualification in the modelling process somehow. Forlocal government, methods for defining and collecting performance indicators have been setout by the Audit Commission [30]. In theory, data integrity for local government performanceindicators should be assured by the activities of District Audit, who check on behalf of the AuditCommission that the data published nationally can be verified by conventional audit procedures.However, one of the problems associated with performance indicator systems is that the sameorganization is responsible for collecting the data on which it will be judged. There have beenwell publicized problems in the U.K. caused by this, where the temptation to manipulate the datacan even lead to adverse manipulation of the system being monitored [31].

Returning to our benefit administration data, there were 331 district councils between 1993–1994 and 1995–1996, which returned valid performance indicator data for each year in respectof three benefits administered by local government. These benefits are Council Tax benefit, RentAllowance and Housing Benefit. It needs to be noted that there is considerable range in the scaleof the various operations. In terms of Council Tax benefit, the number of new claims processedranges from 67 to 178 000 with a mean of 8992. For rent allowance, the number of new claimsprocessed ranged from 130 to 48 100 with a mean of 3903 and for housing benefit the number ofnew cases processed ranged from 6 to 60 600 with a mean of 2941. The considerable variationin the size of the structures needed to handle these benefits is not accounted for in any way inthe official performance indicators, indeed, from 1996 to 1997 the numerator and denominatordata ceased to be collected separately at all. However, by way of introducing these data, Table Igives the mean and extreme performance indicators reported (the percentage of new benefit claimsprocessed within 14 days).

Given the interest in ranking performance indicators, Figure 1 presents the published performanceindicator for 1995–1996 on the proportion of Council Tax Benefit Claims paid within 14 days.Results for Housing Benefit and Rent Allowance present a similar picture to this over the previoustwo years. One feature that should be highlighted is the considerable range in terms of the scaleof operation at different councils, hence the lower part of the chart denotes the number of newbenefit claims processed in a year.

Table I. Summary of performance indicators for the three benefitadministration performance indicators.

Benefit Minimum (%) Mean (%) Maximum (%)

Council Tax 10.4 78.4 100Rent Allowance 1.0 75.3 100Housing Benefit 12.8 81.7 100



0 20 40 60 80 100

0.00

0.01

0.02

0.03

0.04

Percentage of claims paid on time

Pro

port

ion

of a

utho

ritie

s

0 50 100 150 200 250 300

0102030405060

Authority size and rank

Processing of Council Tax benefit in 1995–1996

Authority Rank

Tho

usan

ds o

f cla

ims

proc

esse

d

Figure 1. Performance indicators for Council Tax Benefit (1995–1996); upper chart gives a histogramdenoting performance of authorities paying within 14 days, lower chart indicating the total number ofnew benefit claims processed in the year against ‘rank’ of the authority with respect to this indicator.

5. RESULTS

Three mean centered covariates were considered: year, cost of processing a claim, total size ofclaimant base. Note that n just records the number of claims in that category, not the total numberof claimants for which an authority has responsibility. A relatively large value of n with respect tothe overall size of the claimant base therefore reflects a relatively higher turnover of clients, this isnot something we have explicitly considered in these models. Quantile models were fitted to the5th, 10th, 25th, 50th (median), 75th, 90th and 95th percentiles. As mentioned earlier, particularinterest surrounds the 25th and 75th percentiles as these are taken as ‘target’ values in localgovernment management in the U.K.

The R environment was used throughout this study [32]. Apart from the modifications outlinedabove to create pseudo-normal variates, the MCMC sampler was based on the one reported earlier[11]. Typically, at least 5000 iterations were required for burn in and typically a further 25 000were used for inference.

Figure 2 provides an example of the trace plots from fitting the median response to the housingbenefit performance indicator. Formal convergence checking was undertaken using Gelman andRubin’s R [33], Geweke’s statistic [34] and Heidelberger and Welch’s [35]. There is a degree ofprior sensitivity in these models, not necessarily in the most extreme quantiles. As mentioned, wechose priors with a relatively small variance. This gives reasonable probability mass over the rangeof values, and was intended as a compromise between giving the prior distribution an informative



mean, yet maintaining a reasonable acceptance rate. A wider variance does allow the sampler toreach a combination of values and stay trapped, a lower variance leads to samplers that do notexplore the posterior distribution well. Where possible, we have maintained the same prior valuesfor the whole range of quantiles fitted, which may not be entirely sensible.

5000 15000 250000.8

1.4

2.0

Iteration number

β0

5000 15000 25000

0.0

–0.008–0.15

0.2

Iteration number

β1

5000 15000 25000

0.00

Iteration number

β2

5000 15000 25000

0.002

Iteration number

β3

Figure 2. Trace plots for posterior samples for the four covariates resulting from fitting quantile regressionfor median response to housing benefit administration data.

5th 10th 25th 50th 75th 90th 95th

0

–1

1

2

Housing Benefit Performance:Intercept

Percentile

β0

Figure 3. Boxplots summarizing posterior density for the intercept for seven quantileregression models fitted to 1st, 5th, 25th, 50th (median), 75th, 95th and 99th percentiles

to housing benefit administration data.



Considering the housing benefit indicator in the most detail. Figure 3 summarizes the posteriorfor �0, the intercept for the seven models fitted. These results confirm that there is a limiting upperbound (100% performance cannot be improved upon).

Figure 4 illustrates perhaps the most interesting set of posterior distributions obtained from thismodelling exercise and gives the posterior distribution for the parameter associating the cost ofprocessing each claimant with the level of performance seen. While we must caution over theprecise economic interpretation, given the probability mass associated with the seven parameterestimates, there does not appear to be an association consistent with increased finance leadingto improved performance, indeed while there is some posterior probability mass to support thisin the very highest quantiles, the impression from the lower quantiles is that increasing financeis associated with decreased performance. It should however be noted that the level of financereceived by each authority is subject to a complex set of arrangements, and interpretation of thisfinding is limited. It may well be that higher finance is a proxy for authorities operating in a morechallenging context.

Figure 5 gives the posterior distribution for the parameter associating the size of the overallclaimant base with reported performance. The results are much less clear than Figure 4, butwould seem to suggest that there are little apparent economies of scale. Again however, giventhe complexities of local government organization this finding needs to be interpreted with care.Larger authorities are more likely to be urban but have a higher turnover of clients and generallybe operating in more challenging circumstances.

Finally, Figure 6 denotes the posterior distribution for the parameter associating time withperformance. It gives perhaps the most obvious results, but equally gives the most convincingdemonstration of the value of quantile regression. As performance improves towards the 100%limit, it is perhaps obvious that the better authorities will improve less, hence the lower valueof this parameter for the 75th, 95th and 99th percentile. Equally, this demonstrates the value ofquantile regression. A single summary, such as that provided by the median regression, representsan over-simplistic model of performance over time. While the results will be discussed in more


0.000

–0.010

Housing Benefit: Association between cost and performance

Percentile

β3

Figure 4. Boxplots summarizing posterior density for the relationship between cost and performance forseven quantile regression models fitted to 1st, 5th, 25th, 50th (median), 75th, 95th and 99th percentiles





0.0

–0.1

–0.2

0.1

Housing Benefit: Association between claimant base and performance

Percentile

β2

Figure 5. Boxplots summarizing posterior density for the relationship between size of claimant base andperformance for seven quantile regression models fitted to 1st, 5th, 25th, 50th (median), 75th, 95th and

99th percentiles to housing benefit administration data.


0.0

–0.2

0.2

0.4

0.6

Housing Benefit: Association between time and performance

Percentile

β1

Figure 6. Boxplots summarizing posterior density for the relationship between time and performance forseven quantile regression models fitted to 1st, 5th, 25th, 50th (median), 75th, 95th and 99th percentiles


detail next, the suggestion is that it is the lower performing authorities which most improve overtime.

We next provide some brief results on Council Tax benefit administration. Four sets of results aredepicted in Figure 7, it can be seen that the intercepts �0 follow a similar pattern to that seen withhousing benefit (reported in Figure 3). More notably, the trend over time for Council Tax benefit




0

–1

–0.05

–0.15

–0.25–0.2

–0.005

–0.015

1

2

Council Tax Benefit Performance: Intercept

Percentile

β0


0.005

Council Tax Benefit: Association between cost and performance

Percentile

β3


0.05

Council Tax Benefit: Association between claimant base and performance

Percentile

β2


0.0

0.2

0.4

0.6

Council Tax Benefit: Association between time and performance

Percentile

β1

Figure 7. Boxplots summarizing posterior density for the intercept (upper left), the rela-tionship between cost and performance (upper right), the relationship between claimantbase and performance (lower left), the relationship between time and performance (lowerright), for seven quantile regression models fitted to 1st, 5th, 25th, 50th (median), 75th,

95th and 99th percentiles to Council Tax benefit administration data.

reported in the lower right panel of Figure 7 is almost identical to that seen for housing benefit(reported in Figure 6) albeit the parameters tend to be smaller and more dispersed. Nevertheless,there is a clear pattern, lower performing authorities appear to have increased their performancemore over time than the higher performing authorities, a pattern that is apparent with both sets ofindicators. However, with regard to the relationship with the other two covariates, with the possibleexception of the 95th percentile response for the association between cost and performance mostof the posterior density mass is below zero. It therefore generally appears for Council Tax benefitthat there is a negative relationship between performance and both cost and size. As has beenmentioned earlier, this may well reflect on confounding factors such as larger, better financedauthorities being those that operate in more challenging areas with high client turnover.

Finally, some results from fitting models to the rent allowance administration data are presentedin Figure 8. Unlike the other two benefits, there is really very little evidence of an associationbetween the cost of processing claims and the performance at any quantile level. Conversely, whilethere is considerable uncertainty (and overlap), it appears that there is an association betweenincreasing size and attained performance across the quantile range.

6. DISCUSSION

Some small points need to be recorded with regard to the data analysed. The three-year periodcovered was selected because it is the only three years for which numerator and denominator data




0

–1

–0.2

–0.4–0.4

1

2

Rent Allowance Performance: Intercept

Percentile

β0


0.005

–0.005

–0.015

Rent Allowance: Association between cost and performance

Percentile

β3


0.0

Rent Allowance: Association between claimant base and performance

Percentile

β2


0.0

0.2

0.4

0.6

Rent Allowance: Association between time and performance

Percentile

β1

Figure 8. Boxplots summarizing posterior density for the intercept (upper left), the relationship betweencost and performance (upper right), the relationship between claimant base and performance (lower left),the relationship between time and performance (lower right), for seven quantile regression models fittedto 1st, 5th, 25th, 50th (median), 75th, 95th and 99th percentiles to rent allowance administration data.

were available, the deterioration in data quality following this period where only information onthe percentage of claims paid on time was collected limits our ability to extend this analysis. Therehave been managerial improvements to this set of performance indicators. One obvious way ofimproving reported performance would be merely to process all applications without checking; thescope for this ‘perverse incentive’ has been designed out of these indicators in that data on over-payments has been collected more recently. It should also be noted that these kinds of indicatorsare not necessarily regarded as being entirely sensible [23]. Specifically, for any indicator thatreports percentage activity carried out within a target deadline, once an individual claimant missesthe 14 day target there is no incentive to prioritize them over a claimant within the 14 day target.Again, to some extent this has also been designed out of the system in that information on averagetime to pay claims is currently reported. Nevertheless, binomial indicators of this type remaincommonplace throughout the U.K. public sector.

It is acknowledged that to some extent the clearest results of our quantile regression analysisare the most obvious. However, we refer to other work using quantile regression with count datato examine the effect of health care reform on patient–doctor visits [36]. That paper reported afinding that regular visitors were less affected by policy changes than infrequent visitors. In theresults reported here, we have also demonstrated the value of a quantile regression approach witha binomial indicator bounded by 1 (100% of claims processed over time); those authorities closestto this boundary show less improvement over time than those authorities that are generally furtherfrom this boundary at the outset of the data collection period. Trite as this result may seem, it shouldbe strongly noted that these results cannot necessarily be gleaned from conventional regression on



the conditional mean response. We note that considerable effort, particularly using Bayesian andnon-Bayesian multilevel models, has been expended in the analysis of performance indicators. Asa result, for example, there are very strong recommendations that league table positions should bequantified by measures of uncertainty [1]. Yet we demonstrate here that substantively importantinformation can be omitted by regressing on the conditional mean. The modest improvement in themedian performance under-regards the larger improvement seen in the lowest quantiles examined.Because we are considering a bounded indicator, our interpretation of this finding is limited, butthis need not be the case if other indicators were considered. The potential policy implications ofsuch a finding are clear, whatever ‘average’ level of improvement might be reported, it is the lowerperforming authorities that are improving most. Nevertheless, even this finding needs qualification.While large improvements in reported performance indicators are common on introduction of a newperformance management system this is not necessarily matched by large changes in performanceon the ground. Especially when using performance indicators that are byproducts of administrativesystems, it has been noted that early changes are largely due to the attention paid to the need torecord data in a particular way [37]. We may therefore be looking at results that indicate increasingattention to the data collection, rather than to the changes in the number of benefits processed ontime.

With this caveat, some of the other results are more interesting, if less conclusive. For example,there are suggestions that the performance attained by lower quantile authorities declines withincreasing financial resource, the opposite may almost be true for upper quantile authorities. Thishas clear, if challenging, implications for public policy and needs have to be examined morecarefully. Nevertheless, the potential for confounding underlying such findings is quite striking,although we repeat that there is a tremendous potential for confounding in these data, for examplethe apparently ‘better’ financed authorities are receiving this money as a result of working in themost challenging areas. Considerable statistical work has already been directed towards dealingwith such confounding which is always problematic when dealing with observational data, wehighlight here work carried out examining ‘Potential Confounding Factors’ specifically in thecontext of Higher Education performance indicators in the U.K. [38].

One other administrative problem with performance indicators are that there can often be aconsiderable time lag between the data collection period and the publication of such data. Inthis context, it may be noted that considerable work has been expended to model data in thepresence of such latency, for example in the context of car insurance claims [39]. There may bescope for modelling of performance indicators in a similar manner, which could have considerablemanagement benefit by provide more timely predictions of attained performance.

Clearly one important extension to our technique will be to model the binomial response directlyas a generalized linear model by making appropriate alterations to the loss function. Some workis still needed to ensure that we can do this with a binomial response and ensure propriety of theposterior; hence our use of the pseudo-normal variates in this application. Of course, alternativesto this approach are possible, for example, a method called Bayesian exponentially tilted empiricallikelihood has recently been reported [40].

As will be clear from the results given here, we have in fact three response variables (threedifferent benefits); one would assume some commonality in terms of the relationship betweencovariates and the number of claims processed on time. There is a growing interest in modellingsuch data in a multivariate setting [41]. One popular technique with some parallels here is stochasticfrontier regression [42]. With that model, interest is focused on estimating an efficiency frontieracross all variables, facilitated by modelling a joint distribution for a one-sided random effect u



which is assumed to capture the ‘inefficiency’. It is conventionally modelled as

log(x)= log{F(y)}+u+v (9)

where F(y) represents an unknown efficiency frontier surface, u represents inefficiency and v

represents errors in x. The canonical F(y) is the Cobb–Douglas function [43]. This is given by

log{F(y)}=�0+�1 log(y1)+·· ·+�p log(yp) (10)

There has been particular governmental interest in using this technique to examine police perfor-mance indicators in the U.K. [44]. There has also been a robust critique of the way the techniquecan be misused in this context in the statistical literature [45]. Nevertheless, we feel that thereare some parallels in the role of the carefully constructed error term � in the quantile regressioncheck function in formula 1, and the one-sided error term u in formula 9. It is certainly plausiblethat one might wish to obtain some shrinkage of model parameters across the three responses inthe same way as is possible with conventional multivariate regression on the conditional mean[46]. The potential for such an approach has been demonstrated with road safety performanceindicators [47]. There has also been work published on median regression for multiple outcomesin genotoxicity that models dependence by an underlying factor analysis model [48]. We wouldsuggest that multivariate quantile regression, perhaps based on latent structure models, will alsoadd to the value of the univariate Bayesian quantile regression presented here and provide a richerset of information than is possible from stochastic frontier regression.

7. CONCLUSION

We have extended quantile regression techniques so that binomial variables can be modelledin a Bayesian framework. This allows us to explore the uncertainty associated with the modelparameters. The value of this approach is illustrated with reference to a set of U.K. local governmentperformance indicators. It is very clear that our models provide a much richer set of information thancan be gleaned from other models currently used to interpret a set of performance indicators. It isclear that over the three years considered those authorities with a lower reported performance werethe ones that saw greatest improvement in their published performance. There are some suggestionsthat lower performing authorities generally do not show improved performance for greater finance.Moreover, given explicit interest in assessing ‘top’ and ‘bottom’ quartile performance within theU.K. local government, we anticipate much greater use of these models in the future.

REFERENCES

1. Goldstein H, Spiegelhalter D. League tables and their limitations: statistical issues in comparisons of institutionalperformance (with Discussion). Journal of the Royal Statistical Society, Series A 1996; 159:385–444.

2. Audit Commission. Aiming to Improve: The Principles of Performance Management. Audit Commission: London,2000.

3. Smith P. The use of performance indicators in the public sector. Journal of the Royal Statistical Society, Series A1990; 153:53–72.

4. Koenker R, Basset G. Regression quantiles. Econometrica 1978; 46:33–50.5. Yu K, Lu Z, Stander J. Quantile regression: applications and current research areas. Journal of the Royal Statistical

Society, Series D 2003; 52:331–350.6. Koenker R, Hallock K. Quantile regression. Journal of Economic Perspectives 2001; 15:143–156.



7. Tsionas EG. Bayesian quantile inference. Journal of Statistical Computation and Simulation 2003; 73(9):659–674.8. Kottas A, Gelfand AE. Bayesian semiparametric median regression modelling. Journal of the American Statistical

Association 2001; 96(456):1458–1468.9. Hanson T, Johnson WO. Modelling regression error with a mixture of Polya trees. Journal of the American

Statistical Association 2002; 97(460):1020–1033.10. Dunson DB, Taylor JA. Approximate Bayesian inference for quantiles. Journal of Nonparametric Statistics 2005;

17:385–400. DOI: 10.1080/10485250500039049.11. Yu K, Moyeed R. Bayesian quantile regression. Statistics and Probability Letters 2001; 54:437–447.12. Pollard D. Asymptotics for least absolute deviation regression. Econometric Theory 1991; 7:186–199.13. Yee TW. Quantile regression via vector generalized additive models. Statistics in Medicine 2004; 23:2295–2315.14. Machado JA, Santos Silva J. Quantiles for counts. Journal of the American Statistical Association 2005; 100:

1226–1237.15. Beluzzo Jr W. Semiparametric approaches to welfare evaluation in binary response models. Journal of Business

and Economic Statistics 2004; 22(3):322–330.16. Lee M-J. Median regression for ordered discrete response. Journal of Econometrics 1992; 51:59–77.17. Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian Data Analysis (2nd edn). Chapman & Hall/CRC: Boca

Raton, 2003.18. Cabinet Office. Wiring it up. Cabinet Office: London, 2000.19. Treasury HM. Choosing the Right Fabric: A Framework for Performance Information. HM Treasury: London,

2001.20. Sprigg J. The codes of practice—using the information. In Codes of Practice on the Publication of Information,

Institute for Local Government (ed.). University of Birmingham: Birmingham, 1981; 55–65.21. National Audit Office. Measuring the Performance of Government Departments. National Audit Office: London,

2001.22. Department of the Environment. Transport and the Regions. Best Value and Audit Commission Performance

Indicators for 2000/2001. Volume 1: The Performance Indicators Including the Publication of InformationDirection 1999 (England). DETR: London, 1999.

23. Royal Statistical Society Working Party on Performance Monitoring in the Public Services. Performance indicators:Good, bad and ugly. Journal of the Royal Statistical Society, Series A 2005; 168:1–27.

24. House of Commons Public Adminstration Select Committee. On Target? Government by Measurement vol. 1.Stationary Office: London, 2003.

25. Radin BA. The government performance and Results Act (GPRA: Hydra Headed Monster or Flexible ManagementTool). Public Administration Review 1998; 58(4):307–316.

26. Osborne D, Gaebler T. Reinventing Government. Addison Wesley: Reading, MA, 1992.27. National Partnership for Reinventing Government. Balancing Measures: Best Practices in Performance

Management. National Partnership for Reinventing Government: Washington, DC, 1999.28. Treasury HM. Spending Review 2000, with Public Service Agreements and Service Delivery Agreements. HM

Treasury: London, 2000.29. OECD-PUMA. Performance Auditing and the Modernisation of Government. OECD: Paris, 1996.30. Audit Commission. On Target: The Practice of Performance Indicators. Audit Commission: London, 2000.31. National Audit Office. Inappropriate Adjustments to NHS Waiting Lists. National Audit Office: London, 2001.32. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for

Statistical Computing: Vienna, Austria, 2005. ISBN 3-900051-07-0.33. Gelman A, Rubin D. Inference from iterative simulation using multiple sequences. Statistical Science 1992;

7:457–511.34. Geweke J. Evaluating the accuracy of sampling-based approaches to calculating posterior moments. In Bayesian

Statistics 4, Bernado J, Berger J, Dawid A, Smith A (eds). Clarendon Press: Oxford, 1992.35. Heidelberger P, Welch P. Simulation run length control in the presence of an initial transient. Journal of Operations

Research 1983; 31:1109–1144.36. Winkelmann R. Reforming health care; evidence from quantile regression for counts. Journal of Health Economics

2006; 25:131–145.37. Poister TH, Streib G. Performance measurement in municipal government: assessing the state of the practice.

Public Adminstration Review 1999; 59(4):325–335.38. Draper D, Gittoes M. Statistical analysis of performance indicators in U.K. higher education (with Discussion).

Journal of the Royal Statistical Society, Series A 2004; 167:449–474.



39. Hu XJ, Lawless JF, Suzuki K. Nonparametric estimation of a lifetime distribution when censoring times aremissing. Technometrics 1998; 40:3–13.

40. Schennach SM. Bayesian exponentially tilted empirical likelihood. Biometrika 2005; 92:31–46.41. Hauck K, Street A. Performance assessment in the context of multiple objectives: a multivariate multilevel

analysis. Technical Report, York University, 2004.42. Aigner D, Lovell C, Schmidt P. Formulation and estimation of stochastic frontier production function models.

Journal of Econometrics 1977; 6:21–37.43. Meeusen W, van den Broeck J. Efficiency estimation from Cobb–Douglas production functions with composed

error. International Economic Review 1977; 8:435–444.44. Spottiswoode C. Improving Police Performance: A New Approach to Measuring Police Efficiency. Public Services

Productivity Panel, HM Treasury: London, 2000.45. Stone M. How not to measure the efficiency of public services (and how one might). Journal of the Royal

Statistical Society, Series A 2002; 165:405–434.46. Breiman L, Friedman JH. Predicting multivariate responses in multiple linear regression. Journal of the Royal

Statistical Society, Series B 1997; 59:3–54.47. Bailey TC, Hewson PJ. Simultaneous modelling of multiple traffic safety performance indicators using a

multivariate generalised linear mixed model. Journal of the Royal Statistical Society, Series A 2004; 167(3):551–567.

48. Dunson DB, Watson M, Taylor JA. Bayesian latent variable models for median regression on multiple outcomes.Biometrics 2003; 59(2):296–304.


quantile regression for binary performance indicators

Documents