‘in the eye of the beholder’: perceptions of local impact in english health action zones

10
Social Science & Medicine 59 (2004) 1603–1612 ‘In the eye of the beholder’: perceptions of local impact in English Health Action Zones Helen Sullivan a, *, Ken Judge b , Kate Sewel b a Faculty of the Built Environment, University of the West of England, Cities Research Centre, Coldharbour Lane, Bristol, BS16 1QY, UK b Health Promotion Policy Unit, University of Glasgow, 1 Lilybank Gardens, Glasgow, G12 8RZ, UK Abstract Contemporary efforts to promote population health improvement and to reduce inequalities in the UK are characterised by their complexity as they engage with a multiplicity of agencies and sectors. Additionally, the emphasis on promoting evidence-based practice has challenged evaluators tasked with collecting and interpreting evidence of impact in complex local health economies. National policy makers, local implementers and other stakeholders will have varying perspectives on impact and the Labour Government’s centralising tendencies have acted to ‘crowd out’ local voices from the policy process. Drawing on the national evaluation of Health Action Zones (HAZ) this article ‘gives voice’ to local stakeholders and their perceptions of impact. Informed by a Theories of Change perspective, we explore HAZ interventions to articulate the nature of impact and its limits. We analyse the claims made by local HAZs with reference to the evidence base and examine their significance in the context of overall HAZ objectives. We conclude that local implementer perspectives are no less sophisticated than those at the policy centre of central government, but that they are informed by three important factors: the local context, a need to be pragmatic and the limited potency of evidence in the public policy system. r 2004 Elsevier Ltd. All rights reserved. Keywords: Evaluation; Impact; Health Action Zones; Local perspectives; UK Background The UK government’s pursuit of evidence-based policy making has stimulated various initiatives to improve public services and tackle cross-cutting pro- blems (Davies, Nutley, & Smith, 2000; Newman, 2001; Sullivan & Skelcher, 2002). The recent programme of action to ‘modernise’ health services and to reduce health inequalities is one example of this approach (Powell, 2002). The evaluator’s task here is to obtain evidence about ‘what works’, which involves making judgements about impact and success. This is challen- ging because: the effects of individual interventions will be many and inter-linked, judgements about impact need to consider the force and the fact of any impact, that is whether impact is ‘sufficient’ given investment, and ‘success’ is likely to be inter- preted differently amongst stakeholders. Initiatives designed by central government but delivered through local partnerships will elicit different perspectives about impacts and success which, if they are not taken account of, could hamper shared learning and adversely affect central/local relations. Indeed, this article contends that the centralising tendencies of the Labour Government have already limited the capacity of local perspectives to inform national policy debates. Drawing on the experience of the national Health Action Zone (HAZ) initiative in England the article aims to ‘give voice’ to local stakeholders by exploring issues of impact and success from their perspective. Informed by the Theories of Change approach, it examines HAZ interventions deemed successful by local stakeholders and articulates the nature of impact and its limits as experienced within localities. ARTICLE IN PRESS *Corresponding author. Tel.: +44-117-328-3999. E-mail address: [email protected] (H. Sullivan). 0277-9536/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.socscimed.2004.02.013

Upload: helen-sullivan

Post on 12-Sep-2016

212 views

Category:

Documents


0 download

TRANSCRIPT

Social Science & Medicine 59 (2004) 1603–1612

ARTICLE IN PRESS

*Correspond

E-mail addr

0277-9536/$ - se

doi:10.1016/j.so

‘In the eye of the beholder’: perceptions of local impact inEnglish Health Action Zones

Helen Sullivana,*, Ken Judgeb, Kate Sewelb

a Faculty of the Built Environment, University of the West of England, Cities Research Centre, Coldharbour Lane, Bristol, BS16 1QY, UKb Health Promotion Policy Unit, University of Glasgow, 1 Lilybank Gardens, Glasgow, G12 8RZ, UK

Abstract

Contemporary efforts to promote population health improvement and to reduce inequalities in the UK are

characterised by their complexity as they engage with a multiplicity of agencies and sectors. Additionally, the emphasis

on promoting evidence-based practice has challenged evaluators tasked with collecting and interpreting evidence of

impact in complex local health economies. National policy makers, local implementers and other stakeholders will have

varying perspectives on impact and the Labour Government’s centralising tendencies have acted to ‘crowd out’ local

voices from the policy process. Drawing on the national evaluation of Health Action Zones (HAZ) this article ‘gives

voice’ to local stakeholders and their perceptions of impact. Informed by a Theories of Change perspective, we explore

HAZ interventions to articulate the nature of impact and its limits. We analyse the claims made by local HAZs with

reference to the evidence base and examine their significance in the context of overall HAZ objectives. We conclude that

local implementer perspectives are no less sophisticated than those at the policy centre of central government, but that

they are informed by three important factors: the local context, a need to be pragmatic and the limited potency of

evidence in the public policy system.

r 2004 Elsevier Ltd. All rights reserved.

Keywords: Evaluation; Impact; Health Action Zones; Local perspectives; UK

Background

The UK government’s pursuit of evidence-based

policy making has stimulated various initiatives to

improve public services and tackle cross-cutting pro-

blems (Davies, Nutley, & Smith, 2000; Newman, 2001;

Sullivan & Skelcher, 2002). The recent programme of

action to ‘modernise’ health services and to reduce

health inequalities is one example of this approach

(Powell, 2002). The evaluator’s task here is to obtain

evidence about ‘what works’, which involves making

judgements about impact and success. This is challen-

ging because: the effects of individual interventions

will be many and inter-linked, judgements about

impact need to consider the force and the fact of

any impact, that is whether impact is ‘sufficient’ given

ing author. Tel.: +44-117-328-3999.

ess: [email protected] (H. Sullivan).

e front matter r 2004 Elsevier Ltd. All rights reserve

cscimed.2004.02.013

investment, and ‘success’ is likely to be inter-

preted differently amongst stakeholders. Initiatives

designed by central government but delivered through

local partnerships will elicit different perspectives

about impacts and success which, if they are not

taken account of, could hamper shared learning and

adversely affect central/local relations. Indeed, this

article contends that the centralising tendencies of

the Labour Government have already limited the

capacity of local perspectives to inform national policy

debates. Drawing on the experience of the national

Health Action Zone (HAZ) initiative in England the

article aims to ‘give voice’ to local stakeholders by

exploring issues of impact and success from their

perspective. Informed by the Theories of Change

approach, it examines HAZ interventions deemed

successful by local stakeholders and articulates the

nature of impact and its limits as experienced within

localities.

d.

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121604

HAZs were one of the first New Labour Area Based

Initiatives (ABIs), established to act as ‘trailblazers’,

developing innovative cross-sector approaches to health

improvement and health inequalities. Between 1997 and

1999, 26 local partnerships bid successfully for HAZ

status in England. HAZs varied enormously in size,

configuration across health and local authority bound-

aries and geographical location. Each was required to

develop locally owned implementation plans and to

monitor and report progress annually to the Depart-

ment of Health (the ‘host’ central government depart-

ment) via a nationally sanctioned performance

management regime. The Department also funded a

national evaluation of Health Action Zones to, ‘identify

and assess the conditions in which strategies to create a

more substantial capacity for local collaboration result in

the adoption of change mechanisms that lead to the

modernisation of services and a reduction in health

inequalities’ (Bauld & Judge, 2002, p. 9). Individual

HAZs were expected to commission local evaluations

and a national/local evaluation network was established

to facilitate learning across HAZs.

Evaluating impact and theories of change

For Chen, evaluating impact is about understanding,

‘how successful the program is in achieving its purposes

and/or through what kinds of causal mechanisms it will

operate’ (1990, p. 143). Chen also links evidence of

impact with judgements of success about the interven-

tion. This is not necessarily a straightforward relation-

ship but one mediated by the operating context (Pawson

& Tilley, 1997). For example, the specified purposes for

the HAZ initiative (reducing health inequalities, moder-

nising services and empowering communities) were

ambitious, long term and contingent upon several

factors, including the deployment of appropriate colla-

borative capacity within the zone. Shortly after their

establishment, changes in national political leadership

and health strategy destablised HAZ partnerships by

overlaying new national priorities on local programmes,

realigning HAZ budgets and offering only short-term

commitment to their continuance. Evaluator assess-

ments of impact in this turbulent context are likely to be

modest. However, local (or other) judgements about the

success of the programme may be more positive,

informed by perceptions of achievement in spite rather

than because of the prevailing environment.

Beyond the specific context, other factors can limit the

utility of evidence in relation to impact. Political factors

are significant in the UK as the political cycle means that

governments are frequently unwilling to wait for new

policy interventions to ‘bed down’ before demanding

evidence of impact. Central–local relations are such that

the argument that wide stakeholder engagement in

evaluation can improve the evidence base is diluted

and a ‘top down’ perspective dominates which ‘crowds

out’ the views of others (Henry, 2002; Nutley, Davies, &

Walter, 2002). Technical factors arising from the

complexity of the relationships associated with meeting

cross-cutting goals mean that causality may not easily be

established. Attempts to unravel causal relationships can

result in costly evaluation programmes which may not

be considered worthwhile investments. Finally, institu-

tions in the UK are less likely to be persuaded of the

value of evaluation (Sanderson, 2002). Sanderson (2001)

identifies the tendency to ‘blame’ officials rather than

provide the support to address problems as a key

cultural factor rendering local government ‘antithetical’

to a learning culture. This is also evident within

central government, notwithstanding its expressed com-

mitment to evidence-based practice (Powell & Ex-

worthy, 2001).

An approach that claims to take account of context,

involve relevant stakeholders and address causality is

Theories of Change, defined by Connell and Kubisch

(1998, p. 16) as ‘a systematic and cumulative study of the

links between activities, outcomes and contexts of the

initiative’. Theories of Change is one of a range of

theory-driven approaches to evaluation. It also meets

Chen’s criteria for impact evaluations having both of the

following characteristics:

* ‘y[T]he impact evaluation uses theory-guided stra-

tegies to generate a broad evidence base to assess the

impact of the treatment on the outcome.* When specifying the outcome in the study, the impact

evaluation uses both the stakeholders’ views and the

existing theory and knowledge related to the program

to assess the important intended and unintended

impacts’ (Chen, 1990, p. 144).

Theories of Change was adopted by the national

evaluation team as a means of achieving its aims and

supporting the development of a learning ethos. While

our experience of using the approach led us to identify

limits to its utility in complex policy systems (see Barnes,

Matka, & Sullivan, 2003; Barnes, Sullivan, & Matka,

2004), its focus on evidencing linkages between inter-

ventions and outcomes proved helpful in exploring how

stakeholders made judgements about impact. This is

discussed below.

Methods

In our study of impact, we identified HAZs that had

not been subject to in-depth case analysis in order to,

capture evidence that might not otherwise have been

available to the national evaluation team and to give

these HAZs an opportunity to articulate their percep-

tions in the context of the evaluation. Eight HAZs (out

of 10) were able to participate. We asked each to identify

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1605

three key local successes and then visited each HAZ to

explore the reasoning and evidence base for their

selection. HAZs responded by identifying between three

and eight local successes and we included them all. In

each HAZ we reviewed core HAZ documentation,

interviewed the HAZ manager and those responsible

for the ‘successful’ initiatives. Where possible we also

interviewed key partners, attended meetings of ‘success-

ful’ projects, engaged with community members, inter-

viewed local evaluators and examined local evaluation

material. We sought to answer the following questions:

* Was the HAZ strategy implemented as planned?* What criteria were used to judge success?* What evidence supports this judgement?* How significant are these projects/processes to the

overall HAZ strategy?* To what outcomes do these successes contribute?* Could this project/process have been successful

without HAZ?* How far have the goals of the HAZ been achieved?* How far have the needs of targeted groups been met?* What contribution has HAZ made to: addressing

inequalities, modernising services, enhancing learn-

ing, developing new modes of governance and

improving community involvement (the national

HAZ objectives)?

For the analysis of local HAZ impact, we adopted a

modified version of Theories of Change. Fig. 1 outlines

the HAZ evaluation’s Theory of Change framework

which begins by examining community needs and goals

and then specifies appropriate interventions. As our

starting point in the impact analysis was the range of

interventions/activities/processes localities had identified

as ‘successful’ we needed to use Theories of Change

rather differently. Drawing on our documentary data,

interview transcripts and observation materials, we

examined the specified interventions to establish the

rationales given for them and to assess how far they

were justified as part of a wider HAZ strategy. We

also scrutinised the more general claims made about

Community Resources

and Challenges

Rationale for

Intervention

Purposeinvestmen

activitieinterventio

process

Context ‘ChMe

Strategy

Fig. 1. HAZ evaluation and Theories of Chan

processes to define success and considered within that:

the use of criteria to determine success, the kind of

evidence given to justify success and the wider role

played by evaluation in the HAZ. We considered how

far the selected interventions were judged to be

successful locally by reference to the evidence presented

of success and its limits. We contextualised this by

examining how significant these interventions were to

the wider local HAZ strategy and considering the

contribution these interventions made to the national

HAZ objectives (see above). What is reflected here is not

the logic of each intervention (potentially 34), but the

most interesting general arguments and differences

across the cases, with discussion of what can be claimed

for the specific contribution of HAZ and its ‘impact’

locally.

Local HAZ impact

The nature and diversity of HAZs spawned a wide

variety of activities and the eight case studies specified 34

‘successful’ interventions. These we classified and then

grouped in relation to their core focus:

* service development* disease focus* targeted population or social group* promotion of learning.

As Fig. 2 illustrates, most locally identified successes

were service related and focused on children and families

or employment. Only two interventions were disease

focused and both of these reflected national priorities.

‘Successful’ interventions were specified in relation to

a number of population groups though in Brent HAZ it

was argued that all their interventions addressed the

needs of Black and minority ethnic communities.

Included among the six learning interventions were the

development of new techniques such as Geographic

Information Systems (GIS) and Health Impact

Assessment (HIA) employed alongside new ways of

ful t in

s, n andes

Negotiation of prospectively

specified expected

consequences yields practical

milestones

ange’ chanism Outcomes

StrategicGoals

Targets

ge. (Source: adopted from Judge (2000)).

ARTICLE IN PRESS

Focus Service (n=16)

Disease (n=2)

Population/Group (n=10)

Learning (n=6)

Employment (n=3) Children and Family support (n=4) Healthy schools Welfare advice (n=2) Voluntary sector development (n=2) Outreach services- mental health Health Living Centres Minor ailments – pharmacy scheme Intermediate care

Cancer Pulmonary rehabilitation

Young people Community participation (n=3) Smoking cessation Ethnic minorities (n=2) Older people (n=2) Sexual and reproductive health

Health impact assessment (n=2) Stakeholder evaluation GIS Whole system events Education and training

Fig. 2. Successful HAZ initiatives.

H. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121606

working, including whole systems events and stake-

holder evaluation.

In many cases multiple foci are identifiable, so,

smoking cessation has a disease as well as a population

focus and innovations in intermediate care have wider

implications for learning about ‘what works’ in terms of

joining up and modernising services. While some

interventions are specific, such as the minor ailments

pharmacy scheme, others represent programmes of

activity containing a number of streams of work, such

as Leicester HAZ’s Children’s strategy.

Rationales

Establishing a rationale requires those involved to

articulate why and how a proposed activity will lead to

pre-determined outcomes. Rationales ranged from those

that were intuitive and cited as a ‘common sense’

response to local circumstances, such as Bury and

Rochdale HAZ’s decision to promote change in health

service design and delivery for Black and minority ethnic

communities because, ‘everyone knows that service take-

up among Black and minority ethnic groups is poor’, to

those that were supported by considerable material

evidence, often from examples of similar projects else-

where. For example, in Cornwall and the Isles of Scilly a

pulmonary rehabilitation programme was based on

project workers’ access to evidence about the impact

of such programmes in the US. Elsewhere, project

rationale was derived from a close examination of the

local context. This was the case in Wolverhampton

where an employment scheme emerged following a

convergence of factors including: long-term recruitment

and retention problems in local health services, a

predicted increase in demand for staff following health

service expansion, significant incidence of long-term

unemployment in the locality and better intelligence

about the employment aspirations of local Black and

minority ethnic groups.

It was more difficult to establish links between

rationales for specific interventions and the local HAZ

strategy. This may have been because the links between

interventions and improved health or reduced health

inequalities were perceived as self evident and not

requiring more explicit justification (increased income

arising from a successful benefit take-up campaign

contributes to better health among beneficiaries). In

other cases, it may be because those individuals involved

in specific interventions were unaware of the HAZ

strategy and saw it as someone else’s job to link the two.

There was evidence that core HAZ staff had engaged in

some form of strategic planning in all case studies,

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1607

though to different degrees. In Wakefield, the core HAZ

team oversaw the production of a planning document

which linked specific projects to themes and to over-

arching goals, while in Leicester there was evidence of

this kind of planning within work streams.

An influencing factor was the instability of many

HAZ strategies over time. In a minority of cases this was

generated by the local context, so, in one HAZ where

there was little experience of cross-sector partnership

working, capacity gaps were evident, leading one

interviewee to reflect on how the HAZ strategy could

be compromised because, ‘every time you advertise a job

you have a fear of no applications and then a fear of the

standard of the applicants’.

However, the most pronounced instability caused to

HAZ strategies was that prompted by national changes.

Three issues emerged. The first was the shift in HAZ

priorities following a change in Secretary of State for

Health. The impact on local strategies depended on how

much pre-existing commitment there was to the prio-

rities of coronary heart disease, cancer and mental

health. The development of National Service

Frameworks (national operating arrangements

for key service areas) also interrupted the implementa-

tion of some strategies as local priorities changed to

capitalise on national opportunities. But the

biggest interruption to HAZ strategies was the

persistent uncertainty about the priority of the HAZ

initiative within the Department of Health from 2000.

This resulted in the truncation of certain projects,

difficulties in staff recruitment and a reorienting of

attention away from HAZ to newer, more promising

initiatives.

Local definitions of success

Within HAZs definitions of success comprised several

elements, including national emphases, local experience

and/or individual perspectives. Five key features

emerged:

* Measurable indicators of success were important for

HAZ projects although they were of variable utility

particularly if HAZs were measuring what could be

measured (outputs) rather than what should be

measured (outcomes).* At the same time qualitative material was frequently

considered more telling of impact. This included

information from project users, increased participa-

tion by stakeholders in projects and the ‘stories’ told

by beneficiaries of the ‘transformations’ wrought by

HAZ.* Acknowledging the contribution of process elements

was very important, in particular, the development of

collaborative capacity within localities, such as

partnership and community involvement infrastruc-

ture.* Linked to this was the adoption of new ways of

thinking and working, either as part of a deliberate

strategy as in ‘whole systems learning’ or as an

indirect result of a HAZ intervention.* Finally, a key element of HAZ success was its

contribution over the long term. This included the

identification of how particular interventions were

‘rolled out’ to a wider group, possibly through the

use of other funding or mainstreaming.

The use of criteria to measure success

The criteria used by HAZs were a mixture of those set

by project commissioners and those set within the

project. HAZ targets were generally quantitative

although qualitative measures were used in their

absence. However, these were difficult to make tangible.

For example, Leicester’s mental health outreach pro-

gramme sought to assess impact on users by considering

changes to their quality of life. However, which quality

of life measures were appropriate and how to establish

and assess changes in relation to these were identified as

major challenges for the project staff and users. Across

the cases the setting of criteria or measures was

exclusively positive, i.e. focused upon what the inter-

vention was intended to achieve, with no references to

the assessment of any contra-indicators of HAZ

activities.

Successful HAZ interventions made use of both

‘explicit’ and ‘implicit’ logic in their assessments (Four-

nier, 1995). For HAZ interventions based on national

priorities, success rested with their achievement of pre-

specified targets (teenage pregnancy or smoking cessa-

tion), indicating the use of ‘explicit logic’. However,

other projects operated on the basis of ‘implicit logic’,

where judgements about performance were made on the

basis of what the project actually achieved given its

context.

Target setting was considered to be straightforward

where there were national standards but more difficult

where projects had to determine targets for themselves.

For example, in the ‘community grants’ schemes of Bury

and Rochdale and Wolverhampton communities had to

identify a health dimension to work they were doing in

order to receive money from the scheme. This was time

consuming and resource intensive as groups required

support from workers to fulfil this obligation. It also

highlights the importance of tailoring assessment re-

quirements to the resources invested. Elsewhere, target

setting was derived from experience, with the welfare

benefits’ take-up work in Wakefield containing targets

derived from the project managers’ estimation of what

was possible given the budget available. Having the

opportunity to review and revise targets was considered

vital, particularly in innovative projects. This relied

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121608

upon a regular reviewing process which could allow a

project ‘have a go’ before changing targets if they proved

to be wildly over- or under-optimistic.

The links between targets and outcomes were often ill-

defined both in relation to specific activities and in

linking HAZ targets to the achievement of wider HAZ

goals. In Brent and Bury and Rochdale HAZs, the use

of Health Impact Assessment techniques had alerted

workers to the need for the stakeholder deliberation

process to establish clear links between specified

activities and health outcomes and the difficulty of

doing so. In other cases, links between targets and wider

goals were unclear because target setting had assumed

another purpose. Wolverhampton’s teenage pregnancy

scheme specified a very ambitious target, not because it

was necessarily achievable but because of the message it

sent to others about the HAZs’ intent in relation to

teenage pregnancy.

The use of evidence

Many kinds of evidence were presented to support

claims of successful interventions. The extraction of

‘hard evidence’ was a dominant concern partly

reflecting the performance management ethos of

central government, but also reflecting a professional

desire to ‘prove’ the health benefits of interventions to

their clinical colleagues. However, frequently

it was ‘soft data’ which was more readily available and

relevant to understanding the impact of a specific

intervention.

Sometimes it was possible to provide evidence of the

specific health impact of an intervention, as in the case

of the pulmonary rehabilitation scheme. On other

occasions evidence suggestive of a potential contribution

to health derived from more immediate positive impacts

elsewhere. The child health and well-being project run in

Bury and Rochdale HAZ worked with homeless families

in hostels to address the needs of children in ‘families

with multiple problems and chaotic lifestyles’. Some of

these problems were directly health related but the

project’s emphasis was to provide children with a stable

and safe environment to promote their general well

being.

Evidence was used to support ‘stories of individual

transformation’, particularly in relation to interventions

with a community involvement or development focus,

such as the improved sense of self esteem of a man who

had been long-term unemployed but found a new lease

of life as a result of the HAZ sponsored volunteering

scheme in Wakefield. The movement of HAZ staff to

better jobs as a result of their achievements in HAZ was

cited as an indicator of success, not just in terms of

individual achievement but through the expectation that

the learning from HAZ would be translated into new

environments.

Within projects evidence was vital in determining

future directions. In Hull and East Riding HAZ the

success of the smoking cessation scheme provided a

direct catalyst to the development of a wider strategy for

the area. On some occasions the success of a local

project coincided with a determination by central

government to adopt a similar policy nationally, as in

the case of Wakefield HAZ’s project to improve

pathways of care for cancer patients. There were also

examples where evidence had influenced the redirection

of projects. For example, a ‘family learning initiative’ in

one HAZ appeared to have achieved the target number

of participants. However, closer examination revealed

that the scheme was not working with the constituency-

in-need. Consequently, the project was redesigned and

redirected to reach the relevant group.

For a number of interventions the ambition was to

affect change in the ‘quality of life’ of those involved. A

key problem for many HAZs was that relevant data

simply were not available in a usable form, as data were

collected on different scales, over different time periods

and with different degrees of population coverage.

This use of evidence to both plan for and evaluate

interventions was cited as key in the development of

Leicester HAZ’s ‘children and families’ strategy, where

the team used the national performance management

regime to facilitate peer analysis of both key needs and

appropriate responses to achieve positive outcomes for

children. The strategy was managed through regular

reviews and the collection and analysis of project-level

evidence. For stakeholders in Leicester, this approach

was considered very important in the absence of

nationally determined children’s targets; providing

opportunities to develop locally appropriate interven-

tions. In the words of the strategy director, ‘The

government throws you lots of pieces of different

puzzles and they are very interested to know what the

picture is that you are producing’.

While all HAZs engaged in the collection of evidence

through the performance monitoring process, it was

difficult to establish how meaningful that information

was outside of its specific purpose. In some cases HAZ

interviewees dismissed it as irrelevant and unhelpful to

their management of the scheme.

The role of evaluation

HAZs fulfilled their obligation to establish local

evaluation in a variety of ways. Some like Cornwall

and the Isles of Scilly awarded significant funding to

local evaluators to support the implementation of the

programme and evaluate impact, resulting in a wide

range of evaluation activities (Asthana, 2002). Others,

including Wakefield employed external local evaluators

throughout the life of HAZ though with more modest

resources applied to support learning through evalua-

tion (McCabe, Wilde, & Wilson, 2001). However, there

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1609

were different routes. In Bury and Rochdale there was

an expressed preference for ‘doing’, so summative local

evaluation was commissioned in the latter stages. Brent

HAZ focused on an expansive internal performance

monitoring process, supported by separately funded

evaluations of specific projects and an innovative

‘citizen’s jury’ to consider HAZ impact in 2002.

Wolverhampton HAZ took the decision to employ

‘community evaluators’, individuals who were trained

by the HAZ and evaluated HAZ projects from a

stakeholder perspective (the results sometimes creating

tension with project managers).

Evaluation was not always viewed positively by HAZ

officials. Typical concerns were that, ‘they are only

telling us what we already know’, and that evaluators

were so close to projects or so constrained by the data

they collected that they were, ‘not able to make objective

judgements’. There were contrary views from those

HAZs with substantial investments in evaluation. For

example, in Cornwall and the Isles of Scilly evaluation

was valued in relation to HAZ and beyond,

‘I think actually the use of evaluation is now

becoming a much more integral element of service

development, so they are looking much more care-

fully atyhow they’re going to know if it’s effective

andyI think that has becomeya more usual

element of service development’.

None of the HAZs believed that a greater evaluation

investment would have benefited them. This may be a

reflection of their experiences with central government in

relation to performance management in the latter part of

the HAZ programme. HAZs complained of continuing

to send monitoring returns to government offices but

receiving no response in 2002. It may also have been due

to the fact that in some HAZs investment in evaluation

was not perceived to have informed decision making.

Judgements of success

The eight HAZ case studies highlighted a number of

ways in which HAZ had been successful. It had:

* Introduced a non-medical perspective to health, not

just focused on medicine and cure,* Encouraged closer working relationships between

health and social services,* Facilitated change to/introduction of mainstream

services,* Prepared the ground for lasting real partnership

working—HAZ provided the necessary infrastruc-

ture/framework for coordinating everyone,* Stimulated the involvement of ‘the public’ as citizens

and service users,* Raised the profile of important health-related issues

and got them onto local agendas,

* Precipitated a culture/attitude shift and introduced

the ‘HAZ way of working’,* Facilitated shared learning,* Enabled experimentation.

Most of these examples point to qualitative rather

than quantitative measures and make reference to

process issues and new ways of working leading to

longer-term improvements. However, respondents were

quick to point out that the potential impact of HAZ had

been limited, both by national policy changes and the

uncertainty they introduced and other factors, such as:

the loss of interest amongst key ‘champions’, the

attendant feelings of isolation experienced by some

project workers and the lack of real community

involvement despite considerable activity. There were

particular suggestions that the emphasis on targets may

have paradoxically acted to limit local success and

therefore dilute local impact. Respondents argued that

the early demands for evidence of ‘quick wins’ from

central government required them to commit to targets

too early and subsequently reduced their room for

manoeuvre if the targets proved inappropriate.

The significance of successful projects to the overall HAZ

strategy

A number of the interventions outlined in Fig. 2 were

expressly concerned with the reduction of health inequal-

ities. This manifested itself in targeted action in

particular geographical areas, such as the Healthy

Living Centres in Bury and Rochdale and community

action within the employment workstream in Leicester.

Elsewhere, it resulted in prioritised action for particular

target groups including Black and minority ethnic

communities in the employment project in Wolver-

hampton and school children in the Healthy Schools

initiative in Wakefield. While HAZ officials were

confident about their rationale for these actions, they

were also aware that they needed to be sustained over

time and linked into a wider programme to complement

these specific interventions. The instability associated

with the HAZ initiative limited these possibilities and

meant that their contribution to reducing health

inequalities would be partial.

Local HAZs sought to realise national ambitions

through targeting activity at particular groups. The

rationale here was that these groups were poorly served

by mainstream agencies and HAZ provided an oppor-

tunity to address this, and/or that targeted action would

result in long-term benefits. Target groups included

Black and minority ethnic communities, children, young

and older people. Sometimes activity was aimed at all of

the members of a particular group, but usually factors

such as deprivation or health status delimited eligibility.

Whether HAZs could claim they had met the expressed

needs of these groups depended upon: the evidence of

ARTICLE IN PRESS

1 Each of which have extensive publications listed on the

national HAZ website, www.haznet,org.uk/hazs/evidence/local.

H. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121610

‘need’ that had existed in advance of HAZ (and upon

which the justification for targeting was based), the

degree to which group members were involved in their

‘needs assessment’ and the availability of information to

indicate that identified needs had been met. HAZs were

able to provide evidence in relation to the first two of

these and sometimes they could provide evidence of

stakeholder evaluation. Notwithstanding this, the inter-

ventions’ futures were uncertain unless they were

adopted into the mainstream.

More evidence was available concerning the contribu-

tion of HAZ to the modernisation of services. Services

had been ‘joined-up’ to improve their effectiveness,

like the intermediate care programme in Hull and

East Riding, service ‘gaps’ had been filled, including

the child health and homelessness initiative developed

in Bury and Rochdale and completely new services

were developed, such as the employment services in

Wolverhampton and Leicester HAZs. These interven-

tions each demonstrated synergy: the intermediate

care programme stimulated wider thinking amongst

health and social care professionals about how other

services could be better ‘joined-up’, the child health and

homelessness initiative was extended and expanded via

the use of a new government programme (the Children’s

Fund), the employment project in Wolverhampton

formed the basis for good practice within the new

primary care service, while the employment programme

in Leicester became a key part of the delivery of local

training.

The assessment in relation to community empower-

ment is more mixed. All eight HAZs invested in

community involvement or development activities, and

there was evidence that HAZ had contributed to

enhancing the influence of community and voluntary

sector members in key decision making forums. In

Brent, the HAZ project led to the community and

voluntary sectors having a significant impact on way in

which new national policy instruments, Local Strategic

Partnerships, evolved including how they were repre-

sented on this partnership. However, there was a

widespread concern that the HAZ contribution may

not be sustainable and that any progress in promoting

community empowerment may be short-lived. Often

(notwithstanding the Brent example) it was difficult to

identify the ways in which communities had become

more influential, particularly at the strategic level. For

one HAZ this was not entirely unexpected as it perceived

the ethos of the government’s health service modernisa-

tion programme to be entirely opposed to that of

HAZ—the target-driven top down approach of the

former overpowering the community empowerment

ethos of the latter. Beyond this was an acknowledgement

that community empowerment required long-term

commitment coupled with a clear analysis of what is

sought through ‘empowerment’ and what is involved in

facilitating ‘empowerment’, which may not have been

present in all HAZs.

HAZs experienced considerable success in contribut-

ing to the understanding and building of partnership as

a new mode of governance. Contributions were identified

amongst neighbourhood, sectoral and strategic partner-

ships, and this was consistently highlighted and evi-

denced in each of the eight HAZs. Partnership activity

provided localities with greater understanding of the

kind of capacity needed to support collaboration. It also

helped to create the capacity for new directions in local

action, for example, in one HAZ the experience of

partnership working enabled a focus on neighbourhoods

to develop that had not been permitted previously owing

to political tensions. It enabled new opportunities to be

identified and taken, as in Leicester where the partner-

ship capacity of the HAZ enabled it to take advantage of

new initiatives such as ‘LIFT’, a capital improvements

programme.

Finally, while a number of HAZs referred to the

learning generated from HAZ that they and others had

benefited from, this article has also recounted incidences

where the product of local learning has been diminished

following other pressures on decision making. Overall,

while some HAZs documented their learning more

comprehensively than others,1 it remained difficult to

identify means by which learning had been system-

atically collected and embedded.

Conclusion

This article has examined ways in which under-

standings of health ‘success’ and ‘impact’ are articulated

and applied. Detailed study of eight HAZs has suggested

that local perspectives are no less sophisticated than

those of the national policy centre, but that they are

informed by three important factors.

First, the local context is significant in determining

where effort will be expended and impact felt and this

may not fit with central government objectives or a

nationally determined ‘expected focus’. Secondly, the

need to justify their existence to unpredictable national

funders means that localities have become adept at

laying claim to impacts in all sorts of areas. To this

extent there is an element of ‘knowing what the rules of

the (political) game are and being able to play by them’.

Finally, local agencies have become attuned to the way

in which evaluation of ‘impact’ remains constrained by

the technical, political and cultural limits highlighted

early in this paper. What has resulted is the operation of

a cost/benefit balancing act where evaluation is only

entered into when it is clearly beneficial or unavoidable

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1611

or where it will not detract from the expenditure of

resources on intervention. One consequence of this was

that while HAZs found it possible to specify and

evidence impact at project and to some extent work-

stream level, evidencing impact beyond this, such as in

relation to achievement of wider cross-cutting outcomes,

was not easily obtained. Henry (2002) suggests that to

do this requires a closer linkage between programme

indicators and social outcomes, which would almost

certainly require a greater investment in evaluation than

many HAZs have been prepared to make.

Overall, HAZs do not appear to have been over-

ambitious in their claims. They readily identified the

limits to what could be claimed for health inequalities

(timescales too short, the need for national policies to

accompany local action) but were more confident in

their claims for service modernisation. The greatest local

impact of HAZ, however, rests with its contribution to

improving local capacity to collaborate (see Adams,

Alcock, & Brown, 2002; Clarke, Jones, Carr, Molyneux,

& Proctor, 2002; Halliday, Richardson, & Ashtana,

2002, for different local perspectives and Barnes et al.

(2004) for a HAZ wide discussion of this). However,

even here what is often cited cannot be reduced to the

effect of a single intervention, as in many areas HAZ did

not act alone but rather in concert with other initiatives

to achieve common goals.

The relative flexibility of funding meant that local

HAZs were able to play a variety of roles as

circumstances demanded including catalyst, accelerator

of agreed activity, partnership broker, ‘safe’ experimen-

ter and risk taker and vacuum filler. While these

contributions were not always recognised by the centre,

they were vitally important in securing the impact of

HAZ at local level.

At the heart of the tensions identified in this paper are

the abrasive central/local relations that characterise the

workings of key institutions such as local government

and health and which are currently manifest in the

‘national standards versus local diversity’ debate (Glen-

dinning, Powell, & Rummery, 2002; Stewart, 2003).

Local partnerships’ determination to access resources

released through national initiatives make them prone to

over-claiming what can be achieved in their bids. This

inflates expectations and complicates the relationship

between evaluators and localities. In this regard,

approaches such as Theories of Change have proved

useful in facilitating more considered deliberation about

community needs and possibilities but here too their

potential is hampered by the desire to win funds or

freedoms to act.

Evaluation, particularly of ABIs has been adversely

affected by this tension as national and local evaluation

perspectives run the risk of persisting in opposition

rather than combining to more positive effect. The

potential for complementarity is significant given the

current acknowledgement that context does matter.

Local evaluations are able to provide a richer picture

of the fate of interventions in their local context,

incorporate wider stakeholder perspectives, and explore

the potential value of actions taken in one initiative to

subsequent initiatives or mainstream activity. National

evaluations are better positioned to deliver macroana-

lysis of the relevant policy instrument. However, without

a recognition of the value of both the potential for

complementarity will not be realised. At best,

there will be attempts to bring the two together through

learning networks (as in the HAZ experience) while at

worst the national will attempt to dictate terms to the

local.

Acknowledgements

The authors are grateful to the eight HAZs that

agreed to participate in this element of the national

evaluation. This paper is based on research funded by

the Department of Health. The views expressed are

those of the authors and not necessarily those of the

Department of Health.

References

Adams, L., Alcock, P., & Brown, G. (2002). Wakefield and

district HAZ, partnerships and learning. In L. Bauld, & K.

Judge (Eds.), Learning from health action zones (pp. 85–94).

Chichester: Aeneas Press.

Asthana, S. (2002). Evaluation Programme. Report to steering

group, Cornwall and the Isles of Scilly HAZ, University of

Plymouth.

Barnes, M., Matka, E., & Sullivan, H. (2003). Evidence,

understanding and complexity. Evaluation, 9, 265–284.

Barnes, M., Sullivan, H., & Matka, E. (2004). The development

of collaborative capacity in health action zones. A final report

from the national evaluation. University of Birmingham,

Birmingham.

Bauld, L., & Judge, K. (2002). The development of

health action zones. In L. Bauld, & K. Judge (Eds.),

Learning from health action zones (pp. 1–16). Chichester:

Aeneas Press.

Chen, H. T. (1990). Theory driven evaluation. London: Sage.

Clarke, C., Jones, D., Carr, S., Molyneux, J., & Proctor, S.

(2002). Tyne and Wear HAZ—Patterns of engagement and

entrenchment. Newcastle: University of Northumbria at

Newcastle.

Connell, J. P., & Kubisch, A. C. (1998). Applying a Theory of

Change approach to the evaluation of comprehensive

community initiatives: progress, prospects and problems.

In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell

(Eds.), New approaches to evaluating community initiatives:

Volume 2—Theory, measurement and analysis (pp. 15–44).

Washington, DC: The Aspen Institute.

ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121612

Davies, H. T. O., Nutley, S. M., & Smith, P. C. (2000). What

works? Evidence based policy and practice in public services.

Bristol: Policy Press.

Fournier, D. (1995). Establishing evaluative conclusions: A

distinction between general and working logic. New direc-

tions for program evaluation, 68, 15–32.

Glendinning, C., Powell, M., & Rummery, K. (Eds.). (2002).

Partnerships, new labour and the governance of welfare.

Bristol: Policy Press.

Halliday, J., Richardson, S., & Asthana, S. (2002). A role for

HAZs within local strategic partnerships: Lessons from the

south west. In L. Bauld, & K. Judge (Eds.), Learning from

health action zones (pp. 95–108). Chichester: Aeneas Press.

Henry, G. T. (2002). Choosing criteria to judge program

success. Evaluation, 8, 182–204.

Judge, K. (2000). Testing evaluation to the limits: The case of

English health action zones. Journal of Health Services

Research and Policy, 5, 3–6.

McCabe, A., Wilde, P., & Wilson, M. (2001). Wakefield HAZ:

Project case studies. ENACT, University of Birmingham

and Confederation of Community Workers’ Training

Groups Collective.

Newman, J. (2001). Modernising governance. London: Sage.

Nutley, S., Davies, H., & Walter, I. (2002). Evidence based

policy and practice: Cross-sector lessons from the UK. ESRC

Working Paper 9, ESRC UK Centre for Evidence Based

Policy and Practice, Queen Mary, University of London.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. London:

Sage.

Powell, M. (Ed.). (2002). Evaluating new labour’s welfare

reforms. Bristol: Policy Press.

Powell, M., & Exworthy, M. (2001). Joined-up solutions to

address health inequalities: Analysing policy, process and

resource streams. Public Money and Management, 21,

21–26.

Sanderson, I. (2001). Performance management, evaluation and

learning in ‘modern’ local government. Public Administra-

tion, 79, 297–313.

Sanderson, I. (2002). Evaluation, policy learning and evidence

based policy making. Public Administration, 80, 1–22.

Stewart, J. (2003). Modernising British local government.

Basingstoke: Palgrave.

Sullivan, H., & Skelcher, C. (2002). Working across boundaries.

Collaboration in public services. Basingstoke: Palgrave.