‘in the eye of the beholder’: perceptions of local impact in english health action zones
TRANSCRIPT
Social Science & Medicine 59 (2004) 1603–1612
ARTICLE IN PRESS
*Correspond
E-mail addr
0277-9536/$ - se
doi:10.1016/j.so
‘In the eye of the beholder’: perceptions of local impact inEnglish Health Action Zones
Helen Sullivana,*, Ken Judgeb, Kate Sewelb
a Faculty of the Built Environment, University of the West of England, Cities Research Centre, Coldharbour Lane, Bristol, BS16 1QY, UKb Health Promotion Policy Unit, University of Glasgow, 1 Lilybank Gardens, Glasgow, G12 8RZ, UK
Abstract
Contemporary efforts to promote population health improvement and to reduce inequalities in the UK are
characterised by their complexity as they engage with a multiplicity of agencies and sectors. Additionally, the emphasis
on promoting evidence-based practice has challenged evaluators tasked with collecting and interpreting evidence of
impact in complex local health economies. National policy makers, local implementers and other stakeholders will have
varying perspectives on impact and the Labour Government’s centralising tendencies have acted to ‘crowd out’ local
voices from the policy process. Drawing on the national evaluation of Health Action Zones (HAZ) this article ‘gives
voice’ to local stakeholders and their perceptions of impact. Informed by a Theories of Change perspective, we explore
HAZ interventions to articulate the nature of impact and its limits. We analyse the claims made by local HAZs with
reference to the evidence base and examine their significance in the context of overall HAZ objectives. We conclude that
local implementer perspectives are no less sophisticated than those at the policy centre of central government, but that
they are informed by three important factors: the local context, a need to be pragmatic and the limited potency of
evidence in the public policy system.
r 2004 Elsevier Ltd. All rights reserved.
Keywords: Evaluation; Impact; Health Action Zones; Local perspectives; UK
Background
The UK government’s pursuit of evidence-based
policy making has stimulated various initiatives to
improve public services and tackle cross-cutting pro-
blems (Davies, Nutley, & Smith, 2000; Newman, 2001;
Sullivan & Skelcher, 2002). The recent programme of
action to ‘modernise’ health services and to reduce
health inequalities is one example of this approach
(Powell, 2002). The evaluator’s task here is to obtain
evidence about ‘what works’, which involves making
judgements about impact and success. This is challen-
ging because: the effects of individual interventions
will be many and inter-linked, judgements about
impact need to consider the force and the fact of
any impact, that is whether impact is ‘sufficient’ given
ing author. Tel.: +44-117-328-3999.
ess: [email protected] (H. Sullivan).
e front matter r 2004 Elsevier Ltd. All rights reserve
cscimed.2004.02.013
investment, and ‘success’ is likely to be inter-
preted differently amongst stakeholders. Initiatives
designed by central government but delivered through
local partnerships will elicit different perspectives
about impacts and success which, if they are not
taken account of, could hamper shared learning and
adversely affect central/local relations. Indeed, this
article contends that the centralising tendencies of
the Labour Government have already limited the
capacity of local perspectives to inform national policy
debates. Drawing on the experience of the national
Health Action Zone (HAZ) initiative in England the
article aims to ‘give voice’ to local stakeholders by
exploring issues of impact and success from their
perspective. Informed by the Theories of Change
approach, it examines HAZ interventions deemed
successful by local stakeholders and articulates the
nature of impact and its limits as experienced within
localities.
d.
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121604
HAZs were one of the first New Labour Area Based
Initiatives (ABIs), established to act as ‘trailblazers’,
developing innovative cross-sector approaches to health
improvement and health inequalities. Between 1997 and
1999, 26 local partnerships bid successfully for HAZ
status in England. HAZs varied enormously in size,
configuration across health and local authority bound-
aries and geographical location. Each was required to
develop locally owned implementation plans and to
monitor and report progress annually to the Depart-
ment of Health (the ‘host’ central government depart-
ment) via a nationally sanctioned performance
management regime. The Department also funded a
national evaluation of Health Action Zones to, ‘identify
and assess the conditions in which strategies to create a
more substantial capacity for local collaboration result in
the adoption of change mechanisms that lead to the
modernisation of services and a reduction in health
inequalities’ (Bauld & Judge, 2002, p. 9). Individual
HAZs were expected to commission local evaluations
and a national/local evaluation network was established
to facilitate learning across HAZs.
Evaluating impact and theories of change
For Chen, evaluating impact is about understanding,
‘how successful the program is in achieving its purposes
and/or through what kinds of causal mechanisms it will
operate’ (1990, p. 143). Chen also links evidence of
impact with judgements of success about the interven-
tion. This is not necessarily a straightforward relation-
ship but one mediated by the operating context (Pawson
& Tilley, 1997). For example, the specified purposes for
the HAZ initiative (reducing health inequalities, moder-
nising services and empowering communities) were
ambitious, long term and contingent upon several
factors, including the deployment of appropriate colla-
borative capacity within the zone. Shortly after their
establishment, changes in national political leadership
and health strategy destablised HAZ partnerships by
overlaying new national priorities on local programmes,
realigning HAZ budgets and offering only short-term
commitment to their continuance. Evaluator assess-
ments of impact in this turbulent context are likely to be
modest. However, local (or other) judgements about the
success of the programme may be more positive,
informed by perceptions of achievement in spite rather
than because of the prevailing environment.
Beyond the specific context, other factors can limit the
utility of evidence in relation to impact. Political factors
are significant in the UK as the political cycle means that
governments are frequently unwilling to wait for new
policy interventions to ‘bed down’ before demanding
evidence of impact. Central–local relations are such that
the argument that wide stakeholder engagement in
evaluation can improve the evidence base is diluted
and a ‘top down’ perspective dominates which ‘crowds
out’ the views of others (Henry, 2002; Nutley, Davies, &
Walter, 2002). Technical factors arising from the
complexity of the relationships associated with meeting
cross-cutting goals mean that causality may not easily be
established. Attempts to unravel causal relationships can
result in costly evaluation programmes which may not
be considered worthwhile investments. Finally, institu-
tions in the UK are less likely to be persuaded of the
value of evaluation (Sanderson, 2002). Sanderson (2001)
identifies the tendency to ‘blame’ officials rather than
provide the support to address problems as a key
cultural factor rendering local government ‘antithetical’
to a learning culture. This is also evident within
central government, notwithstanding its expressed com-
mitment to evidence-based practice (Powell & Ex-
worthy, 2001).
An approach that claims to take account of context,
involve relevant stakeholders and address causality is
Theories of Change, defined by Connell and Kubisch
(1998, p. 16) as ‘a systematic and cumulative study of the
links between activities, outcomes and contexts of the
initiative’. Theories of Change is one of a range of
theory-driven approaches to evaluation. It also meets
Chen’s criteria for impact evaluations having both of the
following characteristics:
* ‘y[T]he impact evaluation uses theory-guided stra-
tegies to generate a broad evidence base to assess the
impact of the treatment on the outcome.* When specifying the outcome in the study, the impact
evaluation uses both the stakeholders’ views and the
existing theory and knowledge related to the program
to assess the important intended and unintended
impacts’ (Chen, 1990, p. 144).
Theories of Change was adopted by the national
evaluation team as a means of achieving its aims and
supporting the development of a learning ethos. While
our experience of using the approach led us to identify
limits to its utility in complex policy systems (see Barnes,
Matka, & Sullivan, 2003; Barnes, Sullivan, & Matka,
2004), its focus on evidencing linkages between inter-
ventions and outcomes proved helpful in exploring how
stakeholders made judgements about impact. This is
discussed below.
Methods
In our study of impact, we identified HAZs that had
not been subject to in-depth case analysis in order to,
capture evidence that might not otherwise have been
available to the national evaluation team and to give
these HAZs an opportunity to articulate their percep-
tions in the context of the evaluation. Eight HAZs (out
of 10) were able to participate. We asked each to identify
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1605
three key local successes and then visited each HAZ to
explore the reasoning and evidence base for their
selection. HAZs responded by identifying between three
and eight local successes and we included them all. In
each HAZ we reviewed core HAZ documentation,
interviewed the HAZ manager and those responsible
for the ‘successful’ initiatives. Where possible we also
interviewed key partners, attended meetings of ‘success-
ful’ projects, engaged with community members, inter-
viewed local evaluators and examined local evaluation
material. We sought to answer the following questions:
* Was the HAZ strategy implemented as planned?* What criteria were used to judge success?* What evidence supports this judgement?* How significant are these projects/processes to the
overall HAZ strategy?* To what outcomes do these successes contribute?* Could this project/process have been successful
without HAZ?* How far have the goals of the HAZ been achieved?* How far have the needs of targeted groups been met?* What contribution has HAZ made to: addressing
inequalities, modernising services, enhancing learn-
ing, developing new modes of governance and
improving community involvement (the national
HAZ objectives)?
For the analysis of local HAZ impact, we adopted a
modified version of Theories of Change. Fig. 1 outlines
the HAZ evaluation’s Theory of Change framework
which begins by examining community needs and goals
and then specifies appropriate interventions. As our
starting point in the impact analysis was the range of
interventions/activities/processes localities had identified
as ‘successful’ we needed to use Theories of Change
rather differently. Drawing on our documentary data,
interview transcripts and observation materials, we
examined the specified interventions to establish the
rationales given for them and to assess how far they
were justified as part of a wider HAZ strategy. We
also scrutinised the more general claims made about
Community Resources
and Challenges
Rationale for
Intervention
Purposeinvestmen
activitieinterventio
process
Context ‘ChMe
Strategy
Fig. 1. HAZ evaluation and Theories of Chan
processes to define success and considered within that:
the use of criteria to determine success, the kind of
evidence given to justify success and the wider role
played by evaluation in the HAZ. We considered how
far the selected interventions were judged to be
successful locally by reference to the evidence presented
of success and its limits. We contextualised this by
examining how significant these interventions were to
the wider local HAZ strategy and considering the
contribution these interventions made to the national
HAZ objectives (see above). What is reflected here is not
the logic of each intervention (potentially 34), but the
most interesting general arguments and differences
across the cases, with discussion of what can be claimed
for the specific contribution of HAZ and its ‘impact’
locally.
Local HAZ impact
The nature and diversity of HAZs spawned a wide
variety of activities and the eight case studies specified 34
‘successful’ interventions. These we classified and then
grouped in relation to their core focus:
* service development* disease focus* targeted population or social group* promotion of learning.
As Fig. 2 illustrates, most locally identified successes
were service related and focused on children and families
or employment. Only two interventions were disease
focused and both of these reflected national priorities.
‘Successful’ interventions were specified in relation to
a number of population groups though in Brent HAZ it
was argued that all their interventions addressed the
needs of Black and minority ethnic communities.
Included among the six learning interventions were the
development of new techniques such as Geographic
Information Systems (GIS) and Health Impact
Assessment (HIA) employed alongside new ways of
ful t in
s, n andes
Negotiation of prospectively
specified expected
consequences yields practical
milestones
ange’ chanism Outcomes
StrategicGoals
Targets
ge. (Source: adopted from Judge (2000)).
ARTICLE IN PRESS
Focus Service (n=16)
Disease (n=2)
Population/Group (n=10)
Learning (n=6)
Employment (n=3) Children and Family support (n=4) Healthy schools Welfare advice (n=2) Voluntary sector development (n=2) Outreach services- mental health Health Living Centres Minor ailments – pharmacy scheme Intermediate care
Cancer Pulmonary rehabilitation
Young people Community participation (n=3) Smoking cessation Ethnic minorities (n=2) Older people (n=2) Sexual and reproductive health
Health impact assessment (n=2) Stakeholder evaluation GIS Whole system events Education and training
Fig. 2. Successful HAZ initiatives.
H. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121606
working, including whole systems events and stake-
holder evaluation.
In many cases multiple foci are identifiable, so,
smoking cessation has a disease as well as a population
focus and innovations in intermediate care have wider
implications for learning about ‘what works’ in terms of
joining up and modernising services. While some
interventions are specific, such as the minor ailments
pharmacy scheme, others represent programmes of
activity containing a number of streams of work, such
as Leicester HAZ’s Children’s strategy.
Rationales
Establishing a rationale requires those involved to
articulate why and how a proposed activity will lead to
pre-determined outcomes. Rationales ranged from those
that were intuitive and cited as a ‘common sense’
response to local circumstances, such as Bury and
Rochdale HAZ’s decision to promote change in health
service design and delivery for Black and minority ethnic
communities because, ‘everyone knows that service take-
up among Black and minority ethnic groups is poor’, to
those that were supported by considerable material
evidence, often from examples of similar projects else-
where. For example, in Cornwall and the Isles of Scilly a
pulmonary rehabilitation programme was based on
project workers’ access to evidence about the impact
of such programmes in the US. Elsewhere, project
rationale was derived from a close examination of the
local context. This was the case in Wolverhampton
where an employment scheme emerged following a
convergence of factors including: long-term recruitment
and retention problems in local health services, a
predicted increase in demand for staff following health
service expansion, significant incidence of long-term
unemployment in the locality and better intelligence
about the employment aspirations of local Black and
minority ethnic groups.
It was more difficult to establish links between
rationales for specific interventions and the local HAZ
strategy. This may have been because the links between
interventions and improved health or reduced health
inequalities were perceived as self evident and not
requiring more explicit justification (increased income
arising from a successful benefit take-up campaign
contributes to better health among beneficiaries). In
other cases, it may be because those individuals involved
in specific interventions were unaware of the HAZ
strategy and saw it as someone else’s job to link the two.
There was evidence that core HAZ staff had engaged in
some form of strategic planning in all case studies,
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1607
though to different degrees. In Wakefield, the core HAZ
team oversaw the production of a planning document
which linked specific projects to themes and to over-
arching goals, while in Leicester there was evidence of
this kind of planning within work streams.
An influencing factor was the instability of many
HAZ strategies over time. In a minority of cases this was
generated by the local context, so, in one HAZ where
there was little experience of cross-sector partnership
working, capacity gaps were evident, leading one
interviewee to reflect on how the HAZ strategy could
be compromised because, ‘every time you advertise a job
you have a fear of no applications and then a fear of the
standard of the applicants’.
However, the most pronounced instability caused to
HAZ strategies was that prompted by national changes.
Three issues emerged. The first was the shift in HAZ
priorities following a change in Secretary of State for
Health. The impact on local strategies depended on how
much pre-existing commitment there was to the prio-
rities of coronary heart disease, cancer and mental
health. The development of National Service
Frameworks (national operating arrangements
for key service areas) also interrupted the implementa-
tion of some strategies as local priorities changed to
capitalise on national opportunities. But the
biggest interruption to HAZ strategies was the
persistent uncertainty about the priority of the HAZ
initiative within the Department of Health from 2000.
This resulted in the truncation of certain projects,
difficulties in staff recruitment and a reorienting of
attention away from HAZ to newer, more promising
initiatives.
Local definitions of success
Within HAZs definitions of success comprised several
elements, including national emphases, local experience
and/or individual perspectives. Five key features
emerged:
* Measurable indicators of success were important for
HAZ projects although they were of variable utility
particularly if HAZs were measuring what could be
measured (outputs) rather than what should be
measured (outcomes).* At the same time qualitative material was frequently
considered more telling of impact. This included
information from project users, increased participa-
tion by stakeholders in projects and the ‘stories’ told
by beneficiaries of the ‘transformations’ wrought by
HAZ.* Acknowledging the contribution of process elements
was very important, in particular, the development of
collaborative capacity within localities, such as
partnership and community involvement infrastruc-
ture.* Linked to this was the adoption of new ways of
thinking and working, either as part of a deliberate
strategy as in ‘whole systems learning’ or as an
indirect result of a HAZ intervention.* Finally, a key element of HAZ success was its
contribution over the long term. This included the
identification of how particular interventions were
‘rolled out’ to a wider group, possibly through the
use of other funding or mainstreaming.
The use of criteria to measure success
The criteria used by HAZs were a mixture of those set
by project commissioners and those set within the
project. HAZ targets were generally quantitative
although qualitative measures were used in their
absence. However, these were difficult to make tangible.
For example, Leicester’s mental health outreach pro-
gramme sought to assess impact on users by considering
changes to their quality of life. However, which quality
of life measures were appropriate and how to establish
and assess changes in relation to these were identified as
major challenges for the project staff and users. Across
the cases the setting of criteria or measures was
exclusively positive, i.e. focused upon what the inter-
vention was intended to achieve, with no references to
the assessment of any contra-indicators of HAZ
activities.
Successful HAZ interventions made use of both
‘explicit’ and ‘implicit’ logic in their assessments (Four-
nier, 1995). For HAZ interventions based on national
priorities, success rested with their achievement of pre-
specified targets (teenage pregnancy or smoking cessa-
tion), indicating the use of ‘explicit logic’. However,
other projects operated on the basis of ‘implicit logic’,
where judgements about performance were made on the
basis of what the project actually achieved given its
context.
Target setting was considered to be straightforward
where there were national standards but more difficult
where projects had to determine targets for themselves.
For example, in the ‘community grants’ schemes of Bury
and Rochdale and Wolverhampton communities had to
identify a health dimension to work they were doing in
order to receive money from the scheme. This was time
consuming and resource intensive as groups required
support from workers to fulfil this obligation. It also
highlights the importance of tailoring assessment re-
quirements to the resources invested. Elsewhere, target
setting was derived from experience, with the welfare
benefits’ take-up work in Wakefield containing targets
derived from the project managers’ estimation of what
was possible given the budget available. Having the
opportunity to review and revise targets was considered
vital, particularly in innovative projects. This relied
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121608
upon a regular reviewing process which could allow a
project ‘have a go’ before changing targets if they proved
to be wildly over- or under-optimistic.
The links between targets and outcomes were often ill-
defined both in relation to specific activities and in
linking HAZ targets to the achievement of wider HAZ
goals. In Brent and Bury and Rochdale HAZs, the use
of Health Impact Assessment techniques had alerted
workers to the need for the stakeholder deliberation
process to establish clear links between specified
activities and health outcomes and the difficulty of
doing so. In other cases, links between targets and wider
goals were unclear because target setting had assumed
another purpose. Wolverhampton’s teenage pregnancy
scheme specified a very ambitious target, not because it
was necessarily achievable but because of the message it
sent to others about the HAZs’ intent in relation to
teenage pregnancy.
The use of evidence
Many kinds of evidence were presented to support
claims of successful interventions. The extraction of
‘hard evidence’ was a dominant concern partly
reflecting the performance management ethos of
central government, but also reflecting a professional
desire to ‘prove’ the health benefits of interventions to
their clinical colleagues. However, frequently
it was ‘soft data’ which was more readily available and
relevant to understanding the impact of a specific
intervention.
Sometimes it was possible to provide evidence of the
specific health impact of an intervention, as in the case
of the pulmonary rehabilitation scheme. On other
occasions evidence suggestive of a potential contribution
to health derived from more immediate positive impacts
elsewhere. The child health and well-being project run in
Bury and Rochdale HAZ worked with homeless families
in hostels to address the needs of children in ‘families
with multiple problems and chaotic lifestyles’. Some of
these problems were directly health related but the
project’s emphasis was to provide children with a stable
and safe environment to promote their general well
being.
Evidence was used to support ‘stories of individual
transformation’, particularly in relation to interventions
with a community involvement or development focus,
such as the improved sense of self esteem of a man who
had been long-term unemployed but found a new lease
of life as a result of the HAZ sponsored volunteering
scheme in Wakefield. The movement of HAZ staff to
better jobs as a result of their achievements in HAZ was
cited as an indicator of success, not just in terms of
individual achievement but through the expectation that
the learning from HAZ would be translated into new
environments.
Within projects evidence was vital in determining
future directions. In Hull and East Riding HAZ the
success of the smoking cessation scheme provided a
direct catalyst to the development of a wider strategy for
the area. On some occasions the success of a local
project coincided with a determination by central
government to adopt a similar policy nationally, as in
the case of Wakefield HAZ’s project to improve
pathways of care for cancer patients. There were also
examples where evidence had influenced the redirection
of projects. For example, a ‘family learning initiative’ in
one HAZ appeared to have achieved the target number
of participants. However, closer examination revealed
that the scheme was not working with the constituency-
in-need. Consequently, the project was redesigned and
redirected to reach the relevant group.
For a number of interventions the ambition was to
affect change in the ‘quality of life’ of those involved. A
key problem for many HAZs was that relevant data
simply were not available in a usable form, as data were
collected on different scales, over different time periods
and with different degrees of population coverage.
This use of evidence to both plan for and evaluate
interventions was cited as key in the development of
Leicester HAZ’s ‘children and families’ strategy, where
the team used the national performance management
regime to facilitate peer analysis of both key needs and
appropriate responses to achieve positive outcomes for
children. The strategy was managed through regular
reviews and the collection and analysis of project-level
evidence. For stakeholders in Leicester, this approach
was considered very important in the absence of
nationally determined children’s targets; providing
opportunities to develop locally appropriate interven-
tions. In the words of the strategy director, ‘The
government throws you lots of pieces of different
puzzles and they are very interested to know what the
picture is that you are producing’.
While all HAZs engaged in the collection of evidence
through the performance monitoring process, it was
difficult to establish how meaningful that information
was outside of its specific purpose. In some cases HAZ
interviewees dismissed it as irrelevant and unhelpful to
their management of the scheme.
The role of evaluation
HAZs fulfilled their obligation to establish local
evaluation in a variety of ways. Some like Cornwall
and the Isles of Scilly awarded significant funding to
local evaluators to support the implementation of the
programme and evaluate impact, resulting in a wide
range of evaluation activities (Asthana, 2002). Others,
including Wakefield employed external local evaluators
throughout the life of HAZ though with more modest
resources applied to support learning through evalua-
tion (McCabe, Wilde, & Wilson, 2001). However, there
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1609
were different routes. In Bury and Rochdale there was
an expressed preference for ‘doing’, so summative local
evaluation was commissioned in the latter stages. Brent
HAZ focused on an expansive internal performance
monitoring process, supported by separately funded
evaluations of specific projects and an innovative
‘citizen’s jury’ to consider HAZ impact in 2002.
Wolverhampton HAZ took the decision to employ
‘community evaluators’, individuals who were trained
by the HAZ and evaluated HAZ projects from a
stakeholder perspective (the results sometimes creating
tension with project managers).
Evaluation was not always viewed positively by HAZ
officials. Typical concerns were that, ‘they are only
telling us what we already know’, and that evaluators
were so close to projects or so constrained by the data
they collected that they were, ‘not able to make objective
judgements’. There were contrary views from those
HAZs with substantial investments in evaluation. For
example, in Cornwall and the Isles of Scilly evaluation
was valued in relation to HAZ and beyond,
‘I think actually the use of evaluation is now
becoming a much more integral element of service
development, so they are looking much more care-
fully atyhow they’re going to know if it’s effective
andyI think that has becomeya more usual
element of service development’.
None of the HAZs believed that a greater evaluation
investment would have benefited them. This may be a
reflection of their experiences with central government in
relation to performance management in the latter part of
the HAZ programme. HAZs complained of continuing
to send monitoring returns to government offices but
receiving no response in 2002. It may also have been due
to the fact that in some HAZs investment in evaluation
was not perceived to have informed decision making.
Judgements of success
The eight HAZ case studies highlighted a number of
ways in which HAZ had been successful. It had:
* Introduced a non-medical perspective to health, not
just focused on medicine and cure,* Encouraged closer working relationships between
health and social services,* Facilitated change to/introduction of mainstream
services,* Prepared the ground for lasting real partnership
working—HAZ provided the necessary infrastruc-
ture/framework for coordinating everyone,* Stimulated the involvement of ‘the public’ as citizens
and service users,* Raised the profile of important health-related issues
and got them onto local agendas,
* Precipitated a culture/attitude shift and introduced
the ‘HAZ way of working’,* Facilitated shared learning,* Enabled experimentation.
Most of these examples point to qualitative rather
than quantitative measures and make reference to
process issues and new ways of working leading to
longer-term improvements. However, respondents were
quick to point out that the potential impact of HAZ had
been limited, both by national policy changes and the
uncertainty they introduced and other factors, such as:
the loss of interest amongst key ‘champions’, the
attendant feelings of isolation experienced by some
project workers and the lack of real community
involvement despite considerable activity. There were
particular suggestions that the emphasis on targets may
have paradoxically acted to limit local success and
therefore dilute local impact. Respondents argued that
the early demands for evidence of ‘quick wins’ from
central government required them to commit to targets
too early and subsequently reduced their room for
manoeuvre if the targets proved inappropriate.
The significance of successful projects to the overall HAZ
strategy
A number of the interventions outlined in Fig. 2 were
expressly concerned with the reduction of health inequal-
ities. This manifested itself in targeted action in
particular geographical areas, such as the Healthy
Living Centres in Bury and Rochdale and community
action within the employment workstream in Leicester.
Elsewhere, it resulted in prioritised action for particular
target groups including Black and minority ethnic
communities in the employment project in Wolver-
hampton and school children in the Healthy Schools
initiative in Wakefield. While HAZ officials were
confident about their rationale for these actions, they
were also aware that they needed to be sustained over
time and linked into a wider programme to complement
these specific interventions. The instability associated
with the HAZ initiative limited these possibilities and
meant that their contribution to reducing health
inequalities would be partial.
Local HAZs sought to realise national ambitions
through targeting activity at particular groups. The
rationale here was that these groups were poorly served
by mainstream agencies and HAZ provided an oppor-
tunity to address this, and/or that targeted action would
result in long-term benefits. Target groups included
Black and minority ethnic communities, children, young
and older people. Sometimes activity was aimed at all of
the members of a particular group, but usually factors
such as deprivation or health status delimited eligibility.
Whether HAZs could claim they had met the expressed
needs of these groups depended upon: the evidence of
ARTICLE IN PRESS
1 Each of which have extensive publications listed on the
national HAZ website, www.haznet,org.uk/hazs/evidence/local.
H. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121610
‘need’ that had existed in advance of HAZ (and upon
which the justification for targeting was based), the
degree to which group members were involved in their
‘needs assessment’ and the availability of information to
indicate that identified needs had been met. HAZs were
able to provide evidence in relation to the first two of
these and sometimes they could provide evidence of
stakeholder evaluation. Notwithstanding this, the inter-
ventions’ futures were uncertain unless they were
adopted into the mainstream.
More evidence was available concerning the contribu-
tion of HAZ to the modernisation of services. Services
had been ‘joined-up’ to improve their effectiveness,
like the intermediate care programme in Hull and
East Riding, service ‘gaps’ had been filled, including
the child health and homelessness initiative developed
in Bury and Rochdale and completely new services
were developed, such as the employment services in
Wolverhampton and Leicester HAZs. These interven-
tions each demonstrated synergy: the intermediate
care programme stimulated wider thinking amongst
health and social care professionals about how other
services could be better ‘joined-up’, the child health and
homelessness initiative was extended and expanded via
the use of a new government programme (the Children’s
Fund), the employment project in Wolverhampton
formed the basis for good practice within the new
primary care service, while the employment programme
in Leicester became a key part of the delivery of local
training.
The assessment in relation to community empower-
ment is more mixed. All eight HAZs invested in
community involvement or development activities, and
there was evidence that HAZ had contributed to
enhancing the influence of community and voluntary
sector members in key decision making forums. In
Brent, the HAZ project led to the community and
voluntary sectors having a significant impact on way in
which new national policy instruments, Local Strategic
Partnerships, evolved including how they were repre-
sented on this partnership. However, there was a
widespread concern that the HAZ contribution may
not be sustainable and that any progress in promoting
community empowerment may be short-lived. Often
(notwithstanding the Brent example) it was difficult to
identify the ways in which communities had become
more influential, particularly at the strategic level. For
one HAZ this was not entirely unexpected as it perceived
the ethos of the government’s health service modernisa-
tion programme to be entirely opposed to that of
HAZ—the target-driven top down approach of the
former overpowering the community empowerment
ethos of the latter. Beyond this was an acknowledgement
that community empowerment required long-term
commitment coupled with a clear analysis of what is
sought through ‘empowerment’ and what is involved in
facilitating ‘empowerment’, which may not have been
present in all HAZs.
HAZs experienced considerable success in contribut-
ing to the understanding and building of partnership as
a new mode of governance. Contributions were identified
amongst neighbourhood, sectoral and strategic partner-
ships, and this was consistently highlighted and evi-
denced in each of the eight HAZs. Partnership activity
provided localities with greater understanding of the
kind of capacity needed to support collaboration. It also
helped to create the capacity for new directions in local
action, for example, in one HAZ the experience of
partnership working enabled a focus on neighbourhoods
to develop that had not been permitted previously owing
to political tensions. It enabled new opportunities to be
identified and taken, as in Leicester where the partner-
ship capacity of the HAZ enabled it to take advantage of
new initiatives such as ‘LIFT’, a capital improvements
programme.
Finally, while a number of HAZs referred to the
learning generated from HAZ that they and others had
benefited from, this article has also recounted incidences
where the product of local learning has been diminished
following other pressures on decision making. Overall,
while some HAZs documented their learning more
comprehensively than others,1 it remained difficult to
identify means by which learning had been system-
atically collected and embedded.
Conclusion
This article has examined ways in which under-
standings of health ‘success’ and ‘impact’ are articulated
and applied. Detailed study of eight HAZs has suggested
that local perspectives are no less sophisticated than
those of the national policy centre, but that they are
informed by three important factors.
First, the local context is significant in determining
where effort will be expended and impact felt and this
may not fit with central government objectives or a
nationally determined ‘expected focus’. Secondly, the
need to justify their existence to unpredictable national
funders means that localities have become adept at
laying claim to impacts in all sorts of areas. To this
extent there is an element of ‘knowing what the rules of
the (political) game are and being able to play by them’.
Finally, local agencies have become attuned to the way
in which evaluation of ‘impact’ remains constrained by
the technical, political and cultural limits highlighted
early in this paper. What has resulted is the operation of
a cost/benefit balancing act where evaluation is only
entered into when it is clearly beneficial or unavoidable
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–1612 1611
or where it will not detract from the expenditure of
resources on intervention. One consequence of this was
that while HAZs found it possible to specify and
evidence impact at project and to some extent work-
stream level, evidencing impact beyond this, such as in
relation to achievement of wider cross-cutting outcomes,
was not easily obtained. Henry (2002) suggests that to
do this requires a closer linkage between programme
indicators and social outcomes, which would almost
certainly require a greater investment in evaluation than
many HAZs have been prepared to make.
Overall, HAZs do not appear to have been over-
ambitious in their claims. They readily identified the
limits to what could be claimed for health inequalities
(timescales too short, the need for national policies to
accompany local action) but were more confident in
their claims for service modernisation. The greatest local
impact of HAZ, however, rests with its contribution to
improving local capacity to collaborate (see Adams,
Alcock, & Brown, 2002; Clarke, Jones, Carr, Molyneux,
& Proctor, 2002; Halliday, Richardson, & Ashtana,
2002, for different local perspectives and Barnes et al.
(2004) for a HAZ wide discussion of this). However,
even here what is often cited cannot be reduced to the
effect of a single intervention, as in many areas HAZ did
not act alone but rather in concert with other initiatives
to achieve common goals.
The relative flexibility of funding meant that local
HAZs were able to play a variety of roles as
circumstances demanded including catalyst, accelerator
of agreed activity, partnership broker, ‘safe’ experimen-
ter and risk taker and vacuum filler. While these
contributions were not always recognised by the centre,
they were vitally important in securing the impact of
HAZ at local level.
At the heart of the tensions identified in this paper are
the abrasive central/local relations that characterise the
workings of key institutions such as local government
and health and which are currently manifest in the
‘national standards versus local diversity’ debate (Glen-
dinning, Powell, & Rummery, 2002; Stewart, 2003).
Local partnerships’ determination to access resources
released through national initiatives make them prone to
over-claiming what can be achieved in their bids. This
inflates expectations and complicates the relationship
between evaluators and localities. In this regard,
approaches such as Theories of Change have proved
useful in facilitating more considered deliberation about
community needs and possibilities but here too their
potential is hampered by the desire to win funds or
freedoms to act.
Evaluation, particularly of ABIs has been adversely
affected by this tension as national and local evaluation
perspectives run the risk of persisting in opposition
rather than combining to more positive effect. The
potential for complementarity is significant given the
current acknowledgement that context does matter.
Local evaluations are able to provide a richer picture
of the fate of interventions in their local context,
incorporate wider stakeholder perspectives, and explore
the potential value of actions taken in one initiative to
subsequent initiatives or mainstream activity. National
evaluations are better positioned to deliver macroana-
lysis of the relevant policy instrument. However, without
a recognition of the value of both the potential for
complementarity will not be realised. At best,
there will be attempts to bring the two together through
learning networks (as in the HAZ experience) while at
worst the national will attempt to dictate terms to the
local.
Acknowledgements
The authors are grateful to the eight HAZs that
agreed to participate in this element of the national
evaluation. This paper is based on research funded by
the Department of Health. The views expressed are
those of the authors and not necessarily those of the
Department of Health.
References
Adams, L., Alcock, P., & Brown, G. (2002). Wakefield and
district HAZ, partnerships and learning. In L. Bauld, & K.
Judge (Eds.), Learning from health action zones (pp. 85–94).
Chichester: Aeneas Press.
Asthana, S. (2002). Evaluation Programme. Report to steering
group, Cornwall and the Isles of Scilly HAZ, University of
Plymouth.
Barnes, M., Matka, E., & Sullivan, H. (2003). Evidence,
understanding and complexity. Evaluation, 9, 265–284.
Barnes, M., Sullivan, H., & Matka, E. (2004). The development
of collaborative capacity in health action zones. A final report
from the national evaluation. University of Birmingham,
Birmingham.
Bauld, L., & Judge, K. (2002). The development of
health action zones. In L. Bauld, & K. Judge (Eds.),
Learning from health action zones (pp. 1–16). Chichester:
Aeneas Press.
Chen, H. T. (1990). Theory driven evaluation. London: Sage.
Clarke, C., Jones, D., Carr, S., Molyneux, J., & Proctor, S.
(2002). Tyne and Wear HAZ—Patterns of engagement and
entrenchment. Newcastle: University of Northumbria at
Newcastle.
Connell, J. P., & Kubisch, A. C. (1998). Applying a Theory of
Change approach to the evaluation of comprehensive
community initiatives: progress, prospects and problems.
In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell
(Eds.), New approaches to evaluating community initiatives:
Volume 2—Theory, measurement and analysis (pp. 15–44).
Washington, DC: The Aspen Institute.
ARTICLE IN PRESSH. Sullivan et al. / Social Science & Medicine 59 (2004) 1603–16121612
Davies, H. T. O., Nutley, S. M., & Smith, P. C. (2000). What
works? Evidence based policy and practice in public services.
Bristol: Policy Press.
Fournier, D. (1995). Establishing evaluative conclusions: A
distinction between general and working logic. New direc-
tions for program evaluation, 68, 15–32.
Glendinning, C., Powell, M., & Rummery, K. (Eds.). (2002).
Partnerships, new labour and the governance of welfare.
Bristol: Policy Press.
Halliday, J., Richardson, S., & Asthana, S. (2002). A role for
HAZs within local strategic partnerships: Lessons from the
south west. In L. Bauld, & K. Judge (Eds.), Learning from
health action zones (pp. 95–108). Chichester: Aeneas Press.
Henry, G. T. (2002). Choosing criteria to judge program
success. Evaluation, 8, 182–204.
Judge, K. (2000). Testing evaluation to the limits: The case of
English health action zones. Journal of Health Services
Research and Policy, 5, 3–6.
McCabe, A., Wilde, P., & Wilson, M. (2001). Wakefield HAZ:
Project case studies. ENACT, University of Birmingham
and Confederation of Community Workers’ Training
Groups Collective.
Newman, J. (2001). Modernising governance. London: Sage.
Nutley, S., Davies, H., & Walter, I. (2002). Evidence based
policy and practice: Cross-sector lessons from the UK. ESRC
Working Paper 9, ESRC UK Centre for Evidence Based
Policy and Practice, Queen Mary, University of London.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London:
Sage.
Powell, M. (Ed.). (2002). Evaluating new labour’s welfare
reforms. Bristol: Policy Press.
Powell, M., & Exworthy, M. (2001). Joined-up solutions to
address health inequalities: Analysing policy, process and
resource streams. Public Money and Management, 21,
21–26.
Sanderson, I. (2001). Performance management, evaluation and
learning in ‘modern’ local government. Public Administra-
tion, 79, 297–313.
Sanderson, I. (2002). Evaluation, policy learning and evidence
based policy making. Public Administration, 80, 1–22.
Stewart, J. (2003). Modernising British local government.
Basingstoke: Palgrave.
Sullivan, H., & Skelcher, C. (2002). Working across boundaries.
Collaboration in public services. Basingstoke: Palgrave.