MLLSM01

EVENTS POLICY

LECTURE 8

Evaluating Events

Lecture content

This lecture will discuss why monitoring and evaluation are so important to event policy makers and practitioners

It will also review aspects of the evaluation process

It will then reflect upon the principal techniques of evaluation utilised within the events arena

Finally, it will consider why net impacts are the most important facet of evaluating the success or failure of an event policy intervention

The politics of evaluation

Moore & Sykes (2000) argue that “monitoring and evaluation is closely linked with policy development at a strategic level and when specific projects are being designed and implemented” (p203)

Related to the development of event aims and objectives – often derived from political aims

So, the “approach to evaluation, the choice of what is measured and the judgement as to what has been achieved cannot be divorced from the wider political or cultural context” (Moore & Sykes, 2000: p203)

Political actors ‘set the agenda’ (Lukes’ theory of power) and monitoring and evaluation legitimates it and produces transparency and accountability

Evaluating Events

Despite the predominance of economic logic in the evaluation of event impacts until recently, this is changing as the socio-cultural and image impacts rise in importance (Richard and Wilson, 2004)

This does not mean that the economic is now defunct; rather that it has been augmented by a wider understanding of social, cultural and environmental impacts – the economic is not enough

As politics is about the allocation of scare resources, investment in an events-led urban strategy needs to be justified/legitimated on the basis of objective outcome and output measures – does it work (e.g. Commonwealth Games)

Especially as we see “significant public sector expenditure to develop, underwrite and promote event-based tourism development” (Connell & Page, 2005: p64): LA’s and LECs provide seed funding to encourage events growth and need to be

satisfied of a ROI – EventScotland also invests in international/regional and local events on the basis of a return – tourist visits, regional identity, local importance.

The purpose of evaluation

To judge whether there is a rationale for policy intervention/are interventions producing the desired outcomes (Moore & Spires, 2000). But also to: Monitor progress of projects against target – has hosting the

Special Olympics increased overseas visitation to Glasgow/Has the investment in EventScotland been worth it

Review and revision of strategy in light of new information/challenges – policy adjustment (e.g. new approach to extending visitor stay required)

Overall judgement on success or failure of strategy/policy – performance related and identifying causes of failure

Evaluation in both public and private sector event contexts now ever more connected to KPIs – numbers, type of visitors, media profile, reduced reliance on public purse

Internal and External Evaluation

Internal (organisational): Team analysis and performance review

What worked, what can be improved upon, which systems need to be revised, which org structure is most effective

Development of KPIs to quantify objective performance of team: Customer complaints, resident complaints, deadlines

met, media exposure

External (environmental/macro factors): Reinforcing customer-focus Seeking quantitative/qualitative responses on the

customer ‘experience’ Visitor numbers, impression management

When to evaluate

Three principal windows of opportunity for evaluating events:

Pre-event: Expectations, economics, ownership, ticket

purchasesDuring event:

Reliant on dynamic methodologies and support of organisers/funding partners

Post-event: Legacy (traffic, housing, pollution, community

pride) Online presence, building relationship databases

Methods of event evaluation

Quantitative measures: Visitor numbers (how many, where from: pre-booked tickets,

pay as you go) Spend per head (how much spent and on what element of

the event) – economic additionalities Customer surveys, business surveys (Connell & Page,

2005), online mechanisms Qualitative measures:

How did customers find out about the event (Advert, flyer, radio, press, word of mouth)

Expectations and satisfaction (rate satisfaction with event cost, value, product, impressions of destination)

Visual methods (video, camera), interviews, workshops, focus groups

New tools: online digital media can now facilitate customer/visitor interaction and ongoing evaluation

Evaluating the social, cultural & environmental

Waitts (1999; 2003) one of the first to consider socio-cultural dimensions of events: He concluded that Sydney 2000 had brought about a shift from

welfare to entrepreneurial goals; a functional transformation of the cityscapes to generate cultural capital and, the growing importance of media images

However, politically, the economic still garners more support

But, social models are being developed and cultural capital can be as much about image, perception and impressions as about immediate economic return

The environmental is increasingly important as sustainable events become the target of organisers

Evaluating Images

Hiller (1989;1998) and Ritchie & Smith (1991) the first to seriously consider the image effects of major events

Now, as Hall (1992: 155) remarks, “hallmark events may be the regarded as the image makers of modern tourism”

Started out with narrow definition of ‘image’ and one-off studies – failing to consider the multi-dimensional aspects of image and the benefits of longitudinal studies

Richard & Wilson (2004) used designative (informational) and appraisive (feelings, values and meanings) elements of images to evaluate the impact of European Cultural Capital on Rotterdam: Considered residents and tourists and cultural sector stakeholders Focused on images attributes including: modern architecture

(designative), water (designative), multicultural, working city, international, dynamic, culture and art, lots to discover, events (designative), shopping (designative), nightlife (designative), cozy, unsafe

From mega to community evaluation

Evaluation strategies need to be specific to the nature and scope of the event and to be able to identifying potential spatial inequity (Connell & Page, 2005)

In justifying the ‘benefits’ of attracting a large scale event to your city/country, organisers are expected to measure ROI, additional tourist spend, increased dwell time, infrastructural legacy etc.

In contrast, the success (or failure) of a local community events strategy will be justified on the basis of building social capital (networks, sense of belonging), pride in place, sense of community, reductions in crime, quality of life

The challenge for policy makers is that benefits of the mega/hallmark event are easier to document (economic impact assessments) than the longer term and more subjective social and cultural indicators

Policy makers need to develop social impact tools to make the intangible tangible – only then will the social and cultural be taken seriously

Gross and net impacts

Difference between what would have happened had the policy intervention not taken place and what additional benefits have accrued because of an intervention

Links back to Carlsen’s (2004) arguments about multiplier effect – what added value does winning the rights to host the UEFA Cup Final have on a city – that wouldn’t have accrued anyway

To reach net additional impact, value has to be assigned to: Substitution – policy targeted investment takes the place of other investments Displacement – additional activity from some sectors counterbalanced by less

output from others (tourism displacement as a result of Olympic Games hosting)

Indirect impacts – additional spend in the local economy as a result of increased business for some firms

The political issue is whether public sector investment leverages additional benefits and provides a healthy ROI for public stakeholders (e.g. London 2012 regeneration)

Strategic Objectives

ECONOMIC PHYSICAL SOCIAL

Tourism employment

Environment/infrastructure

QOL, community cohesion

Inputs (expenditure)

Scot Exec funds Other public sector Private sector

Activity measures

Business ass Training Neighbourhoodrenewal

Participation in arts

Reduction in crime

Output and Outcome measures

Start –ups Event jobs Ext of cultural activity

No of performances

No of volunteers

Gross impacts

Adapted from PACEC (1999)

Gross additionality – displacement plus linkages/multipliers = Net Impacts

References

Carlsen, J, Getz, D & Soutar, G (2000) ‘Event Evaluation Research’, Event Management, 6 (4): 247-257

Richards, G & Wilson, J (2004) The Impact of Cultural Events on City Image: Rotterdam, Cultural Capital of Europe 2001, Urban Studies, 41 (10): 1931-1951

Connell, J. & Page, J.S. (2005) Evaluating the Economic and Spatial Effects of an Event: The Case of the World Medical and Health Games.  Tourism Geographies, Vol 7 (1), pp 63-85.

Moore, B & Spires, R (2000) Monitoring and Evaluation. In: Roberts, P & Sykes, H (eds) Urban Regeneration: A Handbook, London. Sage