Creative evaluation tools: Beyond ‘tired and tested’Karen Gray Willis Newson
Agenda
• Aims for the workshop• A very quick introduction to ‘Creative and Credible’
evaluation • Practical exercise: Re-imagining ‘tired and tested’ • Summary
Background: Willis Newson• Specialist arts and health consultancy developing
and delivering projects in acute, mental health and community settings
• Long-term collaboration with Norma Daykin, Professor of Arts and Health at University of West of England
• UWE led research-based evaluations of Willis Newson arts in mental health programmes: Moving On & Arts @Callington Road
• Collaborative research and evaluation work culminated in 2 year Knowledge Transfer Partnership (2010/12) Decorative glazing by Stuart Low in Older
Adult Unit, Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health NHS Partnership Trust. Photo; Paul Highnam
Evaluation: Three key principles
• Appropriate• Credible• Ethical
All evaluation needs to embody these principles.
Appropriate Appropriate for:
– Evaluation aims and objectives– Participants or stakeholder group– Project context
Some examples:– Focus groups will not give you
quantitative health outcomes data– Validated scales may not be
useful in projects with fewparticipants
– Questionnaires may not be appropriate for people with questionnaire fatigue
Creative exercise – useful for aims setting. Image : Willis Newson.
Credible
• Timing• Objectivity• Robust application and
analysable results
Some examples:– If you tell focus group participants
that the evaluation findings will help secure funding for next year’s project, will they give you the full story?
– How can you measure outcomes for participants if you don’t know where they started?
Image from a ‘Savouring & Flow’ workshop run by Light Box as part of The Happiness Project. Photo: BS6 Photography.
Ethical
• Respect• Confidentiality• Professional integrity
Some examples:– A participant has given consent
for you to use their words, have you told them exactly where those words will appear?
– A participant has agreed to be a case study in your evaluation, but then changes his mind. What do you do? Collage created during a focus group exercise designed to
explore the impacts of a project on participants.
So, why be creative? • To think around problems• To counter evaluation fatigue• To get meaningful responses that
you can’t get in other ways• To make best use of your skills and
those of your participants• To reward and engage your
respondents • To make evaluation more a part of
your project
“Creativity is not an escape from disciplined thinking, it is an escape with disciplined thinking.”
[Jerry Hirshberg]
Reflective practice drawings by Elizabeth Jane Grosse for Arts at Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health Partnership NHS Trust. Image: Willis Newson.
Tired and tested : Scenario Jenny is the manager of a pilot arts on referral scheme, currently planning how to evaluate it. The project is council-funded and will operate in three GP surgeries in Plymouth, serving the city’s most deprived communities, including those where English is a second language. Intended as a first step intervention for patients with mild to moderate anxiety and depression, it aims to build self-esteem and confidence, develop social networks within the community and offer a positive distraction. Participants will be referred by their GP for a block of 10 sessions. Two local artists will deliver a weekly art session at each surgery covering a variety of different art forms including painting, creative writing and photography. There will be approximately 10 participants in each group, although this number is likely to vary from week to week. A practice nurse will be on hand to offer advice or support or signpost participants to other resources. Jenny needs to evidence the impact the sessions have on participants and advocate for continued funding. She has been asked to use the Patient Health Questionnaire (PHQ9) to measure levels of depression among participants, but she is also eager to understand the subjective impact that the project has on each of the participants and to explore their experience over the 10 weeks.
Tired and Tested: Scenario
Think about:• Who?
– Who will be doing the evaluating?– Who are the participants?
• What?– What form should it take? What will it look like?
• When?– When does it need to happen?
• What are the issues?– Ethics, participant needs, resources, time…
In summary…
• Be appropriate• Be credible• Be ethical• Be creative…
Think about this at every stage:– when you are developing tools– when you are collecting data– when you report and share your
findings
Professional development opportunity:
Evaluating Arts, Health & Wellbeing Initiatives
5 days of training Feb – Apr 2014
Further information and training