Evaluation demystified: Part 2How to be confident, credible and creativeKaren Gray Willis Newson

Presentation agenda

• Background• Academia ↔ Practice• Making it credible• Making it creative

Reflective practice drawings by Elizabeth Jane Grosse for Arts at Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health Partnership NHS Trust. Image: Willis Newson.

Background: Willis Newson• Specialist arts and health consultancy developing

and delivering projects in acute, mental health and community settings

• Long-term collaboration with Norma Daykin, Professor of Arts and Health at University of West of England

• UWE led research-based evaluations of Willis Newson arts in mental health programmes: Moving On & Arts @Callington Road

• Collaborative research and evaluation work culminated in 2 year Knowledge Transfer Partnership (2010/12) Decorative glazing by Stuart Low in Older

Adult Unit, Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health NHS Partnership Trust. Photo; Paul Highnam

UWE-Willis Newson KTP• Improved understanding of academic research and practice-based

evaluation

• Mapped current state of evaluation knowledge and provision within arts and health practice

• Identified barriers to delivery of credible evaluation within arts and health practice

• Internal - a sustainable, credible and creative evaluation framework that we use across our own projects

• External - an approach to sharing knowledge and expertise with the sector through the delivery of mentored or guided evaluations and CPD

• Evaluation – Research continuum

• High quality credible evaluation will help create fertile ground for research

• Research may provide the kind of ‘hard’ evidence we need to convince policy makers, commissioners and funders to support our work

• The rigour of research can be applied to evaluation to ensure credibility at all levels

• The creativity of practice can be applied to research to generate rich, meaningful data about complex interventions

Academia ↔ Practice: Bridging the Gap

Creative and Credible

Credible

• Timing

• Appropriate Tools

• Avoiding Bias

• Accurate analysis and reporting

• Ethics

Creative• Credible

• Meaningful

• Appropriate

• Analysable

Making it Credible : Timing

Project development

Implementation

Post-project

Making it Credible : Appropriate Tools • Appropriate for what?

– The evaluation question– The stakeholder group or participants– The context – The scale of the project – Available time and budget

• Focus groups will not give you quantitative health outcomes data

• Validated scales are not appropriate for small projects with fewer than 100 participants

• Questionnaires may not be appropriate for people with questionnaire fatigue

Making it credible : Bias • If the project artist asks participants what they gained from the arts

workshop will they get the same answer as an ‘outside’ evaluator

• If the project manager analyses the feedback forms will they include negative as well as positive information in the findings?

• If you tell focus group participants that the evaluation findings will help secure funding for next year’s project will they give you the full story?

• If you select case study participants because they have a really good story to tell about the project, will it give a true picture of what people got out of the project?

• How will you ensure objectivity and safeguard against bias?

Making it Credible : Analysis & Reporting• Be careful what you claim!

• If you have 100 participants for a project and 10 fill in feedback forms, but all 10 say that they enjoyed the project, can you say that “participants enjoyed the project”?

• If you use a validated scale, but out of 200 participants, you only manage to get beginning, middle and end scores for 30 participants, can you claim that these scores are representative of the project as a whole?

• A good project and credible evaluation can easily be undermined by over claiming.

Image from a ‘Savouring & Flow’ workshop run by Light Box as part of The Happiness Project. Photo: BS6 Photography.

Making it Credible : Ethics Respect• Appropriate and sensitive to the needs and vulnerabilities of participants • Participants’ right to decline to take part in an evaluation • Participants’ right to withdraw from the evaluation Professionalism• Everyone involved fully informed and adheres to ethical code of practiceResponsibility• Privacy and confidentiality of participants• Ensure participants’ anonymity • Participants are fully informed before they provide consent to take part• Permission obtained to use images, artwork or comments Integrity• Report evaluations accurately and transparently

Creative and credible

Formal, non-creative

Formal, creative

Basic/light-touch, non-creative

Basic/light-touch, creative

Formal / Structured

Creative / Innovative“Creativity is not an escape from disciplined thinking, it is an escape with disciplined thinking.”

[Jerry Hirshberg]

Creative

• Credible • Meaningful• Appropriate• Analysable

Creative

• Credible • Meaningful• Appropriate• Analysable

Creative

• Credible • Meaningful• Appropriate• Analysable

Monitoring station, The Happiness Project. Image: Light Box.

Creative

• Credible • Meaningful• Appropriate• Analysable

“When I came along I felt quite a broken person. I’ve realised you don’t have to be perfect - broken bits make a person. Coming on the course let me stand back and see the whole person.”

Collage created during a focus group exercise designed to explore the impacts of a project on participants. The quote opposite is from the same participant describing her collage

Creative and Credible

Credible• Timing• Appropriate Tools• Avoiding Bias• Accurate analysis and reporting• Ethics

Creative • Credible• Meaningful• Appropriate• Analysable

• Practice and academia working together and learning from each other

• Robust and credible evaluation practice creates fertile ground for research

• Research providing the ‘hard’ evidence we all need

• The rigour of research can be applied to evaluation to ensure credibility at all levels

• The creativity of practice can be applied to research to generate rich, meaningful data about complex interventions

In summary…

Thank you

karen@willisnewson.co.uk