Culture Metrics – developing and testing a digital platform for assessing the quality of the arts

3rd November 2015Abi Gilmore, University of Manchester

John Knell, Culture Counts

#policyweek

11 Welcome and Introduction to Culture Metrics Project. Abigail Gilmore, The University of Manchester & John Knell, Culture Counts

11.20 Using the Culture Metrics system at the Matthew Darbyshire exhibition

12.00 Lunch12.30 Use of Culture Metrics data in organisational practice.

John Knell, Culture Counts & Ronan Brindley, Manchester Art Gallery

12.45 Future research directions: Social Media and Culture Metrics, Kostas Arvanitis & Chiara Zuanni, The University of Manchester

13.00 Roundtable and Discussion. Chaired by Abigail Gilmore, The University of Manchester Alison Clark, Director Combined Arts and North, Arts

Council England Nick Merriman, Director of The Manchester Museum Hasan Bakhshi, Director, Creative Economy in Policy &

Research, NESTA Cimeon Ellerton, Head of Programmes, Audience

Agency

13.45 Final remarks14.00 End

About the projectAmbitions:• To create a system that allows for the cost

effective generation of large-scale data sets on what the cultural sector believes are the key dimensions of ‘quality’

• To explore how, and in what ways, such data will be relevant and useful to cultural organisations in their creative and commercial decision making

NESTA/AHRC/ACE Digital R&D Fund• 12-month project to:– refine the quality metrics– test the method across a wider range of artforms and

settings (60 test events)– develop resources to help organisations run the system

themselves and create a truly automated flexible data platform (we have to be in a position to offer a free trial to funded arts orgs at grant end point)

– carry out academic research to test the value of the approach

Partners• Academic Partner – Abi Gilmore, Kostas Arvanitis,

Franzi Florack, University of Manchester • Tech Partner – Culture Counts • Cultural Partners – 20 of them, including ROH, RSC,

Halle, Contact, Royal Exchange Theatre, Home, Whitworth Art Gallery, Manchester Museum.

2 main categories of metrics

Quality• Quality of product• Quality of experience• Quality & depth of

engagement• Quality of creative process

Organisational Health• Financial metrics• Quality of cultural

leadership• Quality of relationships &

partnerships

Quality dimensions (self, peer, public)Dimension Metric Statement

Rigour ‘It was well thought through and put together’

Distinctiveness ‘It was different from things I’ve experienced before’

Captivation ‘It was absorbing and held my attention’

Relevance ‘It has something to say about the world in which we live’

Meaning ‘It meant something to me personally’

Challenge ‘It was thought provoking’

Enthusiasm ‘I would come to something like this again’

Presentation ‘It was well produced and presented’

Local impact ‘It is important that it’s happening here’

Quality dimensions (self & peer only)

Dimension Metric Statement

Concept ‘It was an interesting idea / programme’

Originality ‘It was ground-breaking’

Risk ‘The artists / curators really challenged themselves with this work’

Excellence (global)

‘It is amongst the best of its type in the world’

Excellence (national)

‘It is amongst the best of its type in the UK’

How do we measure?

• Triangulation of evaluation by self, peers and public

• Online data capture through the Culture Counts system via a web app or via post event surveys (urls posted via email)

• 9 dimensions evaluated by all 3 groups• 5 extra dimensions evaluated by self and peers

Simple interface

What types of insights are generated?

• The triangulation element is vital to the insight and reflection process – allowing cultural organisations to judge the extent to which they are achieving their creative objectives

• Being able to anticipate how audience members and peers will respond to a work is a reflection of the quality / maturity of creative and cultural leadership in an organisation

What types of insights are generated?

• Integrating quality metrics with audience size and profile and financial data allows for a sophisticated assessment of– How successfully organisations balance creative,

commercial and audience objectives– How different kinds of cultural experience create

different kinds of value• Developing prompts to help organisations

understand their evaluation results and use them to reflect on their creative practice

Research and data strategyBring together test event-generated data and analytics with qualitative, collaborative research with technology and cultural partners to answer these questions:

• What is the potential value of Culture Counts to the different stakeholder groups (arts organisations, policy makers, critics, investors, and ‘the public’)?

• How does Culture Counts support and impact upon (DDD) decision-making within these stakeholder groups (for example, in relation to programming, funding and audience motivation to attend further experiences)?

1. What does co-production bring to the appraisal of quality in arts and cultural experiences (and how)? 2. Do bigger data improve D-D-D (and how do we understand the processes which mediate and generate big data)?

Research methods• Thematic literature and evidence reviews• Qualitative research - documented workshops, interviews and

observation and reflective practice with research stakeholders • Critical friends group

Further critical focus on main project innovations

Research findings: co-productionLiterature review findings These endorse Culture Counts as a methodology for performance measurement in terms of “artistic vibrancy” and quality – ownership and leadership by sector, brings together range of internal and external stakeholders. “It’s difficult to do that with everyone so the metrics system I suppose gives you a bit of safety ground if there are peers that are coming who you don’t know so well or have a relationship with. At least it’s managed and structured, it’s moderated by lots of other things, but it’s only really going to work if everybody does it.”

Qualitative research with partners and peers• Cultural partners appreciated the chance to reflect collectively on their evaluation frameworks,

and to add value to the data they already collect • They are ready and willing to integrate with other existing data and value frameworks (e.g.

Audience Finder, their own box office data)However• comparison across organisations (benchmarking) rarely happens organically, and when it does it

tends to be informal - there is interest in comparing findings and experiences across art form but the real value of the data is for internal evaluation

• There are challenges to encouraging peer assessment, but the persuasiveness of digital platform and the standardised ‘metrics’ help to provoke interesting conversations

Research findings: big data

Qualitative research:There is interest in integrating with/using bigger data – e.g. from social media:• Social media outputs can offer chance for ongoing conversations• They are considered the ‘real voice’ of audiences• But at the same time acknowledge as different to survey responses “it’s sometimes

about getting the retweet or favourite from the organisation”• General reticence and lack of awareness about potential of big data “what we need

is ‘just enough data”• There is general caution and wariness about the amount of data collection required

by funders without clear rationale - “we don’t know what they do with it – probably nothing”

Resource and capacity for data management & analysis is low (and ‘big data’ is virtually non-existent) in most arts organisations - Culture Counts is perceived to be an effective way to increase this capacity

Now: your chance to give it a go!

• Alison Whittaker, Culture Counts