Government of Western AustraliaDepartment of Culture and the Arts

Public Value MeasurementFramework

Measuring the Quality of the Arts

Q&A

2

Q&A1. Why is the Department of Culture and the Arts (DCA) trying to measure public value?

In 2011, DCA began a journey to find a way of collecting information about arts and cultural events that would help to demonstrate why arts and culture is important to our community. DCA wanted to know what individuals wanted from arts and cultural experiences, the financial impact of such experiences and how arts and culture help to shape and influence our sense of community, belonging and identity as Western Australians. That is, the public value of arts and culture.

Understanding the public value of arts and culture will inform DCA’s policies and programs and allow us to make better decisions on behalf of the State, the people of Western Australia and the sector.

2. What is the Public Value Measurement Framework (PVMF)?

DCA commissioned the development of the PVMF to understand and measure the public value it creates through its investments in arts and culture, and its role as a development agency for the sector.

Working with our project partners, Pracsys Economics (WA) and the Intelligence Agency Ltd (UK), DCA developed a model within a theoretical framework that considers the full value of arts and cultural experiences as part of the Government cultural funding process.

The logic model is based around the intrinsic, instrumental and institutional values created by arts and cultural experiences. The first phase of work in operationalising the PVMF has focussed on understanding and developing ways to measure the intrinsic value of arts and cultural experiences.

3. What is the intrinsic, instrumental and institutional value of the arts?

The development of DCA’s PVMF was informed by the work of cultural theorist John Holden, who identified the three areas of cultural value as being the intrinsic, instrumental and institutional.

Intrinsic value relates to the value of culture to individuals, centred on experiencing arts and culture and includes our feelings of connection to the artwork and our own personal subjective judgements of its quality based on the way it makes us feel.

Instrumental value relates to the contribution that culture makes to specific economic and social outcomes or policy goals, such as creating employment, attracting tourism, increasing educational outcomes, benefiting health and wellbeing, etc. Instrumental value is often measured through a combination of social and economic impact measures and approaches.

3

Institutional value relates to the value that society collectively places on culture, now and for future generations. Holden describes institutional value as “the contribution of culture to producing a democratic and well-functioning society”.

4. How do you measure the intrinsic value of arts and culture?

The intrinsic value of the arts has always been the most challenging to articulate and quantify. DCA worked closely with its funded arts organisations to identify what the quality and reach aspects of their work could and should measure.

Audiences should be able to respond to questions about an event or experience in the same way, regardless of what the event or experience may be or who is answering the questions. This collaboration led to the development of 15 Quality and Reach dimensions.

A significant aspect of the PVMF work is that artists and organisations have been involved in the co-creation of the dimensions. This has resulted in a set of metrics which have been refined through testing, both here in WA and in the UK.

5. What is CultureCounts?

CultureCounts is a digital application and web portal system that collects and measures data from three groups: self (practitioners), peer and the public about a particular arts and cultural experience.

CultureCounts can also collect information in the same system on audience attendance, sponsorship, investment and profit so the quality of work can be analysed in relation to these variables.

CultureCounts collects information about the quality of an arts experience. The system is based around a set of core metrics which are answered by an organisation, their peers and the public to assess the quality of the experience, which can then be translated into the intrinsic value.

6. How does CultureCounts differ from other online survey tools?

CultureCounts uses technology to collect the same real time data from across the sector using a mobile device. Unlike other online surveys, the questions are co-created with artists and benchmarked internationally; they fit an academically sound theoretical model; are able to be aggregated, scaled and debated through a shared mobile experience; data is collected pre and post an event from artists, their peers and the public and can have other important criteria like audience and profitability compared to quality in real time.

The system does not require focus groups or expensive reports, the data can be collected continuously at marginal cost, and can provide instant reports that can be used in isolation on an event basis or aggregated across a whole season or year.

4

CultureCounts will make it quicker, easier and more cost-effective for arts organisations to collect and provide information about their business. CultureCounts can also be used by funders and investors to see how their investment drives quality and the experience of the arts by all stakeholders.

7. What involvement has WA’s arts and cultural sector had in the development of the PVMF and CultureCounts?

The metrics used to measure the quality of arts and cultural experiences were co-produced with WA’s arts sector through comprehensive consultation and testing in 2012-13. The development of the 15 ‘Quality and Reach’ dimensions were co-created with the arts and culture sector, including artists, peer assessors and key funded organisations. This co-creation of metrics is what sets the DCA work apart from any other public value system developed internationally.

DCA’s key funded organisations will continue to play a critical role in the ongoing development of the PVMF and CultureCounts through the DCA Public Value Pilot.

8. What is the DCA Public Value Pilot?

DCA’s work on the PVMF project over the past few years has involved significant research and consultation towards the development of the framework, the logic model and a system to operationalise the PVMF. The DCA Public Value Pilot (the Pilot) will allow the work done to date to be trialed broadly across WA’s arts and cultural sector, a vital stage in the ongoing development of the PVMF and CultureCounts.

The Public Value Pilot will allow DCA to better understand the type of information that can be collected through CultureCounts and how this information can be used to explain the value of culture and the arts. The Pilot will also help DCA and its project partner, Pracsys Economics, to refine and further develop CultureCounts so that it best meets the needs of arts organisations, artists, funders and the public.

DCA’s key funded organisations will work with DCA during the Public Value Pilot to use the CultureCounts system in its beta stage. Each organisation will have one of their program events uploaded to CultureCounts which will enable the three levels of assessment to take place.

9. How will DCA use the information it collects through the DCA Public Value Pilot?

The Public Value Pilot will allow DCA to better understand the type of information that can be collected through CultureCounts and how this information can be used to explain the value of culture and the arts.

Over time, the data collected through CultureCounts will be aggregated to identify trends across the sector – locally, nationally and internationally. Once we understand the type of information we will receive through the system, DCA will be able to use the data to inform our decision-making processes.

5

10. How will the DCA Public Value Pilot be implemented?

The Pilot will be rolled out in stages from late September 2014. Around four to six of DCA’s key funded arts organisations will participate in each phase of the Pilot. DCA will work with the organisations to introduce and demonstrate how to use the system, input events and analyse and use the data.

At this stage of the development of the CultureCounts system, only DCA will have access to the administration web portal, though the intention is that organisations will be able to control this information themselves in the future.

11. How will organisations be selected to participate in the DCA Public Value Pilot?

All of DCA’s key funded arts organisations will be invited to participate in the Pilot. Each stage of the Pilot will include organisations across a range of different art forms with events or activities during the trial period.

Where possible, these organisations will match organisations participating in trials in other countries to enable an international artistic network to be developed that will assist in further refinements to the system.

12. Will CultureCounts be onerous for arts organisations to use?

During the Pilot, DCA will input all information into the CultureCounts system and will work with the organisations to collect and analyse the data.

Eventually, it is intended that arts organisations will be able to use the system independently.

13. How will the data collected through CultureCounts be reported?

DCA will provide all organisations participating in the Pilot with reports relating to the event or activity they used CultureCounts for.

The Pilot will help us to identify the types of reports that can be generated through the CultureCounts system and what information is useful for the organisations and for Government.

During the Pilot, DCA will publish information about the data collected.

6

14. What are the benefits of using CultureCounts?

CultureCounts has many specific benefits including:

• A rich, descriptive, common language for ‘quality’ and ‘reach’

• Consistent use of terms for artistic assessments

• The ability to compare the quality and reach impacts over a period of time

• The ability to conduct artistic assessments before and after the funded events

• Combining intrinsic and instrumental impacts in a single system

• The ability to run public value metrics alongside traditional market research initiatives

• Enables constant and affordable research

Artists and organisations will be able to use the information collected through CultureCounts to better understand how their work is perceived and enjoyed by audiences and their peers so that they can make more informed artistic and business decisions.

Audiences will be able to have their say about what they have seen in a quick and cost effective way. This will give the public ‘a voice’ in what is programmed and produced and encourage a more engaging relationship between the organisations and their patrons. The data also has the potential to be used in other social media applications so that people can identify what’s on, review events and make decisions accordingly.

15. How long will the Pilot go for?

The Pilot will continue until all of DCA’s key funded organisations have trialled CultureCounts, which is anticipated to take approximately 12 months.

Once all organisations have been involved, DCA will review the Pilot, including the type of information collected through CultureCounts and the effectiveness of the system.

16. Can artists or other organisations who do not receive DCA funding also use CultureCounts?

As part of its contractual arrangements, DCA and its funded arts organisations, will be able to use CultureCounts at no cost. Once CultureCounts has been fully developed, other organisations and individuals will be able to pay a user fee to Pracsys Economics to use the system. The user fee will contribute to the costs associated with the ongoing development of CultureCounts.

7

17. Will the data collected through CultureCounts be confidential?

DCA will have access to all of the data collected through CultureCounts, as commissioners of the system. During the Pilot, DCA will provide organisations with their own data and reports. Specific and identifying information collected through CultureCounts will be kept confidential.

When organisations are able to use the system independently, they will only be able to access their own data.

18. What will happen if the peer and public assessments of an event do not correspond to the self-assessment? Will DCA cut funding?

No. Culture Counts provides information that will assist an organisation with its own decision-making. DCA will use the information collected through CultureCounts and the application of the PVMF to inform its policies and programs, including its funding allocation and acquittal processes, to allow us to make better decisions on behalf of the State, the people of Western Australia and the sector. This does not mean that funding will be cut because of a poor review.

19. What interest has there been in the PVMF work DCA has been doing?

Over the past 18months, the Department has been leading discussions with arts agencies across the country who have shown much interest and willingness to adapt the model and system on a national basis.

DCA’s project partners, Pracsys Economics and Intelligence Agency, have been developing CultureCounts in England through a partnership with Arts Council England and the National Endowment for Science Technology and the Arts.

NESTA recently provided a grant of £300,000 through its Big Data R&D fund to develop CultureCounts further including the capacity for Arts Council England’s client arts organisations to trial the system. Organisations in the next trial include the Royal Opera House, Royal Shakespeare Company and Matthew Bourne.

DCA has presented its work on PVMF at a number of key conferences and events such as Remix Sydney, APACA Hobart and, in September, to Remix NYC.

More questions?

Contact the Research Team at the Department of Culture and the Arts on +61 8 6552 7300 or by email at research@dca.wa.gov.au

Download ‘Measuring the Quality of the Arts’ from DCA’s Research Hub.

Image creditsFront cover: The John Butler Trio at Fremantle Arts Centre. Photo by David Craddock.Back cover: Jonathan Buckels and Rhiannon Newton. ‘FUSE’ by STRUT Dance. Photo by Eva Fernandez.