This document summarizes a meeting about developing the Culture Metrics project, which aims to create a digital platform for assessing the quality of arts and culture. The project tests metrics across various artforms and organizations to measure quality, experience, engagement, and organizational health. Research will explore how the data can help organizations make creative and commercial decisions and will test whether larger data sources like social media can improve insights. Initial findings suggest co-production of metrics brings value but challenges of peer assessment remain, while interest exists in integrating bigger data but capacity is currently low.

1. Culture Metrics – developing
and testing a digital platform for
assessing the quality of the arts
3rd November 2015
Abi Gilmore, University of Manchester
John Knell, Culture Counts
#policyweek

2. 11 Welcome and Introduction to Culture Metrics Project.
Abigail Gilmore, The University of Manchester & John
Knell, Culture Counts
11.20 Using the Culture Metrics system at the Matthew
Darbyshire exhibition
12.00 Lunch
12.30 Use of Culture Metrics data in organisational practice.
John Knell, Culture Counts & Ronan Brindley, Manchester
Art Gallery
12.45 Future research directions: Social Media and Culture
Metrics,
Kostas Arvanitis & Chiara Zuanni, The University of
Manchester
13.00 Roundtable and Discussion.
Chaired by Abigail Gilmore, The University of Manchester
 Alison Clark, Director Combined Arts and North, Arts
Council England
 Nick Merriman, Director of The Manchester Museum
 Hasan Bakhshi, Director, Creative Economy in Policy &
Research, NESTA
 Cimeon Ellerton, Head of Programmes, Audience
Agency
13.45 Final remarks
14.00 End

3. About the project
Ambitions:
• To create a system that allows for the cost
effective generation of large-scale data sets on
what the cultural sector believes are the key
dimensions of ‘quality’
• To explore how, and in what ways, such data
will be relevant and useful to cultural
organisations in their creative and commercial
decision making

4. NESTA/AHRC/ACE Digital R&D Fund
• 12-month project to:
– refine the quality metrics
– test the method across a wider range of artforms and
settings (60 test events)
– develop resources to help organisations run the system
themselves and create a truly automated flexible data
platform (we have to be in a position to offer a free trial to
funded arts orgs at grant end point)
– carry out academic research to test the value of the
approach

5. Partners
• Academic Partner – Abi Gilmore, Kostas Arvanitis, Franzi
Florack, University of Manchester
• Tech Partner – Culture Counts
• Cultural Partners – 20 of them, including ROH, RSC, Halle,
Contact, Royal Exchange Theatre, Home, Whitworth Art
Gallery, Manchester Museum.

6. 2 main categories of metrics
Quality
• Quality of product
• Quality of experience
• Quality & depth of
engagement
• Quality of creative process
Organisational Health
• Financial metrics
• Quality of cultural
leadership
• Quality of relationships &
partnerships

7. Quality dimensions (self, peer, public)
Dimension Metric Statement
Rigour ‘It was well thought through and put together’
Distinctiveness ‘It was different from things I’ve experienced before’
Captivation ‘It was absorbing and held my attention’
Relevance ‘It has something to say about the world in which we live’
Meaning ‘It meant something to me personally’
Challenge ‘It was thought provoking’
Enthusiasm ‘I would come to something like this again’
Presentation ‘It was well produced and presented’
Local impact ‘It is important that it’s happening here’

8. Quality dimensions (self & peer only)
Dimension Metric Statement
Concept ‘It was an interesting idea / programme’
Originality ‘It was ground-breaking’
Risk ‘The artists / curators really challenged themselves with this work’
Excellence
(global)
‘It is amongst the best of its type in the world’
Excellence
(national)
‘It is amongst the best of its type in the UK’

9. How do we measure?
• Triangulation of evaluation by self, peers and
public
• Online data capture through the Culture
Counts system via a web app or via post event
surveys (urls posted via email)
• 9 dimensions evaluated by all 3 groups
• 5 extra dimensions evaluated by self and
peers

10. Simple
interface

11. What types of insights are generated?
• The triangulation element is vital to the insight and
reflection process – allowing cultural organisations to
judge the extent to which they are achieving their
creative objectives
• Being able to anticipate how audience members and
peers will respond to a work is a reflection of the
quality / maturity of creative and cultural leadership
in an organisation

12. What types of insights are generated?
• Integrating quality metrics with audience size
and profile and financial data allows for a
sophisticated assessment of
– How successfully organisations balance creative,
commercial and audience objectives
– How different kinds of cultural experience create
different kinds of value
• Developing prompts to help organisations
understand their evaluation results and use
them to reflect on their creative practice

13. Research and data strategy
Bring together test event-generated data and
analytics with qualitative, collaborative
research with technology and cultural partners
to answer these questions:
• What is the potential value of Culture Counts to the
different stakeholder groups (arts organisations, policy
makers, critics, investors, and ‘the public’)?
• How does Culture Counts support and impact upon (DDD)
decision-making within these stakeholder groups (for
example, in relation to programming, funding and audience
motivation to attend further experiences)?

14. 1. What does co-production bring to the appraisal of
quality in arts and cultural experiences (and how)?
2. Do bigger data improve D-D-D (and how do we
understand the processes which mediate and generate
big data)?
Research methods
• Thematic literature and evidence reviews
• Qualitative research - documented workshops, interviews and
observation and reflective practice with research stakeholders
• Critical friends group
Further critical focus on
main project innovations

15. Research findings: co-production
Literature review findings
These endorse Culture Counts as a methodology for performance measurement in terms of
“artistic vibrancy” and quality – ownership and leadership by sector, brings together range of
internal and external stakeholders.
“It’s difficult to do that with everyone so the metrics system I suppose gives you a bit of
safety ground if there are peers that are coming who you don’t know so well or have a
relationship with. At least it’s managed and structured, it’s moderated by lots of other
things, but it’s only really going to work if everybody does it.”
Qualitative research with partners and peers
• Cultural partners appreciated the chance to reflect collectively on their evaluation
frameworks, and to add value to the data they already collect
• They are ready and willing to integrate with other existing data and value frameworks
(e.g. Audience Finder, their own box office data)
However
• comparison across organisations (benchmarking) rarely happens organically, and when it
does it tends to be informal - there is interest in comparing findings and experiences
across art form but the real value of the data is for internal evaluation
• There are challenges to encouraging peer assessment, but the persuasiveness of digital
platform and the standardised ‘metrics’ help to provoke interesting conversations

16. Research findings: big data
Qualitative research:
There is interest in integrating with/using bigger data – e.g. from social
media:
• Social media outputs can offer chance for ongoing conversations
• They are considered the ‘real voice’ of audiences
• But at the same time acknowledge as different to survey responses “it’s
sometimes about getting the retweet or favourite from the organisation”
• General reticence and lack of awareness about potential of big data “what
we need is ‘just enough data”
• There is general caution and wariness about the amount of data collection
required by funders without clear rationale - “we don’t know what they
do with it – probably nothing”
Resource and capacity for data management & analysis is low (and ‘big data’
is virtually non-existent) in most arts organisations - Culture Counts is
perceived to be an effective way to increase this capacity

17. Now: your chance to give it a go!
• Alison Whittaker, Culture Counts