Evaluating large-scale cultural events; what are we learning?

Over the past 4 years both myself and Prof Jonothan Neelands, (Warwick Business School) been involved in designing and evaluating Coventry UK City of Culture (UKCoC21), and Birmingham 2022 Festival as part of the Birmingham 2022 Commonwealth Games.

With the evaluation findings from Birmingham announced this week (17 Jan) and Coventry due to publish their findings in May this year, we’ve been reflecting on what we’ve learned, as cultural researchers and evaluators and what we think this means for the sector going forwards.

Why Evaluate?

Firstly, why are we so passionate about evaluation, and what are the benefits it can bring?

Well, apart from the obvious argument of ‘ensuring value for (often taxpayers’) money, the main benefits we can identify are:

• Quantifying the various impacts of making a large single investment into culture over a limited timescale and in a specific place
• What has this investment enabled that would not have happened without it?
• What if anything, has this intervention achieved that sustained investment in ongoing cultural organisations has not / cannot?
• Capturing and reporting on how different people have experienced the event differently; what do their stories tell us

Our learnings - the 5 key ingredients of what works

Having been directly involved in collecting and collating the evaluation data for both these mega events, we’ve learned that there are some keys to successful evaluation:

1. Having a sound Theory of Change Model from the beginning
Both of these events had a well-developed and clear Theory of Change model that was handed to the evaluation team on day one and guided programming and investment decisions. Of course, it needed some nuancing as things emerged - as in both cases it was written before any commissions were secured, or programming boots hit the ground. Despite the many adaptations to programme the pandemic forced on Coventry, their original outcomes and impacts for change, focussed and led the changes in cultural responses and programming forced by the pandemic’s disruption.

2. Developing a robust Evaluation Framework
An Evaluation Framework takes each of the outcomes in the Theory of Change and identifies what data it needs to find/collect in order to evidence each outcome. Data will be quantitative or qualitative, and need gathering across multiple different platforms or touchpoints.

A good framework should also identify where ‘baseline’ metrics will be found. For example, if the outcome is to increase or improve something from x to y…. You need to know where you are starting from. What data exists to tell you this, and if it doesn’t how might you find out?

It should also identify where numbers alone are not going to provide suitable evidence. For both projects, case study and other qualitative work has been crucial to understanding the depth and scale of delivery and impact.

3. Designing innovative data collection tools and practices
The way you choose to run the event will have a significant impact on your ability to collect numbers and data about attenders. Birmingham 2022 Festival decided from the outset that everything in the programme would be free, and unticketed. So we knew from day one that we would have to ‘count people’ and gather feedback data ‘on the ground’ rather than rely on sales data reports and emailing surveys after the event.

With over 160 projects taking place over just 6 months, we had to develop a ‘distributed’ model of evaluation, in which every project needed to engage with the evaluation process, collect numbers on their attenders or participants, and gather feedback. The effort required to achieve this across the Festival was phenomenal, and it’s incredible that every single project managed to do it.

We, in turn, provided standard data collection tools, but worked with individual projects where necessary, to adapt these tools or collection methods for certain activities, or audiences. For example, for a project involving adults with learning disabilities it was essential that their voices and responses were included in the overall evaluation, but clear that our methods of collecting those responses needed to be handled differently.

4. Having a central point of truth
For both Coventry 2021 and Birmingham 2022 an Evaluation Co-ordinator was an integral part of the delivery team. Their role is to advocate for evaluation within the organisation, ensuring that due consideration is given at planning and delivery stages to gathering the required evidence; to ensure that data is collected and delivered on time; to quality assure the data as it comes in, checking and validating the figures supplied by each project; and managing the subcontractors / suppliers such as ourselves to make sure we’re all on track.

As part of this approach, all the data collected from projects was fed into a single, central point. This became the ‘single point of truth’ that was used to create all the evaluation statistics so that they could be sliced and diced in various ways, but always add up. Without this, I’m sure we would all have gone slightly mad.

By ensuring that we had a single data repository, with only data that had been checked going into it, we could then supply that data back out, to other evaluators working on specific Festival projects, and create a Festival-wide dashboard that could be updated monthly throughout the project. It is now being used to create bespoke Funder reporting, as well as the final evaluation reports for the Festival.

5. Knitting together the What and the Why
As those of us involved in data analysis are only too aware, numbers can tell many stories, and so the challenge in undertaking evaluation of this kind is around working out when the story is best told in numbers, and when it needs a different approach.

In Coventry and Birmingham Qualitative Case studies formed a crucial part of the evaluation: observation of projects, in development, delivery and close-down phases; interviewing artists and producers throughout the process, and testing with audiences or participants, the extent to which their creative intent was delivered.

We believe that being able to tell the story of how investment in culture and creativity can radically change places, people and the future, is essential to our growth and meaning as a sector going forwards. But we all need to play our part in unearthing, gathering, illuminating and sharing the evidence we have - to make that story the richest and most compelling we know it needs to be.