If you want to learn what the spirit of the people is by means of arithmetic, it goes without saying that that’s extremely difficult.
Leo Tolstoy, “Anna Karenina”
When CP Snow published The Two Cultures and the Scientific Revolution in 1959 it seems unlikely he had the voluntary and community sector in mind. Yet more than 60 years later there remains in the sector a dominant culture wholly antithetical to measurement and the scientific method. That is why the Centre for Thriving Places is to be congratulated for persevering with its Thriving Places Index, now in its third iteration. In doing so, it marks itself out as one of a small but growing insurgency of sector leaders who recognise that, while the rich tapestry of life cannot be boiled down into a few simple numbers, there is much to be gained through the intelligent analysis of social metrics. Moreover, by seeking to balance the strengths and challenges of places, the Centre for Thriving Places seeks to nudge the sector away from a deficit mentality toward one that prioritises more positive aspects of community life.
Of course, the use of mathematics to capture dimensions of social phenomena is not new. The “granddaddy” of modern indices, the Index of Multiple Deprivation (IMD), dates back to 2007 although it in turn was built on the pioneering work of Peter Townsend and others at least two decades earlier. The new generation of numerate sector leaders includes heroes such as 360Giving, DataOrchard, MyCake and the amazing David Kane. And alongside them is a growing band of organisations like the Centre for Thriving Places developing new geospatial indices. There is the Legatum Institute’s Prosperity Index, the Young Foundation’s Community Wellbeing Index and Local Trust’s Community Needs Index.
What marks out the Thriving Places Index is its refusal to boil everything down to a single number. Instead, each place is scored against three headline elements: local conditions, sustainability and equality. Or, in their words:
- Is it a fair and equal place to live?
- Is it sustainable enough so that future generations can flourish?
- Are the conditions present for everyone to do well?
This insistence on the multi-dimensionality of local places is important because it addresses a problem even more intractable than the numerical wrangling needed to produce indices in the first place. That problem is… lazy managers. I’m sorry, I wish there were some way I could have dressed that up to sound less rude but – really – I’ve lost count of the number of people I’ve worked with over the years who see indices as a simple way to avoid complicated decisions.
My first professional encounter with this lazy way of thinking was through an index that wasn’t really an index at all. It was a league table developed back in the 1990s to identify the worst prisons in the country. These were the prisons that the government would go on to privatise. The thing is, prisons are complex, multi-faceted institutions – different sizes, different population mixes, different geographies – so how do you decide which ones are the ‘worst’?
Well in those days the Prison Service had eight Key Performance Indicators to track prison performance: KPI1 was the number of escapes (obviously, and with a target of zero!), KPI4 was the number of hours of prisoners’ purposeful activity per week (with a target of at least 24) and so on. What the consultants did was build statistical models for each of these KPIs. They then compared the actual KPI for each prison with the KPI predicted by the model to see if it was performing better or worse than expected. Finally, they added up these deviations to get an overall measure of relative performance. Prison Service managers could then run their finger down the league table, find the prisons at the bottom and start the privatisation process.
But hold on a second. Go back to KPI1, the number of escapes. What statistical model accurately predicts prison escapes? What independent factors go into that model? It can’t be something like ‘the number of holes in the fence’, that would be absurd. So, how did the consultants construct a model that reliably predicts the number of escapes? The short answer is: they didn’t. A simple review of the diagnostics for KPI1 revealed it was a model with no explanatory power at all (it’s R2 was zero). The consultants weren’t comparing the actual KPI with the predicted KPI, they were comparing the actual KPI with a random number – a wholly pointless exercise.
As a junior analyst at the Home Office, I raised concerns about this with Prison Service management. Surely they wouldn’t want to make decisions that could have profound implications for staff and prisoners alike if they were doing so on the basis of faulty information? So naïve. So, so naïve! Of course they were happy to make decisions on this basis. Ministers had instructed them to find the worst prisons and the league table had given them the answer. Why make life more complicated? All those management textbooks that tell you that managers get paid more as the complexity of their decision making increases? Bunkum. At least it seemed that way in many parts of the public sector in the 1990s.
Fast forward to the present day and it is a relief to see the government’s response to the covid-19 pandemic being “guided by the science”. I am sure I am not the only one reassured to watch the Prime Minister on television flanked by Sir Patrick Vallance and Professor Chris Whitty. There are valuable lessons here for the third sector about bringing in expertise at the right level and listening to it. We seem to love the language of ‘systems thinking’ and ‘theories of change’ without wanting to bother ourselves with the messy detail of systematic data collection and hypothesis testing, let alone the intelligent application of data to decision making. Foundations, for example, too often still point to how much of their funding goes to the ‘most deprived IMD decile’ as if that were an end in itself and without pausing to reflect that – no matter how much social progress they achieve – there will always be a ‘most deprived IMD decile’. The IMD is a superb measure that captures an important aspect of community life – but it is only onemeasure.
That is why the Centre for Thriving Places’s decision to present a trinity of measures is inspired. By forcing decision makers to recognise there might be trade-offs to be made, the Thriving Places Index requires them to adopt a more mature approach. Perhaps most importantly, the index reminds us all that numbers alone will never give us the answers to complex social phenomena but they might just help us to ask more intelligent questions.
This article first appeared on the Centre for Thriving Places blog on 25 March 2020.