5 challenges of evaluating systems change work and how to approach them

More organisations than ever are adopting a ‘systems change’ approach to their work. This means not just trying to solve one problem, but change the wider system that’s causing that problem.

But many organisations are finding is that it’s often hard to evaluate this work. How can you assess your influence on a system?

At NPC, we are fans of systems thinking and a systems change approach. We’ve been discussing the challenge of evaluating systems change work with our clients, at our events, and in our community of practice.

From those conversations, here are five common challenges – and how to approach them:

1. Challenge: It’s difficult to establish cause and effect.
   Approach: Map the system and identify a theory of change.
2. Challenge: It’s hard to measure intangible changes.
   Approach: Ask people in the system.
3. Challenge: You are also part of the system.
   Approach: Listen to diverse voices and evaluate equitably.
4. Challenge: Change takes time and won’t be linear.
   Approach: Recognise the reality of change.
5. Challenge: Systems change requires flexibility.
   Approach: Make evaluation responsive and continuous.

Challenge: It’s difficult to establish cause and effect. Approach: Map the system and identify a theory of change.

When looking at a social system—whether it be the criminal justice system or the child poverty prevention system—it’s clear that there are many interdependent factors influencing change. This makes it incredibly difficult to establish cause and effect, or to attribute outcomes to your interventions.

Evaluations need to consider this complexity and proceed with caution when arguing for your contribution to change.

How can you do this? Tools like NPC’s Complexity Canvas can help you to think through how complexity shows up in the system that you are working with, and how it might affect any intervention.

All this might leave you feeling overwhelmed by the complexity of the system and the challenges in understanding how change happens, but there are ways forward.

Systems mapping can help to identify leverage points at which to target interventions in a system.

This mapping can then form the basis of a systemic theory of change—one that takes into account the context of the wider system, identifies potential feedback loops, and draws out the assumptions and risks underpinning your thinking.

You can then gather evidence on your theory of change to build a plausible story of your contribution and the role of other influencing factors.

Challenge: It’s hard to measure intangible changes. Approach: Ask people in the system.

Successful systems change usually involves intangible changes to the system—such as shifts in relationships, power dynamics, or mental models (how people think about things).

These can often be invisible. They can certainly be challenging to measure. How do you deal with that?

Tools like the Iceberg Model can help you to surface the less visible patterns, structures, and beliefs that underpin more visible change.

Evaluations of systems change work can take this a step further. You may not be able to directly capture the change. But you can capture the perception of the people in the system on what’s changed. You might not be able to measure a changed relationship directly, but you can ask stakeholder how they feel relationships have changed.

For example, in our work with Save the Children UK’s Early Learning Communities and Collaborate we co-developed a ‘maturity model’ to assess the strength of conditions that were seen as needed for long-term systems change.

Diagnostic tools like this can help to drive continuous learning and improvement, as well as tell a compelling story about progress.

Challenge: You are also part of the system. Approach: Listen to diverse voices and evaluate equitably.

One of the hardest aspects of systems change is recognising that we tend to unwittingly contribute to the problems we are trying to solve. For instance, the power dynamics within a coalition for change will often mirror the power dynamics in the wider system that they are trying to alter.

Evaluating change must therefore include looking honestly at our own role in the system, including hearing from a diverse range of voices to balance our pre-conceived image of ourselves.

Bellwether interviews are a powerful tool that can help. (Something we recommended in our report Shifting the Dial.) In a bellwether interview, the interviewee is aware that a range of issues will be discussed, but does not know which organisation the research is for. It’s a good way to reflect on how an issue is understood, and how you and your work is perceived.

For example, in our evaluation of Friends Provident Foundation’s Building Resilient Economies programme we used bellwether interviews to understand the wider landscape and uncover diverse perspectives on the Foundation’s contribution to change.

Our approach to evaluation also needs to match the change that we want to see in the wider system. If evaluating a programme that aims to create a more equitable world, then we need to embed equitable evaluation principles into our approach. This includes considering the systemic inequalities affecting both the evaluation and the programme.

Challenge: Change takes time and won’t be linear. Approach: Recognise the reality of change.

Anyone involved in shifting systems knows that change takes a long time. It will not be linear. And it can be affected by unintended consequences. There is often a long, hard slog of building coalitions for change, shifting power dynamics, or transforming attitudes about an issue over many years.

There can be sudden breakthroughs like a welcome policy change. But there can also be backwards steps, such as a backlash against progress or a ping-back to previous systems.

This presents challenges for conducting evaluations and communicating findings to stakeholders.

In discussions at NPC Ignites and with our community of practice, we heard time and again that people struggle to explain the slow and uneven pace of change to external funders and decision-makers within their own organisation. Programmes can be closed down early or struggle to get funding because of a perceived lack of progress.

There are no easy answers to addressing this problem: it requires all of us involved in funding and evaluating systems change to recognise the reality of how change happens. We at NPC will continue to do our bit to bust myths about systems change and improve evaluation practice.

Challenge: Systems change requires flexibility. Approach: Make evaluation responsive and continuous.

Systems change requires us to learn and adapt as we go. We have to respond to changes in the system and our understanding of our role within it.

Evaluation needs to support these aims by being flexible, responsive and continuous. For example, developmental evaluation approaches can support real-time feedback about emerging challenges to inform decision-making.

Whatever evaluation approach you take, ongoing learning needs sufficient resources. Do it well, and it can enhance your effectiveness.

Finally – mindset is key

I hope these approaches will help you overcome and deal with the challenges of evaluating systems change work.

But I would also say that underpinning all of this advice is your mindset. A systemic mindset to evaluation involves:

• embracing complexity
• capturing more intangible changes
• involving a diverse range of voices
• recognising that change takes time
• continually learning and adapting

If you’re trying to grapple with systems change evaluation and could use some help, get in touch for a free consultation:

Photo by Joey Genovese on Unsplash