Monitoring and Evaluation When Facing Complexity

This blog post was written by Marcus Jenal of Mesopartner. Jenal presented at the MPEP Seminar: Monitoring and Evaluation When Facing Complexity on April 25, 2013.

Development initiatives are faced with various types of problems. Very roughly, they can be categorized as simple problems, like the installation of a solar panel in a hut in rural Africa; complicated problems, like the building of a road, a school, or a hospital; and complex problems, like the establishment of an effective education, health, or market system. Traditionally, all these problems have been tackled with the same set of approaches, relying on pre-project assessments, pre-defined outputs and outcomes, including logical chains on how to achieve them with a set of activities. Added to this was a monitoring system that checks on the implementation of the activities. Evaluations focused on the achievement of the outputs and only recently gave more attention to impacts.

Looking at these different sets of problems, projects have been more effective in tackling simple or complicated problems and have faced much more difficulties when facing complexity. Thus, we need new approaches for complex problems, including a different monitoring and evaluation framework that is specifically tailored to the requirements of these situations. A growing body of literature has been tackling how to approach complex problems in international development^[1].

But what does complexity mean? The field of complex adaptive systems research has made great progress in recent decades and allows us to tap into a set of emerging insights and approaches. Complex adaptive systems are constituted by a high number of heterogeneous interacting and highly interconnected actors. The actors in the system are constantly adapting to new opportunities and to new challenges they are confronted with, and most importantly to each other, as they observe others’ behavior.

Complex adaptive systems exhibit a number of specific characteristics. One of the most important phenomena in complex systems is what we call ‘emergence’. This is a technical term that basically states that the whole is more than the sum of its parts. Certain aspects of these systems only emerges through the interaction of the individual actors, but cannot be observed at the level of these actors. Examples for emerging aspects are communities, which only exist when people that live in the same area interact with each other on a regular basis and develop a set of ‘rules of interaction’ that gives them the advantages that you can find in community life. Another example is the market price, which you cannot observe when looking at individuals, but only emerges through the interactions of market actors.

Another important trait of complex system is non-linearity, meaning that the scale of ‘cause’ is apparently unrelated to the scale of ‘effect’. The introduction of the possibility to transfer money from one mobile phone to another, for example, is a relatively small cause looking at the effect, allowing millions of poor to access financial services.

Complex systems are strongly embedded in their history. To come back to the community example: the rules around which live in a community is organized, from how people interact with each other to how support is given to people in need, has co-evolved over time with the behavior of the individual community members. If we want to intervene in such a system, we have to be aware that this history matters.

All this has the effect that complex systems are inherently hard to predict – both in terms of what intervention works to tackle a specific problem and what effect an intervention will have.

The Systemic M&E Initiative^[2] of the SEEP Network has set out to bring together practitioners with experts in the field of complex adaptive systems to discuss how monitoring and evaluation frameworks can be made more effective when facing complexity. It capitalized on discussions between practitioners on SEEP’s Market Facilitation Initiative’s platform that go back to 2010. With funding of USAID/fhi360, we organized a three-day e-collaboration and a webinar, the opening plenary of the 2012 SEEP Annual Conference^[3], and recorded three podcasts.

A Synthesis Paper^[4] brought together the various discussion threads. It starts out with three major issues practitioners are confronted with in current M&E frameworks:

1. Excessive focus on our direct effects on the poor, ignoring our effects on the wider system.
2. Excessive focus on extraction of information for accountability to the donors, not delivering much useful information for project management to take strategic decisions on the project’s directions.
3. Sustainability understood as longevity of our legacy, checking back in a number of years to assess whether our direct effects on the poor persisted.

Based on these issues and with strong references to experiences of practitioners in the field, we came up with seven principles to guide the development of monitoring and evaluation frameworks that are effectively supporting projects when facing complexity. We believe that with these principles as a starting point, monitoring and evaluation can become more aware of how complex real-world situations often are and function better in these circumstances.

At the same time, we are aware that the simple adoption of a number of principles is not enough. We need a change in paradigm from an emphasis on predictability and control to bottom-up, decentralized and adaptive solutions. The biggest obstacles to reach that are political, not technical, as development policy is still entrenched in strong orthodoxy of controlling the exact results of our interventions.

Footnotes

[1] find a collection here: http://systemic-insight.com/resources/
[2] http://www.seepnetwork.org/
[3] http://www.seepnetwork.org/measuring-impact-in-market-and-financial-systems-pages-20167.php
[4] http://www.seepnetwork.org/monitoring-and-measuring-change-in-market-systems---rethinking-the-current-paradigm-resources-937.php