The Puzzle of Assessing System Change: Three Lessons Learned as Evaluators – Part 3
Image
In a three-part blog series, we share lessons on the process of evaluating the Cambodia Feed the Future Harvest II program focused on what systemic changes the program made progress on in the horticulture sector. (Full report here). The lessons are:
- Evaluate how the ‘puzzle pieces’ of the system are or are not fitting together (Blog 1).
- Recognize that a big contribution to a small system change may be more important than a small contribution to a big system change (Blog 2).
- Start from the program strategy and system context to determine which dimensions of system change to evaluate.
Lesson 3 is the focus of this third and final part of the blog series.
Lesson 3: Start from the program strategy and system context to determine which dimensions of system change to evaluate.
We wanted to be thorough, so we reviewed key literature on dimensions of system change, such as USAID Market Systems Resilience: A Framework for Measurement. We chose 22 dimensions that we thought were relevant, such as buyer-supplier coordination, value chain efficiency, information flows, networks, gender equity, and environmental stewardship. We aimed to evaluate each dimension in terms of scale, sustainability, and depth of change. It was both way too many (22!!) and not specific enough!
Image
We realized during the field research that some dimensions were much more important than others in the context of the crop systems and the uniqueness of each crop with respect to the dimension of changes. In this context, there was substantial overlap among some of the dimensions we had chosen, while others were too general. We had visualized what the puzzle pieces looked like before we had enough information.
In hindsight, it makes more sense to start with the program’s strategy to understand the vision for a functioning market system in each crop to choose which dimensions of system change to evaluate. Then, it helps to refine that list based on early field research. In other words, each system puzzle is unique, so use literature as a menu, not a mandate.
|
Example: In our evaluation plan, we separated agricultural policies and public-private coordination. However, interviews showed that we had to consider these together as public-private coordination was a part of the agricultural policy development process that the program worked on. At the same time, we had to consider more than just policies. Other levels of the business enabling environment (BEE) were arguably more important, such as the regulations and procedures for exporting. (For more on levels of BEE, see Enhancing the Use of Evidence and Results Measurement in Business Environment Reform Programming). |
Reflections:
We defined the dimensions of system change to evaluate after reviewing the program literature and workshops with program staff but before fieldwork started. Looking back, it would have been more effective to delay the definition of dimensions until some interviews with market actors and key informants had been done.
It also would have been more practical to limit the number of dimensions to roughly 6–10, depending on how broadly they were defined. In the end, our analysis focused on 13 dimensions, combining some from our original list and making others more specific. Limiting the number of dimensions can help to keep the evaluation focused on robustly investigating the most important aspects of system change according to the program strategy and the system context.
We also found it was important to explore and pinpoint the most important levels of change for each dimension. In the BEE example, it was helpful to pinpoint missing puzzle pieces at the level of regulations and procedures, not only at the level of policies. This hunt for the ‘key puzzle pieces’ in each relevant dimension further hones the explanation of how, why, and to what extent system changes are happening.
Effectively assessing system changes, both during and after a program, can help donors and practitioners to better design and adaptively manage programs so that they contribute to transformational change with widespread and positive impacts. The assessment itself is fascinating—but tricky! Let’s keep the conversation going on the best ways to practically do it.
Authors:
Alexandra Miehlbradt, Makararavy Ty, and Lara Goldmark
Graphics: Sajda Karmacharya