The Puzzle of Assessing System Change: Three Lessons Learned as Evaluators—Part 2
Image
In this three-part blog series, we share lessons on the process of evaluating the Cambodia Feed the Future Harvest II program focused on what systemic changes the program made progress on in the horticulture sector. (Full report here). The lessons are:
- Evaluate how the ‘puzzle pieces’ of the system are or are not fitting together.
- Recognize that a big contribution to a small system change may be more important than a small contribution to a big system change.
- Start from the program strategy and system context to determine which dimensions of system change to evaluate.
In this blog, we discuss lesson 2 learned in the evaluation process.
Lesson 2: Recognize that a big contribution to a small system change may be more important than a small contribution to a big system change.
Image
It’s tempting to focus primarily on the apparent magnitude of changes in a system. But a program’s contribution to observed changes is just as important in generating useful insights on a program’s strategy.
When many puzzle pieces are in place, impressive gains in system performance may quickly follow a program’s contribution to improving one of those pieces. But taking into account the trajectory of change, we found it useful to consider whether a program’s contribution might have addressed a problem that would have been solved by market actors relatively quickly anyway.
In contrast, when many puzzle pieces are missing, only modest gains in system performance may follow a program’s contribution to getting a key piece in place. Nevertheless, getting that piece in place is essential to future system performance. In addition, the program’s contribution may have been significant, considering there were few other pieces to support its introduction.
This doesn’t mean programs should pursue unrealistic expectations, but it does mean that we, as a field, should recognize early contributions to long-term system change.
|
Example: Harvest II worked in three tree crop subsectors—cashew, mango, and longan. These crops were mostly for export and faced a challenging environment. Many puzzle pieces necessary to directly serve foreign buyers, such as lab services, key technical skills, supportive government processes, and affordable energy, were missing. The challenging environment was exacerbated by COVID-19 border closures and adverse weather associated with climate change. Harvest II implemented a range of interventions to build export readiness and enable firms to serve premium local markets. One of these was working with firms and their supplying farmers to obtain certifications necessary for these markets, such as organic certification, Cambodia Good Agricultural Practices (CamGAP), and Hazards Analysis and Critical Control Points (HACCP). While there has not yet been a substantial uptick in direct exports, Harvest II’s efforts to increase certifications and encourage expansion of the standards market are helping firms start to serve premium local markets and laying a critical foundation for future export growth. |
Reflections:
We evaluated the program’s contribution to putting in place or improving pieces of the system puzzle, irrespective of the magnitude of performance change in the system. Some changes that were modest in scope were nevertheless fundamental to the future performance of the sector. By making a big contribution to these, Harvest II has put a piece in place that can be expected to contribute to significant system change in the future.
To evaluate the contribution of Harvest II to specific system changes, we triangulated findings from the ‘helicopter’ perspective that looked at key trends and why they occurred with a theory-based, ‘intervention’ perspective that followed the expected causal chain from interventions through to system changes.
We found that this approach yielded useful insights into how and to what extent the program contributed to observed changes relative to other factors like government activities, international markets, infrastructure development, and other programs.
This approach is similar to the one outlined in the recent Practitioners’ Guidance to Assessing System Change but places more emphasis on triangulating information on causes for observed changes rather than substantiating contribution claims.
Like the authors of the Practitioners Guidance, we found operationalizing this approach benefited from multiple perspectives.
We started the evaluation with several two-hour workshops with the Harvest II program staff to understand in detail the team’s vision for a functioning market system in each crop and how the program’s interaction with market system stakeholders evolved over the life of the program.
With that foundation, we conducted interviews with firms that the program had interacted with and those they had not, farmers directly and indirectly impacted, associations, and relevant government agencies. We used interviews with key informants to get another perspective and also to fill gaps in our understanding. An analysis of these varying perspectives informed our evaluation of both the extent of changes and the program’s contribution to them.
Authors:
Alexandra Miehlbradt, Makararavy Ty, and Lara Goldmark