Testing Tools for Assessing Systemic Change: A Synthesis and Tool Trial Reports
The Synthesis: This report summarizes the results of a multi-year effort to understand the potential of a set of tools – Standard Measurement Tools, Outcome Harvesting, SenseMaker, and Social Network Analysis– to measure systemic change. Building from tool trials in four countries, plus experience shared by nine other projects or organizations, this synthesis provides practical guidance for practitioners wishing to select and apply tools for capturing systemic change that are relevant in their own contexts. It provides comparative assessments of each tool's utility in (a) capturing indications of systemic change; (b) application along a project cycle; (c) utility for decision-making and reporting; and (d) ease of application. Led by the USAID-funded Leveraging Economic Opportunities project, this effort collaborated closely with the DfID and SDC-funded BEAM Exchange.
Trial of Social Network Analysis: In late 2015, a researcher from MarketShare Associates with the LEO activity conducted a network analysis of traders within the vegetable market system in Sierra Leone, to evaluate network analysis as a technique for describing market system dynamics, evaluating systemic change over time, and identifying potential leverage points for intervention by market systems programs. The DfID-funded Sierra Leone Opportunities for Business Action (SOBA) program, which hosted the analysis and provided data collection and analytical support, intended to use that information for partner selection, performance monitoring, indicator design, and impact measurements over time.
Trial of Outcome Harvesting: In May and June 2016, a researcher with the LEO activity, in collaboration with the BEAM Exchange, used OH to identify and analyze unintended outcomes arising at least partially from the Alliances Lesser Caucasus Programme’s (ALCP’s) successful efforts to facilitate an improvement in the dairy industry in Kvemo Kartli, Georgia. In so doing, we also evaluated the OH approach as a technique for understanding systemic change related to a market systems program.
Trial of Standard Measurement Tools: The synthesis paper explores a variety of standard indicators and measurement tools; as part of LEO's tool trials in the field for this effort, in 2016, LEO experimented with the applicability of a set of tools to measure various aspects of system health. Over the course of three weeks, a team of Agricultural Value Chain (AVC) project staff and external consultants piloted and adjusted eight tools designed to capture aspects of trade relationships and business management that might give insight into norms and networks in Bangladesh. This brief summarizes these 8 simple, survey-based tools.
The SenseMaker: The SenseMaker® is a research approach that gathers narratives (i.e. qualitative data) as well as the self-signified meaning of these narratives (i.e. quantitative data) to understand existing perspectives, beliefs, decisions and norms – or to understand the way these are changing in response to interventions and other environmental factors.