Beyond Downloads, Views, and 'Likes,' How Do You Know Your Research Is Having an Impact?

This blog is written by Laura Kim and Michelle LeMeur of the Canopy Lab for the Feed the Future Market Systems and Partnership (MSP) Activity.

How does one know if their studies have had any influence in the real world? With the COVID-19 pandemic in the rearview mirror (for many), we set out to answer this question following the dissemination of our 2021 and 2022 studies on the impact and implications of the pandemic on the global development workforce.

In fairness, we were prepared for disappointment. After all, even an institution like the World Bank admitted that less than one-third of their reports were ever downloaded. Moreover, we recognized that our two studies were rarely definitive or influential enough to warrant organizations to change their practices and policies, but we were still hoping that their timeliness and hyperfocus on our sector, in conjunction with other studies on COVID-19 and the workplace, would at least stir the pot.

Back in 2021 and 2022, we conducted two studies in partnership with Feed the Future Market Systems and Partnerships Activity (MSP)—a rapid assessment of the impact of COVID-19 on market systems development professionals and another (larger) piece that explored international development professionals' shifting perspectives on their careers in light of the pandemic and their implications for the future diversity of the sector. We dedicated much of our resources to capturing data (which included surveys with over 1,000 respondents and interviews with dozens of individuals) and producing nuanced analyses. We were, and still are, proud of our work and felt that our calls to action would resonate with the international development community. 

Image

To this end, we worked toward making our findings accessible. We conducted three webinars, disseminated the study through LinkedIn posts, presented findings at the annual 2022 Society for International Development (SID)-US conference, and co-wrote a blog with other panelists on the future of global development. From our estimates, we likely reached a global audience of almost 10,000.

There were early signals that our findings were gaining traction. We received positive feedback from colleagues who indicated that the findings validated their experiences and perspectives. We heard from several individuals who noted that the study findings, along with other research, were informing their conversations around crafting their organizations' future-of-work policies, particularly around implications for diversity, equity, and inclusion (DEI):

"One [good thing] about having data was on how different groups felt about the return to work. There was always a sense that internal surveys were insufficient or there wasn't quantifiable data around what other staff's viewpoints were. I was able to say—here is the actual data of the global workforce and the impacts that women of color, people who have caregiving responsibilities, etc. Being able to point to - rather than just anecdotal conversations from the hallways or emails from different staff, to have hundreds of staff, that was fruitful."

We went about answering our question by disseminating a survey to all our webinar participants and had honest conversations with those who were familiar with our findings. In tandem, we also undertook a small literature review to learn from the evidence base on how knowledge is best understood and applied. The overall conclusion was that our work, while well received, did not result in any significant adoption. 

What happened? And what could we have done differently or better? Our reflections, conversations, and reading of the literature led us to the following lessons:

Prioritizing inclusion from the outset can strengthen research design, results, and dissemination.

While "inclusive knowledge management" is relatively nascent—though gaining traction—our literature review shed light on ways in which we could have more intentionally brought an inclusion lens to our research. As part of our co-creation process, we sought input and guidance from a Technical Advisory Committee (TAC), which was diverse in terms of gender, race, and the size of the individual's respective organization. However, this group was composed of senior-level employees at US-based organizations. In retrospect, broadening the group—or having another forum to include perspectives such as non-Western practitioners, those in more junior roles with less decision-making power, and persons with disabilities—would have added nuance to our research design and tools, as well as the ways in which we distributed our findings. 

"I have tried to use parts of your study [as a way to relate to all kinds of marginalized groups]—like veterans, persons with disabilities, that are rural populations, as well. I have emphasized those three groups as part of my outreach to say that [DEI] is not just about race and ethnicity [...], and that has helped when I'm having conversations with people [...], and that's what we could have done more of."

For research findings to be influential, focusing on overall visibility is insufficient.

We needed to rely on existing knowledge platforms/brokers to communicate our findings in a more targeted fashion, particularly to get the attention of decision-makers. As one of our respondents noted, we may have been more effective had we harnessed the power of membership-based organizations. Indeed, this reflection is backed by the literature, which also told us that social capital and networks are vital to knowledge dissemination:

"Among [some platforms], they represent between 500 to 1,000 CEOs and executive directors of international development firms in the community. So they can sponsor a webinar or roundtable or breakfast that convenes CEOs, and they actually attend. So, these platforms give you that access that it takes to get the CEOs and decision-makers on board. They defer their LinkedIn social media to the comms folks. To get the results to the folks directly, you need the leverage of these organizations to convene a CEO or Executive Director webinar, and on top of that, they will put on a webinar on your own. They'll promote it through all their channels."

We learned that sustained efforts are necessary to maintain momentum, particularly in an environment where organizations are feeling paralyzed by a lack of optimal arrangements.

One interviewee noted that while their organization understood that underrepresented employees preferred remote work, how to best engage them and maintain a collaborative atmosphere, particularly in hybrid settings, remained unknown.

"What is the culture of this organization post-pandemic? What are the unwritten rules about how to go about doing business? We're making a concerted effort, especially for people who were hired during the pandemic, who have been given a lot of flexibility and leeway, and now we are showing them the guardrails that were always there but weren't transparent to them."

Another interviewee noted that these struggles are likely leading to declining political interest in organizational change and that companies are returning to more familiar modalities. Overall, we recognized that our study (like most studies on COVID-19 and the workplace) spotlighted issues and ended with a call to action—we did not offer a suite of evidence-based options for organizations to pilot. We did not take advantage of our visibility to continue investigating the topic, including studying organizations' experimentation with different work arrangements.

"I saw a lot of research on these topics [during the pandemic]. Now it's like, 'OK, COVID passed, or the height of it is past.' There's not as much research or work around these topics, so I think people have returned to our more or less normal lives. And I wonder if there's a way for us as a sector to not deprioritize those impacts or those questions we posed and on the longer term impacts of COVID on our ways of working."

Takeaway:

We are learning that any research—no matter how well planned or executed—can only move the needle so far unless 1) they are inclusive and co-created from the onset, 2) they leverage strong social capital to have the ears of decision-makers, and 3) are sustained, interactive, and speak to the practical needs of our audience. Frankly, there is also the element of luck, and opportunities emerge that you cannot always pre-plan. While the outcomes of our studies were less impactful than hoped, our follow-up exercise helped us recognize our shortcomings, examine our expectations, and recalibrate how we move forward with our future studies.

The context of this work is somewhat of a niche area and set of intended 'adopters' in Feed the Future Market Systems and Partnerships’ (MSP's) larger set of activities around market systems development (MSD) and private sector engagement (PSE)—some of which have shown greater progress in uptake. But the lessons remain relevant for MSP and for all of us. This is one of the first influence-harvesting exercises under MSP, but it won't be the last. These are important reminders to be aware of the risks inherent in our lofty aims, to not put blinders on, to design smartly, to continue to get feedback to 'check ourselves,' and continuously adjust and refresh to be more impactful.