Evidence: Credibility and Communication

One preliminary point is that many researchers are not explicitly interested in changing policy – therefore do not put much effort into operational orientation or policy communication. There is an issue of different objectives and incentives for researchers and policymakers.

A point emerging from the cases is that strenuous advocacy efforts are usually required to get research into policy. As the Indian watershed case noted: “Those who set out to influence policy should be prepared for hard work … Though difficult, almost torturous, the effort appears worthwhile.” Research results may take a long time to influence policy – see for example the Peru inflation case (8 years) and Kenya Animal Health case (20 years).

The case study experience suggests it seems rare that a single piece of research leads to a policy change. The case of Measuring Inflation in Peru provides one interesting example. Another example of research-led policy change is the case of Rainwater Harvesting in Tanzania, where a programme of research (and communication) over a ten year period has led to a change of policy.

The case study findings reflect various streams in the literature. First, some highlight how research findings usually percolate slowly into policy arenas (e.g case of G24). Second, research creates stores of knowledge that are drawn upon as a policy need arises (e.g. Chilean fiscal policy and WAPDA in Pakistan). Third, bodies of evidence build up until they reach a “tipping point” when policy changes (Kenya Animal Health Workers case).

What kind of research and communication convinces policymakers?

• The importance of participation is a key issue that has emerged so far. It is seen as particularly crucial for successful implementation – see the cases of DELIVERI in Indonesia; Governance Indicators in the Philippines; Watershed approach in India; AHADS in India case. In the AHADS case, for example, money had been thrown at the problem before – the crucial issue in the enhanced impact recently was to make the project implementation participatory. Participation also emerges as important for public policymaking, for example in the Kuppam case in India using ICTs and Voices in the Street in Romania. This reflects the discussion in the recent assessment of id21 about the importance of local research and community organization.
• It is difficult to make a comment about the quality of the research in all the cases. Nevertheless, the issue of credibility does emerge as important. For example, in the Croatia case, a key issue was “a firm scientific basis (i.e. methodology), divorced from all political (mainly regional) biases was needed as the only defendable argument immune to regionalisms and border-tailoring without objectively firm criteria.” The credibility of an expert panel was important in the case of fiscal policy in Chile. The lack of serious criticism was important in the case of Education policy in Peru. In contrast, contested research hinders policy change – in the Delhi case, different research institutions came to opposite conclusions. Credibility also emerges as especially important for think thanks to have a sustained policy impact – e.g. the credibility of the organization is important for the cases involving TIPS, AFREPREN, GRADE and TERI.
• Another important issue is that theoretical knowledge needs to be contextualised and made applicable to the cultural, political and economic conditions if it is to influence policy. This was the case with policymaking in Iran – where the policy impact was limited because the research was from a single discipline and did not address other relevant issues (see also the case of Morocco). The case of Biodiversity Management in Saudi Arabia provides an interesting example of the way new ideas were linked to traditional management concepts in making research findings operational.
• Policymakers seem to be particularly convinced where something has been piloted and works (E.g. see cases of Rainwater harvesting in Tanzania, Animal Health Workers in Kenya, SMEs in Ukraine, DELIVERI). In the GALASSA case, the pilot was convincing even though the approach was not widely applicable. It may be that policy makers are often more interested in “how to do” than “what to do” research.
• Research with an output / operational orientation or action research seems to have a great impact – see DELIVERI, Public Administration in Ukraine and Caribbean Health cases. For example, the lack of success in the case of health finance reform in St. Lucia, and St. Vincent is attributed to: “very little operational research accompanied or followed the initial situational analyses, and identification of the elements of an organizational and management framework, the cost of services, as well as a legal framework.”
• Where there has been conscious effort to influence policy – with clear communications and influence strategy from the start, there has been greater impact (e.g. DELIVERI and SME cases in Indonesia).
• Interestingly the issue of credibility does not just concern the quality of the research but also the way that research is packaged to make it palatable to policy-makers (e.g. TIPS in South Africa). The evidence from the case studies does support much of the existing literature that the format of the research outputs also matters for policy impact. For example, the Lithuania case concludes: “The format in which the results were produced made the recommendations attractive and ensured a greater policy impact.” Researchers in the case of Education policy in Peru have partly succeeded in influencing policy by making their statistical findings understandable to non-specialists.
• Only a few of the cases involved the media. This may be to do with the lack of access to the media (supply side) or that there is little interest from the media on the issues (demand side). The comparative case of agricultural research in Ghana highlights the engagement of the media as one of the issues that affected the relative success of the two projects. This is also true for the case of Voices in the Street in Romania where the visual and popular impact of the research led to media coverage - including prime time news – which influenced the political environment.

It seems that two overarching issues emerge here in order for research to have a policy impact. The first has to do with relevance (in terms of both substantive focus and operational usefulness). The case of SMEs in Indonesia provides an explicit case where the focus on issues of policy relevance matters, but this issue emerges more broadly. What mechanisms can help ensure topical and operational relevance? Including local people (as with participatory research) can help ensure topical and operational relevance for practice on the ground. Including policymakers in the research process can help ensure topical and operational relevance for public policy. The second set of issues has to do with credibility. Mechanisms to improve here seem to be improving the nature of evidence (issues of rigorous methodology and piloting approaches) and the way research is communicated (e.g. via demonstration or short, clear, jargon-free documents, etc).