Andrew Barnett
Transcript of a presentation at the ODI/INASP Research-Policy Symposium, Oxford, November 16th/17th 2006.

Final Summary and Round-up - Comments to the Research-Policy Symposium

My purpose is to highlight major themes from the first day of this most interesting conference. I believe that there are three major issues.

What is evidence?
We have been told that information is not evidence, and I would fully agree with that, but today's discussions prompt me to ask whether a narrative or a story is evidence. It is clearly not evidence in the sense implied by the phrase 'evidence-based policy'. In the UK, 'evidence-based policy' in recent times came out of our Prime Minister's Office as a contrast to ideologically-based policy. It emerged after we came through a phase of the extreme ideological positions of the Thatcher and Reagan years. The idea of evidence-based policy was precisely to put limits on that ideologically-based policy.

This, and the discussions at today's meeting, suggests that there can be many drivers of policy change: politics, ideology, fantasy, whim, or even evidence. Do we really want government by anecdote?. In this context I took very strong exception to the Oxfam presentation. Duncan Green told us how life is in the 'real world' of lobbying, but I am not sure that this is the world as it ought to be. I think that we have a responsibility to challenge this dumbing down of policy debate and the reasons for policy change.

Nicolas Ducote from Argentina talked a lot about how evidence can help to improve things and I think that he is absolutely right. I think that what our friend from Indonesia was saying about evidence being a stick is also true: it is a means to hold people to account. There is a great quote, which I cannot remember precisely, from John Maynard Keynes, along the lines that governments hate evidence because it limits their room for manoeuvre. Sometimes they simply do not want to know, because it conflicts with their ideological position. I think that the idea that someone mentioned of 'alternative statistics' is a very powerful weapon in these circumstances, and it is of course not offering an anecdote, or the simple story, but in fact quite the reverse; it is trying to add complexity to the understanding of a complex world.

It would be interesting to know if DFID thought they are doing a good job in reducing poverty. But in reality they are probably the last people we should ask about this. It would mean a great deal more if the assessment of their performance were provided by an independent source who could validate the evidence base.

I am forced to conclude from my own experience that Oxfam is probably right to say that policy influence correlates highly with the simplicity of the message, but I would ask what else correlates with the simplicity of the message. Getting it wrong probably correlates highly with the simplicity of the message. This then raises a question, which I think we have not really discussed here, about the validity of evidence. This depends on the circumstances, clearly, but I certainly do not take the postmodernist view that there are domains of knowledge that are all equally important. I think that there are domains of knowledge that have to be taken into account, but in my experience, there are certain types of evidence that are more valid to me than others are. I think once you abandon that idea, you are in a real mess.

In this context I am also very concerned about the word 'advocacy'. To me, the word implies the spinning of the evidence. Dylan Winder reminded us about the recent report of the House of Commons Select Committee on Science and Technology (see: www.publications.parliament.uk/pa/cm200506/cmselect/cmsctech/900/90003.htm). I have not read the report, but I did read the press statement , which really seemed to suggest that the real problem the House of Commons Select Committee was facing was the government's spinning of scientific evidence and how this undermines the credibility of science itself. This decline in the perception of the credibility of government science appears to have been associated in particular to the rise of Bovine Spongiform Encephalopathy (BSE), or the earlier nuclear debate. We are just about to have another big nuclear debate and I worry that nobody believes anybody else's numbers because they are all associated in one way or another with particular lobbyists.

So I would like to end this theme by saying that there are a whole set of important issues about evidence-based policy. John Young seemed to be telling me over coffee that he thinks evidence is anything that works to influence policy and, again, I think that that becomes far too wide a definition. Similarly I think there is a great danger in suggesting that evidence is every possible output of research. If you do that, it seems to me you are loosing sight of something very important about 'evidence' as a basis for informing policy change and practice. If you want to know what research is and what science is, the OECD have written a manual on it which is accepted by all OECD governments, called the Frascati Manual.

Evidence and Policy
The second thing that I wanted to talk about was the interface between evidence and policy. I find it useful to see the knowledge system as a system rather like the food chain. So, for me, one has your Oxfams as the 'top feeders' and one has organisations such as ODI at the next level down, comprising the think-tank people. By the way, huge changes have happened in this sector over the last thirty years. Think-tanks used to have some sense of independence. My understanding is that now most think-tanks are associated with a particular ideological position. That raises a very interesting set of issues and makes being a policy-maker very tricky indeed, because one knows that the people who are supplying information are biasing it, because that is their job. ODI clearly does not spin the evidence (!), although they do have an ideological position associated with ideas like poverty reduction (which seem to me to be an entirely worthwhile thing to be committed to). But the point is to make that ideological position clear.

Further down the food chain there are those groups associated with training, and the provision of text books and even eBooks. At the bottom of the chain there are the 'heavy lifters' or 'bottom-feeders' comprising the people doing the theoretical work, generating the new models and doing the huge long-term epidemiological studies. I would argue that you cannot have one part of the chain without the other. At the moment DFID see the top-feeders as particularly important because there has been a recent pendulum swing in that direction. I was part of the review process for DFID and I think it is probably true that much research had become too fundamental, too abstract and too 'bottom-feeding'. But it is inevitable that they will discover in five or ten years time that unless people are doing the heavy lifting, there will be no material for the 'top-feeders' to use. Taking the Korean example, it took them 30 to 40 years to build the necessary capabilities, and they engage in all aspects of this 'food-chain', not just the policy analysis.

Today we have often talked as if researchers are an undifferentiated block. Another substantial change that has occurred over the last 20 years is that researchers used to be the people who sat in universities. I would say now that in the majority of the countries represented at this meeting, most researchers are no longer based at Universities. Many of them are in consulting services, in NGOs and in a whole set of organisations of that sort. The term 'researcher' needs to be unpacked as different types of researcher have different relevance for policy making.

The validation of different voices
Another theme that came across today was about the validation of other voices. This includes the hugely important issue relating to indigenous technical knowledge and the experiences of 'end-users'. There is a question about the place of these various types of knowledge in the food chain and there may well need to be people who operate as 'intermediaries' who are able to make this knowledge accessible to different audiences. Somebody at the meeting today was talking about being overwhelmed or swamped by knowledge, and I think that the rise of quality-assured portals is one response to this. There are teams of people who perform these quality assurance intermediary functions. It does not help the individual if they get 5000 references to particular bits of knowledge from Google. In relation to development policy, what they need to do is to go to the Institute of Development Studies (IDS) website, which gives their opinion of the top ten things that you really need to read. That is a hugely valuable service.

Models of Policy Change
In terms of conceptual models, we have already mentioned the knowledge food chain. Another that our colleague from Pakistan (Dr Naim) talked about so wonderfully concerns 'innovation systems' and the Korean experience. Somebody recently told me that 'research translates money into knowledge, and innovation converts knowledge into money'. That is a very nice way of looking at it. Both are required, but one produces knowledge and the other produces money. What I take as the lesson from this is that in addition to worrying about the 'supply side' of knowledge creation, one also has to work tremendously hard to strengthen the demand side and, indeed, to bear in mind the huge importance of intermediaries who are working between suppliers and end-users of new knowledge. I once asked John Young what three things he had learnt over the past five years about research and policy and he said, 'Engage, engage, engage.' I think that is also very much the lesson from innovation systems thinking. But I think there is still a question as to whether this innovation model, which is very highly articulated around the world (people around the world really know how to do innovation these days), works in a policy arena. I personally think that it does, but I guess that it is still an open question.

Many people at the meeting talked as if it is only 'governments' that make policy. Of course policy-making is done by lots of other people, including the private sector and households, and there are a whole series of things we did not talk about today that I think we probably should have talked about. Looking at the policy-making area, some of the key ideas from innovation theory were mentioned: engagement is clearly central, that is what we are hearing; intermediaries are important, as are people such as yourselves, including communication specialists. And of course, the dreaded lobbyists have a role in innovation systems.

Where governments are not interested in hearing evidence that might inform policy choices (and this is different to not being capable of hearing such evidence), I think that journalists, and science journalists in particular, play an important role in opening up a space for the rest of policy analysts and communicators to insert their knowledge. Without journalists finding an entry point to disinterested governments that becomes very difficult. So science journalists form another part of my knowledge food chain.

I love the idea that someone mentioned about 'policy windows'. I was reminded (I am an old man) that in 1968 my professor said, 'There are four things that you have to worry about in relation to communications. The impact of communications is a function of: a clear narrative; a knowledge source that is credible to the audience, a message in the right format and timeliness.' So the idea of policy windows is just another way of saying that to be effective in changing policy research inputs must be timely. This makes me feel that I have not heard a lot today that has moved us on from 1968. I know that that is a terribly irritating thing for old men to say to such a young audience, but I do think that you need to keep that in mind. We are talking now about new policy models, but they are not really new; they are just policy models coming round for the second or nth time.

For instance another new model about which I am very excited concerns the idea of 'drivers of change'. This is a new DFID label, but it is really about the political economy of change that has been discussed for many years and is now coming back into vogue and is raising important issues (see Policy Briefs at www.thepolicypractice.com/). 'Drivers of change' is itself a research-based idea. It did not just come to DFID out of the blue; it came from research. In a similar way to a number other ideas that have become DFID policies, the adoption of the drivers of change approach was crucially mediated by an internal product champion. But one of the things that I expect in DFID over the next few years is that as the money goes up and the number of people goes down, there will be fewer and fewer of these internal champions.

As we move on to the issues of political economy and try to understand why change occurs, there are interesting questions about who has the legitimacy to be involved driving such change. Do you want DFID to be engaged in the internal political processes of developing countries? The answer is probably, 'yes, sometimes.' In the discussions today we have had examples where an external force can help progressive elements within a country, but one can certainly envision other circumstances where one probably would not want them to be driving change.

In terms of all the policy models that are emerging from recent discussion, we have seen huge resurgences of 'system thinking' - in everything, not just in innovation. We are abandoning linear views everywhere. Everything is seen as being interconnected with everything else and everything is a part of a system. We are in effect seeing a switch from what might be called the analogies of Newtonian physics, to the analogies of biology. I personally think that that is a very good thing, but it makes policy-making hugely difficult. The new models are talking about interaction, complexity, uncertainty, lack of control. In the 1960s, I used to think that all that governments needed to do was get the right information and they would do the 'right thing', but we then realised that even when they wanted to do the right thing (which they often did not), they found that they could not. This raises the question of what to do in systems in which one does not have control of all of the necessary instruments of change. These are long-term not short-term questions and they are essentially to do with trade-offs. If one does one thing, one then cannot do something else and doing one thing will probably have an impact on something else. That is in contrast with a political process that seems to be short-term and overly responsive to single-issue pressure. The knee-jerk response is precisely the problem that we have in our country and which I think the US also has. There is very little deep analysis of these issues. Someone described the US as operating 'foreign policy by PowerPoint'.

Essentially, we have learned that reductionism is important to understand things, but it can be extremely dangerous when we come to policy implementation. Our friend Duncan Green from Oxfam argued for extreme reductionism and simplification. I think that Oxfam does a great job some of the time, but I question whether Oxfam is part of the problem or part of the solution in much of its advocacy. The acknowledged effectiveness of their narrow simplistic lobbying is, for me at least, a large part of the problem!

So, I will finish with three simplistic messages. The first is: engage, engage, engage. I think that what we have learned today is that this means policy analysts, researchers and communicators engaging with all parts of the policy change process, not just at the beginning. It is something which needs to be done the whole time, because we operate in a system and the system is changing, and because we do not know at the outset everything we will know two or three years down the line, so we need to interact as we become more experienced..

Second, to use a very tiresome English expression, what techniques to use is really a matter of 'horses for courses'. This is a rather silly expression that means, if you are backing a horse, you have to check the ground conditions because certain horses do better when it is dry and some do better when it is wet. The point is not putting everything into one conceptual basket, such as 'the policy process' or 'research' or 'evidence'. It all needs to be unpacked. Clearly, there are certain policies that need to be scientifically based, and based on deep science, not simple stories. There may be policy decisions that can be done on the wing and with much less effort, but often this will not be enough.

Third, for me, the best phrase to come out of this meeting is about building 'policy literacy'. This may be policy literacy on the part of policy-makers who we may think of as being much more stupid than we are, but I would put it the other way around, or at least talk about policy literacy on both sides. Certainly, in my career I have done extremely stupid things by assuming that policy-makers could make changes without having the faintest understanding of the realities that they were facing. For me though, the policy literacy on our side means recognising that the policy-maker is always faced with trade-offs, and most of the policy advice from our side does not come in that form. For instance in my field we may advocate the use of renewable energy, but the policy-makers reply that they do not want to know what the best renewable energy device is, they want to know what is the best device to enable their people to gain access to modern energy services. This is a quite different question. Most analysts find it difficult to answer the complex questions that confront the minister..

I said there were three messages but economists cannot count, as you know. If I could count, I would be an accountant, I certainly would not be here, and I would be retired in the Bahamas.

So I would like to add a fourth point about accountability. I think that there are a whole series of issues about accountability, lobbying and dare I even say it, about a code of conduct. When we are using the evidence, be clear about who is financing it and be clear about what the evidence-base is. Even if you do not say it in your one-page summary, at least have a footnote to say where the evidence came from. As an old man, I have a great attraction to dictatorship and when I am a dictator, I am going to ban single-issue lobbyists, whether they are commercial firms, which are the biggest exponents of it, or NGOs, which are increasingly effective. On that dictatorial note, I will finish.

View summary as pdf (pdf 51kb)