Practical Tools for Evidence Based Policy Making: Developing Lines of Argument

The focus of today is about the practical use of tools for evidence-based policy making. To give you a little bit of background into this, what I’m going to present draws on the work that I’ve been doing for Defra for over the past four odd years. When I started at Defra I was put in charge of the evidence-based policy making project without a clue of what that actually meant. And there’s a couple of things that I actually ended up contributing to. One is Defra’s approach to evidence and innovation which came out of the work on evidence and innovation strategy and if you go to defra.gov.uk and type in various combinations of evidence-based policy making, evidence innovation, you’ll come up with some of the pages there.

I helped to develop the guidance on evidence-based policy making for Defra policy makers drawing from all of the literature, and there are several people in this room who have been extremely helpful in helping me think through that and think what it means. Defra’s evidence and innovation strategy was an attempt by a whole department to systematically map out its entire evidence needs for policy, put them out to consultation and determine from that what would be the resource allocations for evidence, for policy for the next five years. There is an article in the works on why that was extremely difficult, but it fundamentally it came down to a new document which was our approach to evidence and innovation which sets out the way that we think that policy makers in Defra—and this is more or less limited to environmental policy making, we haven’t explored its application in any other areas of policy making. We think there are some underlying principles that can be taken out to other areas of policy making like, I don’t know, say youth offender policy or other sorts of home office policy, but everything that I say I would like you to think about in the context of environmental policy making because that’s what it’s drawn from. There’s another publication that’s coming out in 2008 where basically what I’m about to present is going to be written up.

I’m an independent consultant and I’m sort of mouthing off on behalf of Defra so please don’t take anything that I say as Defra policy. I want to talk about the policy trajectory, first of all, and think about some of the external drivers. There’s a real change in the policy environment towards a recognition that policy goals are increasingly complex, they’re increasingly nebulous, they’re increasingly contestable and open to multiple interpretations. And anytime you put the word sustainability or sustainable in front of a policy goal, you open yourself up to all of this, and not only do you do that but you also start opening departmental policies up to cross-departmental linkages, and so you’ve got what was the DTI—the Department for Trade and Industry— with a focus on energy… suddenly you’ve got Defra’s influencing department with its focus on climate change, you’ve got these incredibly complex interdepartmental linkages and interdepartmental histories that are going to effect not only the way that policy goals are formulated but just the function of an extraordinarily large bureaucracy effects the way that they are going to be delivered.

Another point is that although most of the public service agreement targets in the UK stop at 2012-- there are two of them and Defra holds both of those which go out until 2020 which is sustainable consumption production and 2050 which is climate change. And these are the public service agreements that are agreed with the treasury that drive the formulation of policies. And with the rise of horizon scanning in futures and the office of science and innovation in the foresight program, there is increasing recognition within policy… and it’s fairly well recognized outside policy but how do you actually take it within policy? But how do you take inside the fact that future evidence needs will be interdisciplinary, overlapping and evidence needs are constantly emerging. It’s not a question of setting a policy goal and then moving towards it in the same way that you might set a research hypothesis and move towards it. You’re constantly being battered by new and emerging evidence. And government’s role in fostering innovation is not well understood; there’s a lot of work being done in science policy up at Manchester and done in Sussex on government’s role in innovation, and I think it’s safe to say we really just don’t understand what it is.

But there also these two competing pressures. One is to open up the policy process and to make it more transparent and accountable as part of the democratic process, but there’s another pressure, which is to improve analytical rigor and improve the rigor of the structures that policy-makers use. And I see these as sort of conflicting pressures and Andy Sterling has written about opening up as opposed to closing down. And I think that’s fairly true but there’s another complication on top of that which is that government tends to, especially if science departments such as Defra, we understand what rigor means. We understand how to do it. We’ve got lots of guidelines… we have lots of guidelines 2000 from the office of science and innovation and it tells us how to commission science and research. We’ve got a lot of guidance; there are all sorts of handbooks on how do you commission research, how do you peer review it, how do you run science advisory committees? And that’s part of the program, that’s been programmed into government, but there’s new pressure for opening up, there’s new pressure for civic engagement and for transparency.

I don’t think it’s been programmed in the same sort of way and I think it’s been more ‘projectised’. People say well let’s run a citizen’s jury over here. We’re Department of Health, let’s run a massive engagement exercises over there. Let’s just have a (GM) nation debate. They run these little projects and the problem with projects is in budget crunches those are the first to go, that you retreat into the war of what is programmed and the projects just get shed. So the tendency when there is a budget crunch, which there always is in the public sector, is to go back into what is programmed which is the analytical rigor and to loose the engagement base and that’s been a real problem for us.

We worked very hard to understand from within the policy process what we meant by strategy policy, development policy and delivery and with an enormous amount of word smithing going backwards and forwards we came up with these three definitions which we’ve used and, which we’ve used to help us understand what it is that policy makers are trying to do within… as they play their part in the democratic process. And the bit I want to focus on is the bit in the middle which is policy development which we, in terms of submitting positions to ministers and developing policy options and putting them forward to members for decision, we think that the policy development is about structuring choice for decision makers that’s based on robust evidence. Now this doesn’t mean that ministers are the only decision makers in the policy process. Decision making happens at all levels. It’s a constant filtering of evidence and deciding which evidence goes forward, which evidence does not end up in front of the minister, but the point is that it’s about structuring choice and that it’s about robustness in the evidence base. But you can’t take it apart from, you can’t just subtract that from this definition of strategy which is all about setting the direction of travel, understanding of the direction that you’re going in, and also working on policy delivery which is about... delivery bodies are charged with delivering measurable change, and that idea of measurable change, um, needs to be measured, obviously has to feed up into the process of policy development and into the strategy process.

And this is based on work that actually came from work that I had done at DFID which is about the evidence being contestable, being negotiable, having multiple interpretations. But I want to focus right here not just on evidence but on evidence for policy. I’m talking very specifically about the policy environment, I’m talking about the research environment. I’m talking about evidence that is used in the process in development of structure and choice for decision makers. It’s robust information but there are multiple criteria of robustness. There’s no gold standard and what that means is that robustness in the evidence base includes best practice in stakeholder engagement. What tends to happen in the policy process is that they retreated to this programmed process where analytical rigor is paramount and they go out and test it against various stake holder views and so it’s a slightly sort of, you think about the consultation process for British policy sort of developed with inside… sort of developed within the policy environment, you then put it out to consultation and see who says what about it and then you take it back within the black box of policy making. But what this says is, because it’s negotiable, that’s the wrong way of doing it. You’ve really got to break down the barriers between the black box of policy making and civic engagement.

The setting for the work that we did was sustainable consumption and production policy and which was set out as the new priorities for Defra in 2005. There was a new strategy document. It’s about new products and services, it’s about lower environmental impact, it’s about understanding the global footprint, it’s about making a change, it’s about recycling, it’s about waste. It’s about all these different things but it’s very uncertain because of the problems with defining sustainability, we really don’t know what scale of the challenges either over time or in space if you like, globally. And it’s also, Defra does not really hold policy levers. We cannot raise taxes, we cannot out regulations into account, we cannot…it’s other government departments that hold the policy levers. So the SCP— shorthand for Sustainable Consumption Production-- the SCP program is an influencing program. We need to build an evidence base to influencing others. They can then put regulations or do the negotiations or whatever it is, so that they can then use the policy levers. And the question we had was, how do we construct an evidence-base for SCP policy? We were doing this more or less from scratch. There was one existing program that had a fair amount of evidence and that had done a lot on energy ratings on white goods. But we didn’t have very much more than that and we were given a blank sheet of paper which is really rather exciting.

But there was some serious internal drivers. Very, very small group of people for a very, very large policy area. I mean the evidence team was I don’t know… You may have to help me translate some of you… but there was… The evidence team was one grade seven who was sort of middle level, middle seniority, yeah? One grade seven, one HEO, so there’s… it’s kind of two and a half people to build the evidence base for sustainable consumption policy and it was never going to get any bigger because of the constant efficiency reviews and the constant downward pressure on central government budgets. And also, this a reality that policy makers really have… I mean, it’s got to be an elevator pitch that you give ministers. You are extremely lucky if you get half an hour with a minister to present, you know, green products… something as enormous as that. And it’s complicated by cabinet reshuffles. In between 2005 and 2007 we had three secretaries of state. We had Margaret Beckett, David Miliband, and Hillary Ben. Margaret Beckett’s focus wasn’t on SCP issues. David Miliband was extremely focused on SCP issues, and Hillary Ben we don’t really know. So there was a sudden surge of ministerial interest that went from nothing, it suddenly became very high and then dropped off, and that has a real effect on the way that you work.

So this was the question, how do we provide a sufficient breadth of evidence to ministerial discussions of the policy landscape, whether NSCP covers, it doesn’t cover climate change but it covers pretty much everything else. But ensure rigorous analysis of the alternative interpretations because it’s about sustainability, when the reality is that ministers coming to the team and asking very, very narrow problem specific questions, so how can you set this all in context?

We took the view that the evidence for policy emerges from three things. There has been, especially in a science heavy department, a tendency to focus on data. It was three, it’s now two. This is a good thing, and that’s the sort of certainty that the ministers like. They love the certainty of data, they like to make pronouncements about things that are happening or not happening. But we said, well, behind that, there’s got to be some analytical evidence. There’s got to be (this is where the phrase line of argument came in) there’s got to be a line of argument for why the move from three to two is good or bad. You’ve got to have the rigor underneath it. And if you don’t have that rigor, then you’ve got to go out and engage with you stakeholders to develop it. But if you do that you’ve got to open up the policy process, you’ve got to admit for the alternative lines of argument.

And so we thought, we went back to this tension between opening up and then closing down. We said what we want is to open up but in a structured way. We’ve got to have the analytical rigor. You can’t go at… one of the things that somebody said to me when I was at DFID was the plural of anecdote is anecdotes, it’s not evidence. And so you can’t just go out and choose anecdotes, much as ministers like them, as ministers relate case studies, as much as they can relate to the stories being told and they can then use those and then take them out into the political process, it’s got to be more rigorous than that. So how can we develop what we called a rigorous dialogue? And that was the phrase that stuck in the minds of the policy team was rigorous… how you establish a rigorous dialogue. And Phil Davies at the previous INI seminar was talking about theories of change and how difficult it was to engage policy makers around the ideas of theories of change, and we found that the phrase line of argument worked a lot better and says, then, what is your line of argument. He said, “What’s your theory of change?” and they look at you and as if you’re writing your economics PhD. If you say “what’s the line of argument” it’s a much friendlier way of talking to people.

But what we wanted to do was engage with a very wide variety of stake holders to get this rigorous dialogue. So we can talk about the different definitions of the goal. We can talk about the different values that underpin those definitions, and we can talk about, the SCP went out into a group of stake-holders-- about 25 or 30 people-- and said work with us on sustainability. What is it that you want us to mean by sustainability? And the answer came back, actually we don’t really care what you mean by sustainability because you can put any weasel words up there. What we really care about is the path that you’re going to take for whatever definition this is. That was quite a revelation for us, that people were not that bothered necessarily about an exact definition of the goal but it was very important for them that we took all of the different varieties of paths that we could take.

And so we went back to the literature and I drew up all the stuff from Robert Chambers… I had grown up with all of the Robert Chambers stuff on participation and appraisal of things and we looked also at the UK and US science policy work with Sterling and Jasanoff and Gibbons and we went out to Rayner in political science. We then found a bunch of guys from IBM who were wandering around Defra being intelligent and learned a lot from them in organizational behavior and workshopping techniques. They introduced us to the Cynefin knowledge management framework which was developed by Dave Snowden, and after a lot of sitting and thinking and putting it all together, this is what we came up with.

It’s a technique called the five Why’s. It’s based on a standard organizational change interrogation technique where somebody comes to a consultant and says I’ve got a problem and they say, well, why have you got this problem and you tell them why and then they go why, and the literature says that after five Why’s basically that’s is, you can’t go any further. What we did was to flip this on its head and to say that we’re not trying to drill in to find the absolute root cause of a problem, we’re trying to do it sort of backwards.

The first thing we need to know is to understand why different stakeholders think this particular issue is important, which begins to get into the value system. Then we need to understand why is change happening and part of this is to encourage policy makers to look over the horizon. The time pressure on policy makers means that it is this far ahead of them but it is not helped by the fact that the PSA targets only go out to 2012. So why is the change happening? Where do you get your information on why change is happening?

But then we want to begin to get at this idea that there are multiple pathways and that these different pathways are equally valid and it’s the choice of the decision maker to choose which pathway policy is going to take. So why do we need to intervene in the change process? Why do we need to change the rate of change? Why do we need to change the scale of change? And again people start expressing their values. The way that they express the answers to that depends on their value system. You can’t go to somebody and say, well, “what are the values that you want?” Because that wouldn’t work, but if you say to them, “Why do we need to intervene in the change?” “Why do we need to change the rate of change or the scale of change?” You will begin to get at these underlying values.

And then we say, well, why does government need to intervene, and this begins to get at the role of government in fostering and supporting innovation. We really don’t know, let’s just get a wide variety of views on why government needs to intervene in the change process, and at any of these points you can say ‘no’. And at any of these points you can diverge as a group, and it works better as group work. And then this is all summarised into the line of argument for that particular group at that particular time which is why do we need a policy on this particular issue. And once you have that understanding of why you need a policy on this particular issue, then you can ask the question what evidence do we need? What’s the evidence base that we need to inform policy development?

If you go straight in to a group of stakeholders and you say here’s the policy goal… This is where I have to draw, sorry… You go to a group of stakeholders and you say, here is the policy goal and this is sustainability, and you say what’s the evidence that we need? And you’ll get a whole load of different bits and pieces of evidence, and you might have one sort of lobby group up here that thinks that this sort of evidence is particularly important, but that’s there’s a great big gap in the middle, and you’re making huge logical leaps between the evidence that is there and the goal, and what tends to happen is that people shoehorn it and they say, “Well, of course all of it is important. It’s extremely important.” And then these people shout at those people and then these people shout at these people over here and then, this person goes off with a huff. You get in a situation of knowledge fights where you’ve got dissidents shouting at each other and you’ve got stakeholders shouting at each other. And they’re all shouting about whether this is evidence is better than that evidence, and none of them are thinking about what is the line of argument. Is it this part of the goal that you’re working on? It may be that this group of people are working on that part of the goal, this group of people working on that part of the goal. Unless you map the whole thing out you can’t see any of that. And policy makers can’t use any of those maps to understand that this group is working on this line of argument here and so then therefore they think that this is important. This group is working on this line of argument, and actually they happen to agree that these bits of evidence here are important, but they would include that and it’s for the decision maker to choose which one it is.

But then as I said unless you create the lines of argument first you get a very research driven, you get a very stakeholder… I don’t want to say stake-holder driven, but it’s not lead by an understanding of what the policy goal is. So we did it and we did it for... we did it twice. We did it one for sustainable products and one for our sustainable consumption agenda. The sustainable consumption agenda was difficult but that’s partly because the policy goals weren’t terribly well formed. The sustainable projects agenda I think worked very well, but we had some marvelous groups of people. We had somebody from the meat and livestock commission and somebody from the motor manufactures and traders about the effects of international globalization, and they were going at it hammer and tongs. It was wonderful. It was a really good way of engaging very disparate groups of stakeholders. And we had one line of argument where the group agreed on what the goal was, and then they split off and one group created two lines of argument, which we thought was marvelous and we hope validated the technique.

It was also a very cost effective way of doing this. We had tried previously to go around to policy makers and say what’s the evidence that you need for your policy, and we went out with a blank sheet of paper and we came back with a blank sheet of paper. We had twenty-six pages of very, very, very detailed evidence needs for two workshops. Okay it was quite expensive. It was 120 people in total, you can work out the exact time cost of that, but it was two one-day workshops of about sixty people each. We allowed multiple and competing goals to co-exist, we broadened out thinking about innovation, we actually came up with two policy holes and this is in an area that people had been thinking about for about two or three years, and in two workshops we were able to identify two policy problems that actually made policy teams gulp.

We communicated some very complex messages into policy. We were able to take these lines of argument to the policy teams and to say, these are some alternative lines of argument you may not have thought about. Now how are you going to deal with this? But what we were also going to do was not communicate a research question to the external stakeholders, but to help people understand what the policy question was, which is different from the research question.

We think it’s very good in new policy areas, we think it’s a very good way of scoping an area where there is little evidence. We think it’s also very good in existing policy areas to sort of check how much policy-relevant evidence there is. There may well have be an enormous amount of research-based evidence, but is it actually relevant to the research you’re actually working on? The research hypothesis is not the same as the policy question.

It’s also relevant to somewhere where policy goals have shifted, where you can say, well, how well aligned is the current evidence base, and also where you need to think slightly more strategically about the evidence need for policy because of the backcasting work that the workshop starts off with. We can also show policy makers… we’re trying to get away from this false consensus, from this idea that policy making is about building a consensus that you then present to ministers. It’s about choice. It’s about presenting choice but doing so in a structured way, and so encouraging challenge, encouraging alternative interpretation. And we think it begins to help put the politics back into policy making by encouraging that choice. By saying to the ministers, these are all of the different lines of argument, it’s your choice. And then having that transparency so that when somebody comes back, you can say well, it is a political choice. This was not about false consensus. This isn’t a black box of policy making, closing down. This is actually trying to be open, trying to be transparent. Trying to break down those fuzzy lines and trying to make policy making more fun and political.