The Outcome Mapping Story

I’ll say a few things before I start so we can position ourselves to lines of argument. First of all I think we are already here [points to diagram]. We’ve already decided that this is an important issue. So there’s a lot of background research that has to go into outcome mapping before you can start looking at outcome mapping. Just like, if you would do something, you would have… the assumption is that you would have done your homework. You more or less, you know, have collected your evidence… or that it’s the right intervention to carry out. Second is that although outcome mapping is designed and thought from the perspective of somebody who is outside and trying to bring about change in an environment in which they’re not in, so we’ll talk about sphere of influence you’re trying to bring about change in those you’re trying to influence. You can think about it within government if you wanted to. One of the things that you were saying is that there are of course people within government who you need to influence… Those who have the power to bring about those… the other ones to bring about… the money you need to carry out a policy, so… You can think about outcome mapping from within as well. You are able to in this presentation that I will do.

I’m not going to dwell on too many points. I’m going to try and give you a very brief introduction to what the main principle of what outcome mapping is and I’m going to touch just a few things. I’m going to touch the concept of boundary partners; I’m going to talk about the concept of progress markers. And I think it’s quite interesting if you want a line of argument after you’ve arrived at what you need to do… sorry, after you’ve decided that you need to do something about this, you know… what is your line of argument in terms of what’s going to happen from now on. I’m going to talk a bit about monitoring and evaluation and then finally I’ll tell you about how we use it.

Outcome mapping is not a monitoring and evaluation tool. It’s a planning, monitoring, learning evaluation methodology. The monitoring and evaluation part of it is nothing new. You all do it probably. It’s in the planning where the thing is. That’s the new part. Or at least it was new a long time ago. We’ve already incorporated a lot of the principles and issues that outcome mapping talks about so I think it won’t be that new for you. Before I start, how many of you have actually used outcome mapping? Are well-versed in it?

[pause]

Okay, so I won’t get into the whole because we just did a workshop last week. It was three days, supposed to be five days, so I’m not going to get into detail, I’m just going to try to give you the brief.

The first thing I want you to do, and we don’t want to do this exercise, but as I’m talking I want you to think about who are your stakeholders, so in the context in which we are working, who are the stakeholders of your line of work, who are the actors that affect the outcome of your work? You just think about that, don’t worry, if you can, as we are going through the presentation just write them down and then we might come back to it.

So a very brief history: It’s developed by the IDRC (International Development Research Centre), the evaluations unit of IDRC. One of the things that was happening to them, they were getting a lot of pressure to demonstrate results… and political pressure… CIDA (Canadian International Development Agency) wants to know what’s been the impact of the research that it funded. You want to know what’s happening right down there on down the stream, on the ground… a lot of pressure there. And IDRC evaluation unit has a long time. Their job basically is to go around helping IDRC-funded projects in the evaluation, so knowing constantly. So they tried looking into certain methodologies and the one thing that they came up with was Barry Kibel’s outcome engineering, they thought it was interesting. They tried it out, published a manual they were presenting it and then we at ODI were running a learning community so if any of this is interesting for your in that you can join that. There are some leaflets at the back so that you can join and get a little bit more information.

The thing is that, although the time is this focus on the input, but right in the middle it’s a bit of a black box. We’re really good at setting up objectives and influencing and activities. We’re really good at coming up with these indicators of results, but that part in the middle we’re not very good at. I don’t know if you can read this [slide of a cartoon], I mean I don’t need to explain jokes. “But then a miracle occurs” [quoting the cartoon], you know. And every time we write a proposal, or we write a plan to bring about change there’s this feeling that you’re lying. You’re saying, “Oh, I’m going to do a workshop, or I’m going to publish this paper, or I’m going to meet this policy maker then everything will happen.” In reality, well, you’re hoping that everything will happen. You don’t know. You’ll probably learn about it as you go along. It’s not an exact science. You’re going to have to be at that stage to see if the idea of the workshop is the most appropriate line of action. So, outcome mapping comes somewhere there, trying to say it’s not really about results, it’s not necessarily about having this perfectly logical framework, it’s really about trying to address behaviour changes (or lack of) because that’s really what we’re trying to address. We know we are going to achieve our impact, or results or outcomes, whatever you want to call it, when we get, the key actors to change their behaviour, to do things in a very different way.

One of the things that the outcome mapping team are thinking about is, when people talk about impacts (and use the word impacts any way you want, but let’s imagine impacts as it is right here, you know people have different interpretations)... When people talk about impacts, you get the impression of cause and effect, but in reality it’s an open system. There are so many reasons why things might not go the way you expected—they could be positive they could be negative. We tend to focus on downstream effects, the ultimate effects… the difference in evaluating in terms of the poverty level. How many poor people are there in Uganda today in comparison to five years ago? But in reality the important effects are upstream. It’s about the way that the government in Uganda are doing things differently and the policy makers and other actors are related to each other. Ministries are talking to each other now. NGO’s and governments are working together. Those are very important stages and when we think about impact, we don’t tend to think about them.

When we talk about impact, it seems as if credit comes to us or whoever is carrying out the work, and in reality with so many actors, it’s really difficult to understand the full picture, especially if you come from a research perspective. If you write a paper yes, somebody might reference you but it doesn’t really mean that you were solely responsible. Somebody else might have given that paper to the policy maker, or explained all of that mathematical modeling that you did in your paper that nobody understood. So it’s a contribution to the change.

And the idea of impact also suggests there is some end goal. When I had my funding for two years and the project ends, but in reality the change process never ends... especially the policy process where one starts and another one ends. To get legislation passed you need to now regulate it, you need to get it implemented, you need to communicate it, and it sort of changes… So, when does the policy process start and another one end? It’s very difficult to tell when it leads to another thing all time.

So what is outcome mapping? It’s a method of planning and assessing social effects and internal performance of project performance for programmes for organisations, planning, assessing, monitoring, learning and evaluation methodology, but the key thing though is planning. And the key reason why, and this is sort of the formal definition, the reason why social effects is put there is because they’re trying to look at social change happening in key relationships. Relationships that are flexible, multiple… and it has an emphasis on planning so I will try to focus on the planning part.

It has four stages, four steps. And it’s divided in three. The first part is the intentional design and this is what I was saying that is one of the important parts. The intention is about what are we trying to accomplish. This is the important part. This is the part where we have taken a lot of the lessons.

The first thing is about setting up the vision and the mission. The vision is not the typical vision of an NGO, or of the MDGs, or the world will be a better place in general. It’s related to our space, to where we’re working. To those we are working with, but it’s very visionary and really difficult to do in workshops. If we imagine an ideal world, ninety-five years you’ll get… you’ll run out of funding by twenty years… when the world will be fantastic… But we’re not the whole world, but where we have some sort of control. The mission is what we will do, the little parts—the very small parts of that whole vision of change is what we will carry out. Then we think about boundary partners, that’s the first concept that I want to pick on in a bit. … Those are the organisations and the actors that we are working with directly to accomplish our mission.

The outcome challenges the vision we have of those partners. How do we want those actors to behave in the future? Progress markers refers to how do we want those partners to get from where we are now to that vision of the future. We start to think about that story of change for our actors. Once we have that, those ideas for change we can think about strategy. We can think about how we’re going to get that done. We can look at a particular minister, a particular network to do things to help us achieve the end goal. And step seven is about how are we going to change: what skills are we going to need in order to carry out that strategy, what networks we need to set up, what knowledge we need, etc. It’s a simple planning process. It’s: what do you want to achieve and how are we going to get there and what are we going to do. It’s nothing revolutionary. We had a workshop last week and when we got to this somebody said, “that’s not very revolutionary”. They were expecting something really strange and new and in reality it’s not.

Once we’ve done that intentional design and we know where we want to go and how we are going to get there, then it’s easy to start out collecting information that’s going to allow us to monitor whether we are on-track to achieve that or not. Whether during that process our stories have changed—the way we think the world is going to be moving, the direction the world is going to be moving into has actually shifted. The intentional design is really important because it forces us to collect a lot of information about those that we are working with in order to react and respond to change. Anecdotes are key to this process, and collecting information in a systematic way so that an anecdote becomes something you can use when you need to make an argument for changing your value point, for changing who you are going to work with. Then finally outcome mapping doesn’t really tell you how you are going to evaluate but it gives you a wealth of information that you can use to evaluate, and I’ll get to that towards the end.

So outcome mapping has components from the previous slide, for these typical planning questions. So the Why: Why are we doing something… we’d like to achieve some mission, some ambition. The Who: Who would we like to work with in terms of boundary partners? I’ll explain that a little later. The What: What do we want to give. It’s the ideal vision for those boundary partners. How: that’s the mission, that’s the strategy. That’s about our own organisational practice, how we’re going to change. Okay? Does that make any sense?

The way outcome mapping works is that, usually, we don’t present this. We go step by step and that will give you the whole picture, but I thought it might be worth doing this the other way around.

Here are some of the key concepts, the key principles of outcome mapping. I hope that if anything from the presentation, this is what you take. One of the key things is that we need to think of ourselves, whether we’re in government, whether we’re an NGO, whether we’re a researcher or whether we’re doing more practical work: whatever we’re working in, as part of an interconnected web of relationships. We are not the only actor trying to bring about change. And there might be actors who we don’t even know about who are trying to bring about change. There are actors who aren’t trying to bring about change. They’re just influencing more than just influencing the way we think through his newspapers on certain issues but that might have an influence on the way that we think about other issues because it just creates this narrative in our minds that’s about how the world functions and we take that to another space. We need to be aware that we’re not alone in this.

We also need to recognize that change is continuous, complex. It’s non-linear and you go around a lot. You know, you can go in a certain policy direction and it might reveal something else. You might go back in policy because global media picks an issue and all of a sudden people aren’t too happy about that idea so you’ve got to go back and change.

You cannot control change. A lot of the processes that we’re engaging with are outside our reach so we have no control over them. So we have to be aware of that; we have to embrace that. And what the methodology is trying to do, what the methodology is trying to do is say, is ‘ok this happened, but how do we use this to our own advantage?’ In the face of all this complexity, in the face of all this uncertainty, how can we increase our knowledge of the process that we’re engaged in? If there is complexity, and if there is uncertainty, and if there is difficulty in getting information... if it’s expensive, how can we maximize our capacity to get whatever information we need to get? How can we tell if we’re making a difference if there are so many actors, if information is difficult to come by, if impacts that might be way downstream are very difficult for us to find? It’s very difficult for us to tell if we are making a difference. And how can we recognize who else is making a difference?

The key thing is we have to be collecting information as we go and a lot of the time when we do workshops on how to use research to influence policy or developing communication strategies, that’s where we always start. And you have to be systematic in the way you collect your information and systematic in the way you do your work. What happens is a typical DFID-funded research into policy project is: do lots of research, then do communications then evaluate. But if we miss monitoring we’re never going to be able to assess at the end of the day whether we actually had an impact because all of these different effects that we might not be in control or we were not expecting and how do you say… how do you know if something you’re not expecting… you’re never going to know (it’s this way) if you’re not going to pay attention to it in a systematic way.

Well the first concept is focusing on direct partners. That’s boundary partners. When you think about stakeholders, there could be hundreds of stakeholders. When we try to work with civil society organisations (CSOs) to build the capacity in order to be better at using evidence to influence policy, and we mostly work with civil society organisations in the south, there are hundreds, thousands of organisations we are probably reaching. But we need to identify which of these are working with directly and we look at it again and realize that we are not working with a thousand organisations, we’re working with about twenty. Those are the ones that we can directly influence. We’re trying the best way that we can support a change in them and we contribute to that vision we have of the future.

We try and recognize the limits of our influence. Say if this is us or if this is you, think about who falls within your sphere of influence? Who falls around that, close to your work or to your programme. And you look at who’s the nexus between you and the rest of the world. And all of these organisations that we might have to go through them to influence others—not all CSOs, but two NGOs, two organisations in Latin America, three in Africa, two in Asia. Those are the ones we work with. To do that, we are going to try and influence others.

We can try to think about the mapping in a more realistic way in a step by step process so if this is us and this is our own sphere of control we can influence our partners or we can try. Then they would try to influence those that they work with. And going and going until it gets to the end goal. The research institution isn’t really very relevant because we’re right here in London. Then we talk to (DFID), then DFID talks to DFID Malawi, DFID Malawi talks to the government of Malawi minister of health. The minister talks to the permanent secretary and the permanent secretary talks to… So there’s a lot of chain of connections, and if we go around saying oh yes… we’ve influenced the local government of Norway, they implemented a programme based on our research. Probably they used our research but we are not the ones who influenced them. Somebody else along that line did. Unless we monitor those small influences, we cannot claim that it’s due to us. Unless we monitor that, we cannot learn what else is the best way to reach those final users of our research or those final beneficiaries of our policies.

Going back to the way we tend to think, the idea is this: at the beginning the influence of our programme is quite high. When we start a particular intervention we have a lot of influence. We are doing the research, we are putting the funds, we are getting the consultants. We have a lot of control. As we go along and as we start implementing activities, and as we start finding the outputs, those that are being influenced, are those that start being in control of the situation. We are entering their space, it’s now our boundary partners’ space. The further down we go, we have less control, and the control switches to those that we were working with.

When you look at impacts it’s really the other way around. Where the impacts… if we were an NGO and we were working with women support groups and they were the ones that were going to bring about the impact on their children, the impact that we were measuring is the impact on children, that would escape our control. We would have contributed to it certainly. But that would be the main responsibility of those women that we were working with directly.

Outcome mapping what it does is focuses on this, the very next step after we loose control of the situation, after we loose control of the intervention. Have we actually achieved the changes that we thought we’re going to bring about those positive downstream changes we imagined?

What changes do we talk about? Behaviour changes, not just any changes. In outcome mapping when somebody says, ‘ooh, an objective of mine is that the minister is aware of the research we’ve done’, that’s not important. They have to be aware to do something with it, something that is more sustainable. But just because they read it, doesn’t mean they understand it. And just because they understand something doesn’t mean that they’re going to respond to it. We don’t want to think about the outputs we want to think about the behaviour change, and we mean through the relationships. Development, in the outcome mapping view is done by people.

So while we can try to influence them, we can’t control them so that people can change their behaviour. The things they do, and the relationships they have, and how they relate to others, then we might achieve things through influence.

Another key function in outcome mapping is that we are trying to map behaviour changes. How do we map these behaviour changes? Outcome mapping focuses on progressive changes and progress markers. We are going to say change is going to happen.

Let’s say we’re in October 2007 and we want to bring about change within a group of actors, for example in a project that involved in trade and poverty in Latin America we want to get the Peruvian ministry of trade and minister and her advisors, to incorporate into the work issues around lagging regions and SME’s and indigenous people. They are making decisions also based on that information. That’s not going to happen overnight. It’s going to be progressive steps. If you want, it’s like the scene plans for a movie for one of the characters. I would like to see small changes laid out by the minister. First, her advisors are going to start by reading our work. They will then take the next step that’s going to start calling us, every week, to get more information about using our resource centre. Then they’re going to come to meetings when we invite them. Then they are going to call for meetings and organise meetings themselves. Then they’re going to commission new research in the line of work that we’re doing. Then we’re going to bring us in and introduce us to other ministers and other partners because they think their idea is important. And then they’re going to present a policy draft for us to check. And then they are going to say ‘actually you design a policy and give it to me and we’ll use it’.

It’s a progressive change we’re trying to get in the timed part is we’re trying to get all of the actors to develop their theories of change, or lines of argument or an ideal of how change will happen. And this we do for each actor we are trying to influence. We imagine how change could happen… how change should happen if we are trying to get to that ideal future.

It’s important that these are not set in stone and I’ll get back to that when I get to the evaluation. The idea of making these changes very… of making progressive changes like these for example, and having a list of changes that will take us from now to a very beautiful ideal future… is that we can change them if we go along. It;s not two indicators that are going to be assessed by the end of the programme. By the end of the programme... I mean right here [points to part of diagram where programme finishes] we’re saying that our intervention does not end with the end of the world…. We might say we’re going to get here [again indicates diagram] but change will continue to a better ideal world, so we might continue as we go along because unexpected things happen and we’re recording them because our priorities might change, and we’re recording that, because those actors might move faster than we thought.

What we’re trying to say is that development involves complex interactions, and you can influence but not control your partners. You cannot control them so you need to think to give them this flexibility. A change is going to be progressive and small steps sometimes, because you might get it but you might not too. You need to be flexible and leave this sort of leeway to adapt your work.

An example, I mean this is a way of illustrating what I’ve just said, and some of the parts of outcome mapping. And this is for a very downstream programme—imagine we’re at the level of just losing control. So we have an NGO, and this is us, and we have some ultimate beneficiaries of the work that we want to do. These in black are our boundary partners, but there are other actors involved, we’re just not working with them. So changes in the way beneficiaries and the boundary partners behave, that’s our vision, that’s our ideal in the future. It’s not about the MDGs for the whole world it’s about that space in which we are working.

What we’ll do, which is a little bit (because clearly there are some things which they will do on their own and some other will influence them), that’s our mission. It’s a small part of it. Those are our strategies towards boundary partners, so that’s the types of things we might do. And because of that, they might do things with other partners that might be strategic. We don’t try to influence them to work with us. We tend to also forget who we are working with to influence others, but we might not be trying to influence them, it might be our partners. Because we do some things, our boundary partners will bring about influences with others, which some of them might be to each other and some of them might be to the beneficiaries, and that is what we are trying to assess. Outcome mapping says that you can assess how good your programme is or how your invention is going if you achieve these changes. But there might be other changes from other actors that we have little influence over and we have to be aware of those. Those are some of the sort of the boundary partner outcomes that they themselves carry out.

I’ve touched on a few principles but the key ones for us are boundary partners… we’re not focusing an absolutely everybody. We’re going to narrow down who we’re working with directly when talking about change. Whether it’s based on evidence or not? We need to think about change in a progressive way because it’s going to allow us a plan; those small steps… how do we get from one to the next one, so once we know, what we’re going to do we can take this direction if we do these things… and if you do that, then we can take that direction. But we’re not trying to make any assumptions about today or ten years from now, we’re trying to track, to monitor and track that in the future.

And M&E (measurement & evaluation): once we plan and we had that information available if becomes a bit easier to collect information about what’s going on. The big challenge that outcome mapping addresses in M&E is this: it’s about accountability and learning. We have a project on poverty and trade in Latin America, and we’ve been asked to provide for our accountability. You know, how we are spending the money, what activities are going to, but we also want to learn. We want to achieve the goal of influencing policy by research, but we also want to learn. I don’t want to do the work and close the project hand over the receipts and get the invoices. But I’d also like to learn from the process. It’s a bit of a trade off sometimes, particularly because of the time and cost, but also because of the type of information we need to collect. It’s very different sometimes. When we get asked by donors or by clients, it might be very different than if we get asked by those we are working with, but also by ourselves, we want to manage a really complex programme, we need to know about things that are different. So we need to find a balance; it’s a balancing act.

We might be working on a series of things. The things we are doing as an organisation in general (as an organisation or as a programme or as a project), and there is our partner, those who we are trying to influence. We work with them, on that little part of that where we might intersect. We are ODI and we have a partner in Argentina called CIPPEC. We have a host of things (happening) in ODI. We are interacting with lots of organisations, they are interacting with lots of organisations. So, there’re little parts of both organisations that are coming together and that’s where we are influencing with the activities we do...

We have to recognize the smallness of that intersection because we have to realise that there are all these other things happening. What outcome mapping does essentially is it says, ‘let’s keep track’. When I talked about organisational practices, what do we need to be able to do to carry out the strategy… Outcome mapping provides you with a tool that’s called performance journals. They help you collect information about how you’re changing, on your capacity to communicate and to do research. It asks, How’s that changing? It asks you to think about the changes in your partners. What are those outcomes? What are the changes in the organisation? Organisations need to collect that information as they go along.

Finally, think about your strategy and collect information about what you do. That might change of course because the progress markers might change, as might the context. And as we go along we need to know, we need to collect also information about the situation, about the context because those affect what those partners do and what we are able to do in order to carry out the strategy.

If you haven’t done this planning before hand and haven’t presented to yourself a very clear understanding of your role in this and how change might take place and what you need to do to make those progressive changes happen, it’s very difficult to get a sense of how you need to change, of what you need to measure. What outcome mapping will say is: focus on your planning and then you know the kind of information and collect in a systematic way journals.

As you go along you start reviewing your intentional design… that’s the whole point. It’s not set in stone. We won’t be asking ourselves questions about what we’ve said, we will want to do and what we felt… Does the vision really reflect the program’s dream of the future? That might change. The situational politics might… our contribution to that change might change as well, because we as an organisation, we as a team, we as people might change and we might be able to do other things better than we’re doing right now… And we contribute to that change in another way, we cannot assume that we as an organisation might say.

You need to review your boundary partners, so as you go along, you might realize that people you were working with directly are no longer the right ones to be working with. That the people we’re working with directly are no longer influential. But unless we collect information about them as we go along, we might not be able to add or drop them. We need to review these changes as they happen. We have to ask ourselves, was the change process we set down accurate and useful? Do we need to change it? You need to suggest maybe a different direction… it all changes as we go along. Will the strategies match the change in the context… the changes of our partners? For example, is a national conference is the most relevant thing to do, when it seems that these actors we wanted to get together already got together and it happened because of somebody else’s actions. Those stakeholders that we wanted to get together, got together, so is this event that we have planned for four months the most relevant thing to do? Should we actually try to do something else? If we collect information about those actors we would know they already got together. We would know they were already working together, they would have set up a network, they would have a some type of understanding so we wouldn’t do that.

If DFID, DFID in this case, but in any case, if a donor asks why were you going to change it? Why were you going to reallocate the money? Why reallocate the money, because this is how the actors we were working, how the situation changed, and the evidence I used to make my decisions.

Finally are we doing everything we need to do, so it’s really about having a plan, clearly, a clear plan and then we can do the same things we do to… if I use that information. There are examples of how OM is being used, and these applications can be found on the OM website, www.outcomemapping.ca.

I want to talk about uses. Like I said, it’s used in planning, monitoring & evaluation. We think it’s most useful in planning. We’ve stolen some ideas from the monitoring part, the journal especially.

How do we use it? We use it mostly by stealth. We found that outcome mapping is very difficult to implement. We really need people convinced that this is a good methodology. For instance, VSO has used this for its whole organisation. We also tried it in ODI. We tried first to, by having a… I think it was a one-day workshop. Ben Ramalingam and I went to a training which was a five-day training, and we tried to do it one day, so that didn’t work. Then we said, ‘okay, let’s do each step one time so let’s do it now, we’ll have a staff meeting and then we’ll do it again in two months and then we’ll do the next step in two months. That didn’t work either because each time we did it people had to try to remember what we had talked about two months before. People were asking: What was that behaviour? Why are we doing this? This vision is too idealistic. So every time we had to get people into this new mindset and it sometimes requires mindset, awareness. Think about what awareness leads to. So the policy maker is aware of research, what does that mean? Understand it. What does that lead to? They commission research using our work, that’s a change. And that takes a while for people to get. It’s very hard.

So any way, we tried it that way and it didn’t work. And the I think what we’ve done is… just incorporating some of the principles. So we incorporated some of the key principles. The idea of behaviour, we want to talk about changes in behaviour so we don’t want to know just outputs, but we want to know about outcomes and the outcomes is really what is important. We also tried to incorporate the idea of boundary partners. So, let’s focus on those that we are working with directly and not just absolutely everything. And we’ve also incorporated the idea of progress markers… Starting to think about change in a progressive, step by step way. And finally we have monitoring journals. You know we do this, we try to do it, but every time you hear somebody saying, I read your work and I thought it was interesting, send an email to an impact log, systematically. Every time you see a reference to your work. And do the same with activities—back to office reports, after action reviews. I did this meeting, and I collect information about what you’re doing and the effect it’s having. How your activities are influencing those outcomes and have a look at… change.

In terms of communication, we thought we’d focus on boundary partners, let’s be strategic about this. I think that reflects the way we work. We work a lot through workshops rather than just producing working papers. We do a lot of our communications through workshops, through our relationship with one or two partners, and even our workshops are never off-the-shelf, but they involve a two-month process of preparation, discussing with a client, then doing the preparation to follow up because we’re looking at them as a boundary partner.

We’re doing this KM journals about learning, about collecting information as we go along. And when we do capacity development work, we use the word narrative. I don’t know if you can see that but this is a workshop we did for DFID. We brought together a team, a department. They identified who they wanted to work with and they worked through a process for a draft strategy. The first thing is about setting your objectives or priorities and this is a bit where this comes up. There is a corporate priority. Some things we don’t really know regardless of what they think, so it’s been set. So, Latin American department needs to influence the World Bank and the Inter-American Development Bank. And so when we identify some key stakeholders, we don’t use the word boundary partners, we say priority stakeholders.

Once we’ve identified those actors, then we ask ourselves information about the content, the context, you know… we need to know about that. So lots of tools are being used and… And then we… we say we want to influence success criteria. This… we asked the team in the group to consider the behaviours of those actors today, the behaviours of those actors in the ideal future. And then we challenge them to think about those steps. Short medium and long term milestones, behaviour changes. They have an idea of how they think ideally those actors should change in the very near, medium term and long-term future.

Based on those changes we asked them, using a tool called force field analysis... We asked them to think about how are we going to go from now to there… What are the activities you need to carry out to bring about those changes? And that can include communications, negotiations, more research, collaborations, networking, knowledge management—everything we do. And then finally we ask, how are we going to monitor that? How are you going to monitor what you do, how are you going to monitor the activities you do? Were they successful? How were you going to monitor the changes in those actors [cough] and so that’s the same sort of discussions I’ve just made. But at the end of the workshop we have a discussion about capacity… So to do all of this what do we need to do to be able to carry out the strategy but also to monitor. That’s when people say we need more staff. But we don’t have more staff. Consultants? But we don’t have the budget. Then it’s a process of us trying to work with them in a more long-term process.

So, a workshop we’ve used the same principles of outcome mapping and the same narrative in the workshop, but we’re not using except in the case of behaviours, we’re not using any other words that outcome mapping uses because we found that some of those words, some of those concepts, some of them we have incorporated into our everyday work, but it’s difficult to get people to talk in that way.

The strength, and this is my final slide. It takes me to the very end of a long… the main constraint of outcome mapping is that, even though it’s very useful and very easy to incorporate into our work: just start talking about behaviour, start talking about boundary partners, etc. Just start challenging the people you work with: is that really who you work with? We have a lot of workshops where people say: we want to influence law and parliament. Fine, so who do you work with? Um, so I work in a research centre. Okay fine, you want to influence the. Do you work with them? Ok, no, then who works with them? The advisors. Okay, do you work with the advisors? No. Okay, who works with them? The director of our institution. Ok, so you work with the advisors? No, no the director works with them? Ok, but you can talk to the director? (This is an actual case). Well, I need to get an appointment. Right, but you can just walk to his office? No, I need to send an email explaining why we’re… So basically, they need to get in contact with their director first. That’s their boundary partner.

A lot of the time we talk about influencing public opinion. But do you really want to influence public opinion? I mean, ODI might talk about public opinion but we don’t influence public opinion. We might influence a few journalists and what they write in their newspapers might influence public opinion, but we don’t influence public opinion directly. So let’s not judge ourselves so that we are not influencing public opinion, so that’s the one way we can try to do it. Another thing is our behaviour. Every time somebody says I’m going to do this, okay, but…What kind of behaviour changes is this going to bring out? Is this going to educate the minister and the junior minister? But what does that mean? What is the effect of that indication and that understanding? It’s going to make them know a little bit more about our research. OK, but what does that mean? Oh, they’re going to consult with NGO’s from now on. Now, okay, so I won’t see that happening and that might mean you need to do many more things than just giving them the report and getting them to repeat it back to you.

So, main constraint is time. Trying to incorporate outcome mapping and you want to do the whole thing. Not easy, but it is easy to think about the principles…

Resources: it’s a lot of information. It’s not a small project, you need to prioritize and collect information on absolutely every single boundary partner. You might have four or five. You’re not going to collect every single event that happens. You might want to prioritise.

When there are necessary changes in upward accountability, being able to do this properly you need to get those that are holding you accountable to change the way they think. So for instance, we’re working with a couple of partners for the poverty and trade in Latin America programme. The way I am going to hold them accountable is not by telling them: tell me what you’re going to do and I want to see that happen. I am going to ask, before I pay you the next instalment of the contract I want you to tell me what you learned in the last period. How you used that information to adapt your strategy. So I want to see a document that says lessons learned, what’s useable from that, how it’s being used. We used money, but I’m giving you money and I know that you’re not going to make the same mistakes again and you’ve collected information to help you avoid the same mistakes and to know that you’re being serious about it. Not just doing the activities you said you were going to do so you can get the money and pay the people you’ve already had lined up for the job. So, you’re flexible, you can do whatever you want, but you need to show how you’re learning from your work and the work of others. So I mean, I don’t know if that’s the perfect way of doing it… I can’t think of any other way but it’s trying... trying to change the way we hold people accountable, and outcome mapping suggests that we hold them accountable based on lessons learned.

It requires important changes in the user of the methodology. We need to think about new concepts sometimes, and that might be a problem. It can be used to evaluate ex-post. We used it before, and as an example of a specific change you can see the ODI website… when we see a particular change we see track back, so you would say okay this happened, and what are the small progressive changes that happened before that? So, a minister writes a new piece of legislation that he proposes to parliament. So, what happened before that? The minister had a consultation series...

You could track down that process of change and by looking at that process, then you can assess the role that you play in that and link it with the roles that others play… mostly it’s been changed to focus on a particular big important change and collect more information about that. But really it’s designed for the ex-ante, the planning bit. It would be very easy to collect information about what happened and your contribution to it if you’re collecting information as you go along.