|
Key Issues Surrounding EBP
Discussion of EBP tends to cluster around three key issues. Firstly,
we focus on the kinds of evidence which are used and its credibility.
Second are issues surrounding the way in which evidence is incorporated
into the policymaking process. Finally, we highlight the issue that
many factors other than evidence affect the way policy is made;
policymaking is inherently a political process. Click on the links
below for information on each issue:
Evidence for policy processes
Central to making evidence more accessible to policymakers is the
need to go to the very basis of what is meant by the term 'evidence'
and explore how this can, should and does affect policymaking. Evidence
tends to be portrayed as an a-political, neutral and objective policy
tool. This image is encouraged by the Labour government's claim
that policy will now be shaped by evidence; thereby implying that
the era of ideologically driven politics is over (Nutley, 2003:
3). It is however neither neutral nor uncontested; instead evidence
is a fundamentally ambiguous term.
(i) Different types of evidence
What counts as evidence varies as much from the researchers working
practice to the end use of the evidence. The UK Cabinet Office attempted
to define its understanding of evidence in its 1999 White Paper
Modernising Government, according to which evidence is 'expert knowledge;
published research; existing research; stakeholder consultations;
previous policy evaluations; the Internet; outcomes from consultations;
costings of policy options; output from economic and statistical
modelling' (Cabinet Office, 1999: 33). The breadth of what is considered
evidence is therefore wide and dynamic (Shaxson, 2005). Marston
and Watts (2003) supported this interpretation, listing a rich and
varied variety. Possible sources include photographs, literary texts,
official files, autobiographical material such as diaries and letters,
newspaper files and ethnographic and particular observer accounts.
We take the view that evidence-based policy should be based on systematic
evidence (both the hard and soft evidence types highlighted above).
Therefore we believe that evidence-based policy should be based
on research based evidence. The key however is that we adopt a very
general, though widely accepted, definition of research as 'any
systematic effort to increase the stock of knowledge' (OECD, 1981).
Thus we include all kinds of evidence as long as they have been
collected through a systematic process. This may include any systematic
process of critical investigation and evaluation, theory building,
data collection, analysis and codification related to development
policy and practice. It also includes action research, i.e. self-reflection
by practitioners orientated towards the enhancement of direct practice.
(ii) Hierarchy of evidence
Despite this broad eclectic definition of evidence, it would be
a mistake to assume that in reality all forms of evidence share
equal importance, relevance or weighting. Departments and units
within the government tend to make hierarchical judgements in choosing
what evidence to use, where and how - these decisions are often
deeply embedded in assumptions over validity and power. For example,
UK public sector policy relies on a limited range of 'top-end' evidence,
centring on empirical research, policy evaluation and expert knowledge,
and thereby creates an implicit hierarchy. Government departments
are not alone in their preferences towards empirical research, viewing
it as the most reliable form of evidence.
Marston and Watts (2003: 151) categorised evidence through a social
sciences perspective as either 'hard' or 'soft', implying objective
versus subjective forms. Hard evidence is said to consist of: primary
quantitative data collected by researchers from experiments; secondary
quantitative social and epidemiological data collected by government
agencies; clinical trials; and interview or questionnaire-based
social surveys. This is in contrast to 'soft' evidence which is
viewed as consisting of qualitative data such as ethnographic accounts
and autobiographical materials. Upsur et al (2001: 91) argue that
qualitative methodologies receive little attention in the social
sciences. Such categorisation as prescribed in Marston and Watts
(2003) lead to risks and failures with the EBP approach. This because
a hierarchy which is weighted in favour of hard evidence, creates
a risk and a limitation that EBP will in fact be used to ignore
evidence that comes low in the hierarchy. This would include tacit
forms of knowledge, practice-based wisdom and, perhaps most importantly,
the voices of ordinary citizens - the 'voices of the poor'. The
implication is therefore that an EBP approach should take into consideration
a wide breadth of sources of research, not just hard evidence. This
understanding continues to embrace a wide range of factors, such
as voice and consultations, if the evidence is collected through
a systematic process. Although it is this idea which we promote,
for ease of use the following discussion shortens the term 'research
based evidence' to 'evidence'.
(iii) Attempts to establish what evidence is useful to policymakers
As not all forms of evidence share an equal validity or weighting,
when the government attempts to create a broad understanding of
what evidence is useful, there are inevitable questions. Therefore,
in order to come to some agreement over what constitutes useful
evidence, we highlight the work of Louise Shaxson who identifies
some of the key characteristics of evidence that influence whether
it is used (Shaxson, 2005: 102). We also draw on the work of the
RAPID programme at ODI that has been focusing on these issues in
developing countries. The key issues are outlined below:
1. Quality / accuracy / objectivity
This refers to the accuracy of the evidence. Is the evidence correctly
describing what it purports to do? There are arithmetical (are the
numbers added up correctly?), statistical (were the cause and effect
parameter correctly specified?) and representative (do the quotes
from people really represent what the body of people felt?) issues
that can be considered to address the relevance of evidence. There
are also issues surrounding the objectivity of the evidence and
its sources. It is important to question the bias in the evidence
base to deepen our understanding of how it conditions our interpretation
of the evidence for policy (Shaxson, 2005: 107).
2. Credibility
Credible evidence relies on a strong and clear line of argument;
tried and tested analytical methods; analytical rigour throughout
the processes of data collection and analysis; and on clear presentation
of the conclusions (Shaxson, 2005: 106). This relates to the reliability
of the evidence and therefore whether or not we can depend on the
evidence for monitoring, evaluation or impact assessments - planning
for the lessons learned approach. In reality, it is very difficult
for policymakers to check evidence, therefore they often rely on
the reputation of the source as a proxy. For example, research and
research findings from academic institutions and reputable think-tanks
tend to be viewed as more credible than, for instance, civil society
groups or community leaders opinions. Reputation, however, is subjective
and depends on the decision maker. Advice on employment morale will
be more easily accepted from a highly reputable human resources
consulting firm than from a small NGO.
3. Relevance
The key issue here is that evidence is timely, topical and has policy
implications. The type of evidence one refers to matters greatly
according to the audience it is being presented to and the likely
impact it can create. For example, policymakers in the public sector
would be more interested in evidence of action (what has already
happened) rather than opinion. This is supported by the findings
of Moseley and Tierney (2004: 114) when investigating the problems
relating to the implementation of EBP. Implicit to the idea of relevance
is the need for the evidence to be generalisable. This refers to
whether there is extensive information or just selective case studies
and therefore how easily applicable the argument is. It also relates
to the way in which we make inferences. For some types of information,
generalisability will refer primarily to sampling procedures; for
others, it will be more about our understanding of context. It is
particularly applicable when pilot studies precede a wider roll-out
(Shaxson, 2005: 106).
4. Practicalities
This relates to the extent to which the evidence is accessible to
policymakers; whether policymakers have access to it in a useful
form and therefore the ease with which it can be translated into
policy. It also refers to the cost of the policy implications of
the research, and therefore whether it is feasible and affordable.
 Policy
stages and the use of evidence
Following Lasswell (1977), the most common approach to the study
of public policy disaggregates the process into a number of functional
components. A conceptual model of the policy cycle is shown visually
in the diagram (figure 2). It is important to emphasise that policy
processes are never as linear, or cyclical, as implied in the model.
But, looking at the policy process in terms of these stages or functional
elements can help us to understand how this process does (or should)
work.
This conceptualisation is important since it demonstrates that
research has the potential to influence the process at any stage
- both informing and correcting planning and implementation, for
instance. Thus this view of policy processes also helps break down
the policy cycle to try to identify the different types of research
or evidence that might be needed. It may well be that success in
influencing an agenda, for example, often requires a different kind
of approach than that needed for influencing the implementation
of policy. The value of this view of the policy process is that
it is not tied to a particular set of institutions, thus enabling
the analysis of a range of actors (not just government) and the
way they interact across policy issue, component of the process
and time.
For the purposes of this paper, the functions of the policy processes
are simplified into four categories:
- Agenda Setting: awareness of and priority given to an
issue or problem;
- Policy Formulation: the ways (analytical and political)
options and strategies are constructed;
- Policy Implementation: the forms and nature of policy
administration and activities on the ground;
- Monitoring and Policy Evaluation: the nature of monitoring
and evaluation of policy need, design, implementation and impact.
 For
each different part of the policy process, we revise the work of
Pollard and Court
(2005) to outline some specific issues regarding use of evidence.
Figure 3 provides a generic characterisation of the flow of evidence
in the policy process. It is a simplistic diagram, but it also provides
important insights. First, it provides a similar distinction between
agenda setting, formulation and monitoring and the different evidence-collection
processes needed. Secondly, it clearly makes the distinction regarding
the different time constraints between evidence needs for pressing
policy questions and those for longer term strategic policy objectives.
The implication is that different types of evidence are often needed
for different parts of the policy process and that time considerations
are likely to influence the mechanisms available to mobilise evidence.
Politics
'The good news is that evidence can matter. The bad news is
that it often does not', Julius Court speaking on 'The
political context in developing countries', at Does Evidence Matter?
ODI meeting series.
So far we have looked at issues of evidence and policy in a technical
sense. However, policymaking is inherently a political process.
Many factors jostle with evidence to take centre stage in policy
formation both at an individual level and at an organisational level.
Davies (2004: 4-7) describes seven major factors other than evidence,
which inform and influence policymaking (see figure 4).
 |
- Experience, Expertise and Judgement - Human and
intellectual capital, tacit knowledge;
- Resources - Policymaking and implementation occurs
in the context of finite (usually declining) resources,
indicating some kind of cost-benefit exercise;
- Values - These include ideological and political
beliefs. Values are strong driving forces behind policymaking
and often influence the end result;
- Habit and Tradition - Important features which
often defy rational explanation in the twenty-first century;
- Lobbyists, Pressure Groups and Consultants - This
category also includes think tanks, opinion leaders and
the media, all of whom are major influencing powers;
- Pragmatics and Contingencies - of political life,
such as timetables, parliamentary terms, capacities of institutions
and unanticipated contingencies. Although these factors
do not necessarily stand against EBP in principle, they
do not compliment the strategic EBP approach.
|
Weiss (1977) offers an alternative way of categorising the factors
which policy and practice depend on. She puts forward the four I's;
information, interests, ideologies and institutions:
- Information - 'the range of knowledge and ideas that
help people make sense of the current state of affairs, why things
happen as they do, and which new initiatives will help or hinder'
- Interests - i.e. 'self-interests'
- Ideologies - 'philosophies, principles, values and political
orientation'
- Institutions - 'first the institutional environment shapes
the way in which participants interpret their own interests, ideologies,
and information. […] Second, organisational arrangements
affect the decision process itself, such as who is empowered to
make decisions.'
Over time, the four I's interact with each other in a dynamic manner.
While information - and research is included here - does matter,
it is also clear that there are other major issues that impact on
policy. Shaxson (2005: 102) supports this conclusion, arguing that
evidence is a necessary, but not a sufficient, condition for any
decision-making process.
Moseley and Tierney (2004: 115) highlight that EBP is not meant
to be taken on its own. It is only one resource available to practitioners
which should be used with others.
In the realities of the political world, the value assigned to
research is less than prevailing thought or opinion. There are those
in the political world who will perpetually view research 'as the
opposite of action rather than the opposite of ignorance.' (Court
(2004) speaking at ODI Meeting series). Nutley (2003: 12) highlights
the fact that the interaction between policymakers and researchers
is limited by the divergence of these two worlds. They use different
languages, have different priorities, different agendas, different
time scales and different reward systems. Consequentially a communication
gap exists.
Evidence therefore has a tough role to play if it is to gain wider
credibility amongst decision makers. Challenging assumptions and
value systems is a long term, and often difficult, process. Despite
the move towards a more rational method of decision-making practices,
value judgments are often made based upon assumptions. Politics
is implicit in the sphere of policymaking and therefore political
agendas play a key role. Politicians often argue that, to an extraordinary
degree, the political world merely pays lip service to evidence
and research (Vincent
Cable speaking on 'Evidence and UK Politics', at Does Evidence Matter?
ODI meeting series)
Time constraints and the resultant pressure should feature as a
stand-alone factor. Pressure refers to the short-term need to respond
to external demands from senior managers or politicians, regulators
and advocacy groups. Policymakers often need to advise ministers
on pressing issues; this pressure can result in a delay in the use
of EBP approaches, while assumptions, personal or institutional
values form the forefront of knowledge. Consequentially current
knowledge needs to be very quickly synthesised to inform a decision.
However, there is the danger that due to time pressures, evidence
which lacks credibility and accuracy is produced. As a result one
could argue that it would be better to wait until the end of the
peer review process to consider the evidence. The difficulty with
this is that policy may have leapt forward too far and therefore
the evidence would no longer be relevant. Perhaps, therefore, it
would be better to have timely and policy relevant findings, albeit
with a number of caveats and uncertainties.
Back to project index
|
