Policy Pilots

What is the tool?
Policy piloting is an important tool in policy development and delivery and the identification of what works. This is an important recent innovation that allows for the phased introduction of major government policies or programmes. This means that policies can be tested, evaluated and adjusted where necessary, before being rolled out nationally. A recent review of pilots by the UK Strategy Unit (2003) identified the existence of a number of different types of pilots including impact pilots, process pilots and phased implementation projects.

When should you use it?
The Cabinet Office report recommended that 'the full-scale introduction of new policies and delivery mechanisms should, wherever possible, be preceded by closely monitored pilots' (Cabinet Office, 2003). Where pilots are used to test policies it is important that they are completed and that lessons are learned before more widespread implementation. The Cabinet Office Review of Pilots recommended that: once embarked upon, a pilot must be allowed to run its course. Notwithstanding the familiar pressures of government timetables, the full benefits of a policy pilot will not be realised if the policy is rolled out before the results of the pilot have been absorbed and acted upon. Early results may give a misleading picture (Cabinet Office, 2003: Recommendation 6).

How it works?
Pilots are used to test policies before they are broadly or fully implemented. There are a number of key considerations about how pilots should be used in policymaking. Some of these are highlighted below:

• The role of pilots
  
  • Pilots are an important first stage
    
  • Pilots should be used to try innovations that might otherwise be too costly or risky to embark on
    
  • The scale and complexity of any experimental treatment should be proportionate to its likely utility
    
• Pre-conditions
  
  • Once embarked on, a pilot must be allowed to run its course
    
  • Pilots should be preceded by the systematic gathering of evidence
    
  • The purpose of the pilot should be made explicit in advance so that its methods and timetable are framed accordingly
    
• Key properties
  
  • Independence - there should not be pressure to produce 'good news'
    
  • Budgets and timetables should allow for the adequate training of staff to avoid systematic errors
    
  • Provision for interim findings should be made (accompanied by warnings) as it is not always possible to carry out lengthy pilots before policies are introduced
    
• Methods and practices in pilots
  
  • There is no single best method of piloting a policy. Multiple methods of measurement and assessment - including experimental, quasi-experimental and qualitative techniques - should all be considered to get a complete picture
    
  • For policies designed to achieve change in individual behaviour or outcomes, randomised control trials offer the most conclusive test of their likely impact
    
  • For policies designed to achieve change at an area, unit or service level, randomised area or service-based trials offer the most conclusive test of impact
    
• The use of results
  
  • Pilots which reveal policy flaws should be viewed as a success, not a failure
    
  • Appropriate mechanisms should be in place to adapt or abandon a policy or its delivery mechanisms in light of a pilot's findings
    
  • Pilot reports should be made easily accessible to facilitate easy reference of past successes and failures

Example

Earnings Top-Up (ETU) (Department for Work and Pensions)
The ETU pilots assessed the effectiveness of in-work benefits for low-income workers without dependent children, and of improving the lowest-paid workers' chances of getting employment and keeping it. Lessons drawn from the project contributed to a better design of the ETU, including improving take-up and eligibility criteria, information on the significance of advertising the scheme and the role of informal networks in spreading information, and lessons about the interrelationship with other policy areas.

For more information

• 'Trying it Out: The Role of 'Pilots' in Policy Making', Strategy Unit, 2003 [external website pdf]. The Government Chief Social Researcher's Office carried out a review of government pilots. The review was chaired by Professor Roger Jowell, City University, and supported by a panel of senior figures from inside and outside of government. The report is intended to stimulate debate on the use of pilots in policy development, and to provide guidance on the effective use of pilots across government.
• Rondinelli, D. (1993) Development Projects as Policy Experiments: an Adaptive Approach to Development Administration, London: Routledge. Rondinelli suggests that policymakers should look at a number of other projects, as well as pilots.
• The UK Department of Work and Pensions is undertaking a randomised controlled trial (the ERA Project) of three policy initiatives aimed at retaining and advancing in the labour market those people on the lower margins of the workforce. For more information see Morris, S. et al. (2004) Designing a Demonstration Project: an Employment, Retention and Advancement Demonstration for Great Britain, Government Chief Social Researcher's Office, Occasional Paper No. 1 (2nd Edition) London: Cabinet Office.

Sources

• Policy hub website: Reviewing the use of Government Policy Pilots [external website]
  
• Davies, P. (2004) 'Is Evidence-based Government Possible?' Jerry Lee Lecture, presented at the 4th Annual Campbell Collaboration Colloquium, Washington DC.
  
• Strategy Unit (2003) 'Trying it Out: the Role of 'Pilots' in Policymaking: Report of a Review of Government Pilots' [external website]
  

• View this tool as pdf ( 47kb)

Back to Evidence-based tools index or toolkit introduction