|
The Long Term Impacts of
Project Aid: Evidence from China
Speaker: Dr Martin Ravallion, Senior Research
Manager, Development Research Group, World Bank (view
his presentation)
Discussants: Kate Bird, ODI Research Fellow
and Massimiliano Cali, ODI Research Officer
Chair: Andrew Shepherd, Director of Programmes,
Rural Policy and Governance Group (RPGG), ODI
Wednesday 20th September, 1.30-3.00PM
at ODI
Meeting Report
Remarkably little is known about the long-term impacts of
project aid to lagging poor areas. At this ODI event, Dr Martin
Ravallion, Senior Research Manager in the World Bank's Development
Research Group, will present his recent paper which re-visits
a large World Bank-financed rural development project in southwest
China, ten years after it began and four years after disbursements
ended.
Findings include sizeable gains in mean income during the
disbursement period compared to much smaller longer-term impacts
(although some signs of a lasting reduction in extreme poverty
were evident). The paper also covers the difficulties of assessing
long-term impacts.
Copies of the paper, entitled 'Are There Lasting Impacts
of a Poor-Area Development Project?' will be available
at the meeting.
Abstract in full:
Remarkably little is known about the long-term impacts of
project aid to lagging poor areas. The paper re-visits a large
World Bank-financed rural development project in southwest
China, 10 years after it began and four years after disbursements
ended. Survey data were collected on 2,000 households (in
both project and non-project villages) who had first been
surveyed at project commencement in 1995. A double-difference
estimator is used to assess impacts after balancing the initial
covariates of placement. We find sizeable gains in mean income
during the disbursement period, but much smaller long-term
impacts, although there are signs of a lasting reduction in
extreme poverty. We also highlight the difficulties in assessing
long-term impacts, given spillover effects through spending
displacement and knowledge diffusion. Rapid appraisal methods
do not reveal any impacts, although recall biases may well
be the reason.
Meeting Report
1) Remarkably little is known about the long term impacts
of project aid to lagging poor areas. This paper is a technical
assessment of the impacts of a large World Bank-financed rural
development project in Southwest China that was initiated
in 1995 and on which disbursements ended in 2001. The assessment
is made in the context of China's remarkable record in aggregate
poverty reduction (from poverty rates of 53% in 1980 to 8%
in 2001) but with a persistent problem with geographic disparities
- specifically for resource-poor inland lagging regions;
2) The World Bank-supported Southwest Poverty Reduction Project
(SWP) was a classic integrated rural development project.
It aimed to reduce poverty by targeting poor areas (Guangxi,
Guizhou and Yunan with a total population of some 120m people)
and investing in poor farm households and social services
and rural infrastructure. The four key components of the project
were income generating activities; off-farm employment; social
services and infrastructure and institution building and poverty
monitoring. The US$460m project funds were applied to the
following components: agriculture (43%), infrastructure (17%),
rural enterprise (11%), labour mobility (10%), education (9%)
and health, institution building and project and poverty mapping
(all < 5%);
3) The project did not tackle a number of key structural constraints
to rural development in China. These include the restrictions
on internal migration (the houkou system); the administrative
allocation of land undermining the development of a land market
and the high level of uninsured risk for the poor - resulting
in a high level of precautionary savings;
4) The Government of China traditionally focuses upon village
and small-holder investment programmes as the main direct
intervention for fighting poverty. The project was implemented
through a project office with links to government structures.
The SWP project demonstrated a greater extent of integration
across sectors, higher levels of participation with beneficiary
groups, more resources and a much clearer focus on monitoring
the impact of the intervention - compared with conventional
government-driven approaches in poor area programmes. Government
tends to regard the impact of a project as inevitable if the
implementation process is correct;
5) The assessment was very detailed and involved monitoring
closely (daily income and consumption diaries for individuals,
households and communities with monitoring checks every two
or three weeks) over a ten year period 1996 to 2006 on 2000
households in 100 villages within the project area and 2000
households in 100 villages out-with the project area. Cash
consumption and income was augmented with imputed values for
in-kind benefits. The fact that the baseline assessment was
in 1996 - a year after the start of the project - reflected
a failure to adequately communicate the project start date
between government and the evaluation team;
6) Fundamental to evaluation is the recognition that project
impact is the difference between the relevant outcome indicator
with the project and that without it (essentially evaluation
is a missing data problem - how to compare the actual out-turn
with the counterfactual situation in which the project intervention
did not take place);
7) Methodologically, impact evaluation should balance any
selection bias between the treatment group (100 villages in
the project area) and the comparison group outside the project
area that is not a result of the project itself. For instance,
local political economy and knowledge may result in the spill-over
of benefits from the project areas into the comparison group.
As long as any bias is additive and time-invariant, 'diff-in-diff'
will work. However diff-in-diff hides the true impact when
targeted areas have lower growth prospects than the comparison
group - which is obviously a serious issue with projects that
specifically target lagging areas;
8) During the disbursement period, from 1995 to 2000, the
SWP project resulted in a sizeable but transient impact on
income in the treatment area. Consumption was not significantly
affected - due to very high levels of household saving. Analysis
indicates that this saving was precautionary and there is
no evidence that the project had changed the returns to savings
and that the project area was entering a new growth trajectory
(the 'big push' or virtuous cycle growth scenario). The project
did not change agricultural productivity significantly and
there was some evidence of asset value increases in assets
such as livestock and housing. There was a large initial impact
on headcount poverty, as measured by income. However, consumption
changes were mainly observed in the middle-income groups.
There was no sign of any significant impact on overall perceived
standard of living or satisfaction indices. The evidence from
self-perception assessments indicated that much too much weight
is given to current living standards - compared with the project
baseline scenario;
9) There is evidence of a quite significant (about 40%) displacement
of non-project funds from the project area to non-project
areas during the project implementation period. It is also
possible that non-project villages also gained knowledge as
a result of the project. These 'spill-over' effects would
have blurred the impact of the project itself.
Discussion
The discussion clustered around two sets of issues:
Evaluation methods; and
Implication regarding types of project aid.
Evaluation methods
Questions were raised about:
What can be concluded about on the use of qualitative methods
alongside quantitative assessments?
Do income and consumption diaries yield systematic under
or over-reporting?
Could the project result in more benign long-term impacts
where, for instance, savings function to smooth erratic fluctuations
in income?
Could trade between project and non-project areas cloud
the impact of the project?
If the results of project impacts from qualitative self-assessments
were from the same households as those painstakingly filling
in income and consumption diaries over extended periods -
the recall problems of self-assessment may be even bigger
than this study suggests?
Was the high level of saving simply reflecting the lumpy
nature of assets in the project area?
Why were community organisations excluded from the evaluation
process?
In response the Speaker noted:
Generally a mixed methods approach is appropriate in evaluations,
what this exercise has revealed is the dangers of using
qualitative methods in isolation. There is no obvious role
for civil society organisations in an evaluation of this
nature. Generally the use of randomisation methods is not
politically feasible - and may be ethically questionable.
The stimulation to trade, through increasing demand in the
project area, may equalise the benefits between the project
and non-project areas. The key tool to identifying good
comparison areas is a good quality poverty map. The lumpy
nature of investments may, indeed, account for the very
high levels of savings observed in this evaluation.
Implications for this type of project aid
Questions were raised about:
The extent to which the lack of impact generated by the
SWP project reflected its 'enclave' status - with a project
office that was separate from government structures?
Do these results not indicate that project aid of this
type works well in fast-growing local economies, but has a
much smaller marginal impact in slower-growing areas?
Can these findings for a very large project inform evaluations
of the much more common, smaller, projects?
Is it possible to move beyond the Sachs-Easterly dichotomy
about whether aid works or not and instead focus upon a more
nuanced analysis that recognises the importance of local institutions
in making external interventions work - or not?
In response the Speaker noted:
The high standard errors and high levels of variance imply
very differential project impacts in each local area - which
may well support the case that aid effectiveness can be
attributed to the quality of local institutions. A key blockage
in rural China is the policy framework that prevents the
development of a land market - to avoid devaluing the main
non-labour asset of poor rural people. Also the population
registration system, by reducing internal migration, has
a significantly negative impact on poor rural areas. Reform
in these two areas would have a much greater impact than
external development assistance.
(view
Martin Ravallion's presentation)
 |
