Impact Evaluation Influencing Policy: FAO and Social Protection

Pamela Pozarny, FAO
Senior Rural Sociologist and Rural Poverty Reduction Regional Focal Point for Africa
Regional Office for Africa, Accra, Ghana

Silvio Daidone, FAO
Economist
Rome, Italy

Bringing evidence and informed advice to national policy debate is fundamental to effect development and change, but this is not a straightforward linear process, nor is it a question of mere technical expertise. The global playing field and national policy-making landscapes are not clean slates. These political and developmental processes are dense with ongoing negotiations over issues keenly debated and vigorously contested by multiple stakeholder groups. This dynamic web of interests comprises the political economy environment and is the decisive platform in which the interests and priorities of a nation are examined and debated.  Multiple influences, opinions, and sources shape policy decisions, and evidence is just one contribution among many.[1]

The Food and Agriculture Organization (FAO) of the United Nations is a multilateral technical agency with the mandate to work towards achieving the eradication of hunger and poverty, and global food security and healthy and balanced nutrition for all. A core function of the FAO is evidence generation.  To “inform” is enshrined in the FAO constitution: to generate, analyse, interpret and disseminate differentiated and high-quality information and evidence in the areas of its mandate. FAO communicates unbiased technical and statistical information to government policymakers and national stakeholders, enabling them to reach informed policy and programme decisions toward meeting national objectives and priorities including the Sustainable Development Goals (SDGs). FAO functions as a neutral broker assuring its member states high quality expertise and advice based on rigorous and robust evidence with pragmatic viability. As a global point of reference and centre of excellence for issues pertaining to food security, agriculture and nutrition, FAO maintains a reputation of credibility and trustworthiness of information and guidance. Evidence-based policy and programme assistance is central to the work of FAO, and delivering often highly technical evidence through understandable, adaptive, engaging messages is vital to FAO’s success (FAO, 2017).

The Transfer Project is a joint research-learning initiative that strengthens knowledge and practice of social cash transfers in Sub-Sahara Africa to: (i) gain further understanding of the impacts of cash transfers, including how and why impacts are achieved; (ii) garner lessons from the diversity of thematic and operational features of programmes across the region; (iii) promote regional learning and exchange among policy makers, implementers, researchers, and civil society – all towards informing policy. The project comprises FAO, the United Nations Children’s Fund (UNICEF), Save the Children, and the University of North Carolina at Chapel Hill, in partnership with national governments, and national and international researchers.

Under the Transfer Project umbrella, the FAO-led project From Protection to Production (PtoP) was a collaborative research activity including FAO, UNICEF, and national governments, examining linkages among and impacts of social protection, agriculture, and rural development. The mixed-method research included assessing household productive investments, household economy, resilience and inclusion, effects on consumption patterns, and household decision-making, as well as spillovers in the local economy. The overall objective was to understand these linkages to inform policy and specifically to inform and enhance coordination between social and the productive sectors (Davis et al., 2016, 6). Initially supported by UK Department of International Development (DFID), with contributions from European Union (EU), the PtoP project has massively expanded into a wider social protection agenda mainstreamed into FAO’s corporate strategic programme on rural poverty reduction.

The landmark Transfer Project publication, From Evidence to Action: The Story of Cash Transfer and Impact Evaluation Sub-Saharan Africa (Davis et al, 2016) is a synthesis of results of the impacts of cash transfer programmes in the PtoP countries of research, which prominently examines and analyses the drivers, and extent and reasons in which impact evaluation evidence affected cash transfer policy and programme processes in the different countries or not. The flagship publication sheds light and offers key lessons and orientations toward strengthening the potential to influence the political economy of cash transfer programming as a mechanism to address poverty alleviation in Africa (Davis et al, 2016: 2). The book includes eight case studies of country experience drawing on the role impact evaluation evidence played in the cash transfer policy and programming process, leading to programme expansion in Lesotho, Kenya and Zambia; an increase in transfer value in Ghana and in Malawi; changes in budget allocation and government taking-over of the programme in Lesotho; revising programme design and interventions concerning for example multi-sectoral coordination in Ethiopia; targeting in Zimbabwe; and institutional strengthening in Kenya.

The FAO-led project From Protection to Production (PtoP) was a collaborative research activity including FAO, UNICEF, and national governments.  It aimed to examine interactions and impacts of cash transfer programmes on agriculture, rural livelihoods and rural development across seven countries in sub-Saharan Africa. The research assessed household productive investments, household economy, resilience and inclusion, effects on consumption patterns, household decision-making, and spillovers in the local economy. The overall objective was to understand these linkages to inform policy of the importance of multidimensional coordination among sectors[2]. The project demonstrated how important political economy analysis is in enabling evidence to influence policy and programme change.

Evidence-gathering fieldwork

Project evaluations were timely within an environment of prevailing demand, “responding to” or “embedded within” national policy processes or a programme re-thinking in the country.  This required flexibility in the research agenda so that the research was well adapted to national needs.

In some cases, the research truly rode the wave of the policy process. For example, in Lesotho the Rapid Assessment results were critical in early stages of the Child Grants Programme (CGP) to respond to criticism and advocate for roll-out and expansion, while the results about local economy effects were used for advocacy within the Government by the Ministry of Social Development, providing a strategic and solid argument regarding the broader effects of the CGP on increased benefits to the wider community[3]. The strategic importance of evidence generated through the rapid appraisal and local economy study is well documented by the constant referral to findings from these studies in high-level speeches throughout the period.

Similarly, in Ghana the baseline analysis was instrumental in influencing a doubling of the cash transfer amount of the LEAP programme, while the results from the local economy study were used as advocacy tool, notably by Ghana’s President opening the Pan-African conference on Inequalities in April 2014, to demonstrate that social protection and economic growth can be complements.

Evidence-gathering fieldwork

The PtoP project involved and integrated government representatives, and collaborated with national research teams in research projects. For example, in Ghana and Malawi, national universities were fully involved in various phases of the evidence generation agenda, starting from the design of the evaluation and of the survey instruments using different methodologies. They also led the fieldwork, given their deep knowledge of the local context, and contributed significantly to the analytical stage. In the qualitative research, teams were at the heart of fieldwork analysis and synthesis which was conducted through evening team debriefings each night after the day of data collection in the field, to reflect on findings as a group and build a concluding “story” in the field.

High quality, credible and robust research by multidisciplinary mixed methods research teams (including qualitative and quantitative research and local economy-wide simulations) underpinned a triangulation of evidence and strong messaging and packaging presented to national audiences.  Rigor and credibility were of particular importance given the “dependency” myth surrounding cash transfer interventions.[4] For instance, in Zambia the impact evaluation demonstrated both human capital and productive impacts. These results contributed to challenge the perceptions around cash transfers being simple handouts creating dependency. This allowed the Ministry to cite the programme as “an engine for inclusive growth”. Similarly, the multiplier effects calculated with local economy general equilibrium models caught the attention of different audiences in government, especially of the Cabinet Office and the Ministry of Finance.

Fieldwork - evidence gathering
Photo credit: Zahrah Nesbitt-Ahmed

Throughout the project, researchers, government representatives and partners continuously shared research methods, ongoing research (including challenges), and results through regular workshops bringing the main stakeholders together. “Close interaction between different stakeholders promoted the development of strong trusted relationships and, most importantly, national ownership of the process and end-results”[5]. Evaluations were not perceived as stand-alone academic exercises, rather they contributed to an evidence-based policy making culture at all levels (national and regional),

Social protection and cash transfer programmes in specific in Africa are expanding, but within a precarious landscape.  Given the divergent and sometimes formidable views concerning how to best address poverty eradication and hunger, and given existing fiscal constraints, trade-offs are always required. This gives the political economy arena around social protection great importance.  If positioned and communicated appropriately, impact evidence can potentially become a powerful tool of influence.[6],[7] Given the constraints of limited national budgets, research evidence which teases out performance factors can also contribute to programme revisions to maximize value for money.


Footnotes

[1] McCord et al. (2016). The Political Economy of Cash Transfer Evaluations in Sub-Saharan Africa. In Davis et al.  From Evidence to Action: The Story of Cash Transfer and Impact Evaluation Sub-Saharan Africa. Oxford University Press: London, UK. 17-42.

[2] Davis et al (2016). The Transfer Project, Cash Transfer and Impact Evaluations in Sub-Saharan Africa. In Davis et al.  From Evidence to Action: The Story of Cash Transfer and Impact Evaluation Sub-Saharan Africa. Oxford University Press: London, UK. 1-16.

[3] Pellerano et al (2016) ‘’Does Evidence Matter? Role of the Evaluation of the Child Grants Programme in the  consolidation of the Social Protection Sector in Lesotho’’  in Davis et al. (2016)

[4] Handa et al. (2018) Myth-busting? Confronting Six Common Perceptions about Unconditional Cash Transfers as a Poverty Reduction Strategy in Africa. World Bank Research Observer, forthcoming

[5] Davis et al. (2016)

[6] Barrientos Armando and Sony Pellissery (2012). Delivering effective social assistance: Does politics matter? Effective States and Inclusive Development Centre (EISD), University of Manchester, UK.

[7] Hickey, S. 2008. Conceptualizing the politics of social protection in Africa. In: Barrientos, A. and Hulme, D. (eds.) Social Protection for the Poor and Poorest: Concepts, Policies and Politics. London: Palgrave.