Country-led M&E Systems: Better Evidence, Better Policies, Better Development Results

"When we analyze the trends in evaluation worldwide, it is no surprise to see that the traditional and current evaluation practices in the developing world are mainly top-down methodologies, introduced through models with different aid modalities. They are therefore designed and conducted to respond primarily to aid effectiveness. It is also no surprise to observe that evaluation thinking is evolving at a moment when development paradigms are changing priorities and introduce the principles of ownership and mutual accountability."

- From the Preface by Oumoul Khayri Ba Tall, President, International Organization for Cooperation in Evaluation

This 322-page publication seeks to contribute to the debate on country-led monitoring and evaluation systems (CLES) and their ability to enhance evidence-based policy making. It is a publication of the United Nations Children's Fund (UNICEF), in partnership with the World Bank, the United Nations (UN) Economic Commission for Europe, International Development Evaluation Association (IDEAS), International Organization for Cooperation in Evaluation (IOCE), DevInfo, and Multiple Indicator Cluster Surveys (MICS). The publication brings together the vision, lessons learned, and good practices from different stakeholders, who reflect on how CLES can facilitate the availability of evidence relevant to country-specific data needs. Contributors to the volume acknowledge that data are important in monitoring policy reforms and national development goals, while at the same time, share a commitment to ensuring technical rigour through monitoring and evaluation (M&E) capacity development. A thread woven throughout the contributions to this volume is the recognition that, despite its potential, CLES is vulnerable to various challenges, such as bridging the gap between policy-makers (the users of evidence) and statisticians, evaluators, and researchers (the providers of evidence).

Contributors to Part 1 of the resource examine the purpose of CLES. Namely,

• Segone introduces the concept and dynamics of evidence-based policy making, underlining that the main challenge is matching technical rigour with policy relevance. Good evidence has to be technically sound - that is, trustworthy and of high quality - as well as policy-relevant. Country-led evaluations (CLE) are evaluations in which the country which is directly concerned leads and owns the evaluation process by determining: what policy or programme will be evaluated; what evaluation questions will be asked; what methods will be used; what analytical approach will be undertaken; and how the findings will be communicated and ultimately used. He argues that CLE can serve the information needs of the country and, therefore, can be an instrumental agent of change and can be crucial in supporting national development results. Yet, challenges remain in implementing CLES; Segone proposes a way forward.
• Picciotto acknowledges the increasing amount of evaluation of development activities at the country level, explaining why the shift took place. In addition, he analyses what the new orientation implies for aid management, and what challenges it creates for evaluation methods and practices. Finally, Picciotto assesses whether a country-based approach to development evaluation will remain relevant, given the spread of multicountry collaborative development programmes.
• Quesnel explains how an understanding of "strategic intent" is a prerequisite for any relevant and efficient CLES. The strategic intent makes explicit the aim of the developmental intervention being pursued and provides coherence to country efforts and external support. It fosters greater effectiveness of the scenario being implemented and facilitates the measurement of achievements. Quesnel presents a definition of strategic intent and illustrates various applications of the strategic intent at different levels of management, using different results-based paradigms.
• Lundgren and Kennedy describe some of the opportunities and challenges in promoting partner country leadership in development evaluation. In the context of the aid effectiveness agenda, the authors provide an overview of donor efforts to promote joint and partner-led evaluations; support evaluation capacity development; disseminate evaluation standards and resources; and better align and harmonise aid evaluation. They also explore: the challenge of balancing the evaluation needs of the donor, partner, and beneficiary; the need to integrate aid evaluation into partner governance and management systems; and the limitations posed by the lack of an enabling environment for evaluation in many contexts.
• Feinstein analyses a CLE experience, presents a rationale and vision for CLE, and assesses opportunities, achievements, and lessons learned. He explains why the experience so far with CLE has been mixed, if not disappointing. Finally, he concludes by proposing a wider approach which shifts the focus from a specific type of evaluation to CLE systems which generate CLE as products.
• Adrien and Jobin suggest that CLE directly impacts 3 components of good governance: voice, accountability, and the control of corruption. The authors analyse a specific type of CLE: country-led impact evaluations (CLIE), introducing a discussion on impact evaluation and presenting the results of a survey on impact evaluation. Finally, they present the challenges ahead, based on the debate generated at a recent conference on "Evaluation under a Managing-for-development Results Environment."
• Khayri Ba Tall analyses the role of national, regional, and global evaluation organisations in strengthening CLES. She gives an overview of the evaluation networks worldwide, and elaborates on the different functions of evaluation. Finally, she proposes some strategies to strengthen CLES, such as creating a domestic demand for evaluation; extending the evaluation object and scope beyond aid; and improving the supply side through evaluation capacity development.
• Giovannini investigates how citizens see and evaluate official statistics and the role played by the media in this respect, using empirical evidence concerning several OECD countries. Giovannini argues that the value added of official statistics depends on its capacity for creating knowledge in the whole society, not only among policymakers. He draws some conclusions about the need to transform statistical offices from "information providers" to "knowledge builders" for the sake of democracy and good policy.
• Baer argues that understanding customers, marketing, and building relationships are not just side functions or minor activities but, rather, they are closely linked with the reputation, future role, and viability of statistical agencies. For example, the relative importance of each potential user group must be decided before developing a dissemination strategy.

Part II of the volume explores good practices in CLES. Specifically,

• Mackay examines the various ways in which M&E systems can, and are, used to improve government performance. He reviews key trends which are influencing developing countries in building or strengthening existing M&E systems. Mackay also presents ways to raise awareness of the usefulness of M&E creating incentives for its utilisation, and how such incentives can help to create demand for M&E. Finally, he examines the importance of conducting a country diagnosis, to provide a shared understanding of the strengths and weaknesses of existing M&E systems, and to foster a consensus around an action plan for its further strengthening.
• Kusek and Rist present the importance of a strong theory of change. They explain how to successfully build a strong evaluation culture in developing counties and the need for an emphasis on how evaluation can help deliver information and analysis that strengthen programme delivery. They then present the CORAL approach, that is, five questions which need to be answered when thinking through the logic of a programme, or its theory of change. They include: C (what is the concern or concerns most affecting citizens and other stakeholders?) O (what is the outcome or solution sought?) R (what are known or likely risks which will stop the programme being successfully implemented?) A (can key assumptions be tested and measured with information readily available to determine what is, or is not, working?) L (can new programme logic and knowledge be regularly fed back into the programme to revise the design and implementation plan as necessary?)
• Bamberger and Rugh explain how the RealWorld Evaluation (RWE) approach may assist the many evaluators, in developing, transition, and developed countries, who must conduct evaluations within budget, time, data, and political constraints. Determining the most appropriate evaluation design under these kinds of circumstances can be a complicated juggling act, they warn, involving a trade-off between available resources and acceptable standards of evaluation practice. Their concern is that failure to reach satisfactory resolution of this and other trade-offs may lead to low use of evaluation results.
• Segone, Sakvarelidze, and Vadnais examine the contribution of household surveys in general, and the Multiple Indicators Cluster Survey (MICS) in particular, in strengthening CLES. The authors explain how MICS3 was instrumental in enhancing national statistical capacity and quality assurance systems, through national ownership and a technical assistance system. They also present good practices in data dissemination, as well as some examples of how MICS3 data have been used at national, regional, and global level to inform evidence-based policy advocacy and to stimulate further analysis on specific topics, such as child poverty analysis.
• Pron, Oswalt, Segone, and Sakvarelidze argue that to achieve sustainable development outcomes, country-led development strategies must be backed by adequate financing within the global partnership for development. However, this is only possible if timely evidence is available from policy-relevant and technically-reliable CLES. The evidence provided by such systems, owned by developing and transition countries, should inform necessary policies and strategies to ensure progress. The authors explore how DevInfo - a data dissemination system which the UN offers to countries - was designed to facilitate ownership by national authorities and is being used by hundreds of countries worldwide within national and decentralised M&E systems. Selected good practices from Belarus, Moldova, Kyrgyzstan, Serbia, and Tajikistan - among others - are presented.
  

Finally, beginning on page 277 is "Making Data Meaningful: A Guide to Writing Stories about Numbers," which was prepared within the framework of the United Nations Economic Commission for Europe (UNECE) Work Session on Statistical Dissemination and Communication. The guide is intended as a practical tool to help managers, statisticians, and media relations officers use text, tables, graphics, and other information to bring statistics to life using effective writing techniques. It contains suggestions, guidelines, and examples - recognising that there are many practical and cultural differences among statistical offices, and that approaches vary from country to country.