Researching, Monitoring and Evaluating Communication for Development: Trends, Challenges and Approaches

Prepared for the United Nations Inter-agency Group on Communication for Development as part of the United Nations Inter-agency Resource Pack on Research, Monitoring and Evaluation in Communication for Development, this report highlights a number of trends, challenges, and approaches associated with researching, monitoring, and evaluating communication for development (C4D) within the United Nations (UN) context. It has a particular focus on impact assessment, in light of an observed need to increase the number and quality of impact assessments of C4D programmes.

To prepare this report, the authors undertook a literature review and consultations with 11 C4D focal points or monitoring and evaluation (M&E) specialists from 7 UN agencies and a 15-member Expert Panel. Based on this research, the report details a number of principles for effective, appropriate, and sustainable R (researching), M&E of C4D. In summary, principles for effective, appropriate, and sustainable R, M&E of C4D include:

• Uses a participatory approach that respects, legitimises, contextualises and draws on the knowledge and experience of local participants. Methodologies and methods selected are culturally appropriate, complementary, and the most appropriate for different issues and purposes, different types of C4D initiatives, and the aims of the evaluation.
• Recognising that people (including children) have a right to participate in decisions that affect them, facilitates active participation of stakeholders in all stages of the evaluation and ownership of the initiative and its evaluation and is as inclusive as possible of a diversity of groups - addressing issues of gender, class, race, and unequal power and voice. Enables participants to effectively engage in initial discussions about the meaning of fundamental C4D and R, M&E concepts (including participation, ownership, sustainability, equality, and equity).
• Uses openness, freedom, and flexibility in developing evaluation frameworks and plans. This process begins by agreeing on the initiative's objectives and outcomes and clarifying the evaluation's purpose and stakeholders' expectations.
• Takes a long-term view of the process and the benefits of a participatory, mixed methods approach and the use of longitudinal studies to assess impacts and outcomes. The evaluation is proportionate to the scale of the programme, and timeframes to achieve outcomes are realistic.
• Involves staff at all levels in long-term capacity development which focuses on organisations as a whole and the development of learning organisations. Evaluation capacity development aims to increase understanding of the fundamentals of R, M&E, and provides practical guidance and simple, user-friendly information for managers, field staff, and community members.
• Goes beyond a focus on individual behaviour to consider social norms, policies, gender and power relations, culture, and the specific and general development context. Data are disaggregated by gender, age, ethnicity, and other relevant differences.
• Involves developing locally and externally derived indicators and a dynamic, moving theory of change. Considers the short-term, intermediate, and long-term impacts of initiatives. Adjusts baseline information to recognise changes in the context. Triangulates findings to increase rigour and ensure that a diversity of voices and views are included.
• Involves continuous critical reflection and learning. Includes an action component and a process of ongoing improvement through feedback loops as a programme is implemented.
• Shares openly and honestly positive, negative, intended, and unexpected findings with participants, donors and funders, and the larger development community.

The research identified a number of significant and fundamental challenges, tensions, and issues that need to be addressed in order to better meet the UN's development goals (particularly the Millennium Development Goals, or MDGs) and the aims and objectives of C4D and related development activities. These findings are summarised in the Executive Summary of the report in a series of tables that aim to highlight the key characteristics of, and the critical differences between dominant and alternative approaches to, C4D, R, M&E for C4D and evaluation capacity development, in the interest of ensuring that more informed and effective decisions are made. The authors stress that "It is not the approaches to R,M&E themselves, but how they are applied...that is critical here. For example, participatory approaches can be implemented in very top-down ways. They can also present many challenges for time and resource poor UN agencies and their partners whose staff may need to strengthen their evaluation capacities in order to produce more useful and rigorous research." The findings and issues identified by this research are summarised under the following headings:

• Approaches to C4D
• Approaches to R, M&E of C4D
• Assessing the impacts of C4D
• Indicators for C4D impact
• Key approaches, methodologies, and methods for R, M&E of C4D
• Evaluation capacity development

Drawing on new thinking, approaches, and trends in this area, this report presents an emerging framework for R, M&E of C4D which incorporates the principles summarised above. In brief:

1. Conceptual and theoretical framework
   • A holistic perspective based on complexity and systems thinking
   • Takes the wider context into account
   • Focuses on gender, power and social norms
   • Takes a holistic approach to evaluation capacity development that aims to develop learning organisations
   • Evaluation is seen as an ongoing learning and programme improvement process
   • Takes a long-term, sustainable perspective on evaluation and evaluation capacity development.
2. Methodological and reporting framework
   • Adopts an open, flexible approach to designing evaluations and selecting R, M&E approaches, methodologies and methods
   • Uses participatory approaches as appropriate
   • Uses a mixed methods approach and triangulation
   • Impact assessment uses contribution assessment and a dynamic, moving theory of change and involves longitudinal studies
   • Makes more use of qualitative and participatory indicators
   • Evaluation is independent and learns from ‘failures’ and negative results
   • Establishes open communication and feedback systems.

According to the authors, implementing both the conceptual and methodological parts of the framework requires a clear strategy in order to address a number of challenges:

• There is a need for advocacy across the UN and with other organisations and donors to highlight the importance of C4D and R, M&E in development.
• There is a need for greater understanding of the appropriateness and long-term benefits of participatory approaches.
• There is a need to create a common understanding of C4D and its various benefits.
• There is a need to provide sufficient budgets, resources, and time, including for longitudinal studies.
• There is a need to improve capacity in conceptualising, managing, and planning R, M&E of C4D within the UN, and with partners.
• Long-term capacity development for staff at all levels is required, with high quality and yet accessible training and reference resources.
• Creative and innovative strategies need to be employed to develop the framework, using participatory and collaborative methods.
• In addition to the need to collect and present good examples of highly effective R, M&E for C4D, there is also benefit in undertaking meta-evaluations of these examples.
• The open sharing of positive and negative, intended and unexpected findings needs to take place within an environment that understands the benefit of learning from success and failure.
• There is a need to establish a community of practice with online access to expert advice.

Funding for the development of this report was provided by the United Nations Children's Fund (UNICEF), with additional funding from Australia's Creative Industries Faculty, Queensland University of Technology.