Evaluating Digital Citizen Engagement: A Practical Guide

Published by Open Knowledge Repository of The World bank, this guide provides practical steps to assess how digital tools have contributed to citizen engagement and to understand the impact that technology has had on the engagement processes. Using examples and lessons from case studies from Brazil, Uganda, Cameroon, and Kenya, the guide discusses tools and guidelines that can be used to evaluate the expanding field of digital citizen engagement. As stated in the guide, "with growing demand for transparency, accountability and citizen participation in policy making and service provision, engagement between citizens and their governments, as well as with donors and the private sector that deliver government services, is increasingly important. Increased use of technology brings both opportunities and challenges to citizen engagement processes, including opportunities for collecting, analysing and evaluating data about these processes."
The guide was produced to help people understand when the use of digital citizen engagement is appropriate and under what circumstances, how to use it more effectively, and what to expect from its use. It introduces the key issues relating to digital citizen engagement and offers advice and guidance on how to evaluate it, including methods, indicators, challenges, and course corrections that apply to the digital aspect of citizen engagement. The guide complements an existing work on mainstreaming citizen engagement across the World Bank's operations and seeks to add value to this work by focusing on those programmes where technology plays a significant role in citizen engagement activity, and focusing on the evaluation of the effectiveness of both the citizen engagement activity overall, and the role of the technology within it.
An example from one of the 4 field evaluations (case studies) included in the guide can offer a snapshot of what DCE looks like in practice. The evaluators (the authors of this guide) investigated the impact of online voting on the state-wide participatory Budgeting process in Rio Grande do Sul, Brazil. They explored differences between online and offline voters: the extent to which online voting impacted on overall voter turnout; if online voting reached different sections of the population from those who traditionally engage in the face-to-face process; and whether online voters interact with the participatory budgeting process in a different way from those engaged face-to-face.

After introductory materials, the guide offers: Section 2 Digital Citizen Engagement: defines DCE and the benefits and challenges within it, particularly in areas with low technological access.
Section 3 Evaluating Digital Citizen Engagement: outlines existing evaluations of DCE and introduces the construct of five "lenses" (Objective, Control, Participation, Technology, Effects) that can assist in focusing on the most important aspects of DCE when conducting an evaluation.
Section 4 A Practical Guide to Evaluating DCE: explores how to conduct an evaluation of DCE, following standard stages of an evaluation lifecycle (Scoping, Designing, Planning & Implementing, Analysing, Sharing, Reflecting & Learning). Each of the evaluation lifecycle stages is introduced with a cover diagram showing the stage, followed by a box containing a brief explanation of the purpose of the section and what content it covers. Each stage ends with a brief checklist of the key points that an evaluator should have addressed (or at least considered) before moving on to the next stage. Reflections of key lessons and learning from the 4 field evaluations that were undertaken as part of developing this guide are shared for each lifecycle stage.
Section 5 Evaluating the Evaluation - Reflecting on the Process: provides some final thoughts and reflections on the evolution of DCE and what this means for a guide such as this along with a call for wider involvement in the continued evolution of this work.
Toolkit 1 DCE Evaluation Bank: offers examples of primary (assessment/analysis) and supplementary (information gathering) evaluation questions - grouped by lens - and some "satisfaction" questions.
Toolkit 2 Using the Lenses in Scoping and Design: includes a set of considerations and questions that an evaluator might ask during the scoping and design stages, again grouped by lens.
Appendices: includes global examples of DCE interventions, results indicators for citizen engagement, and an overview of the Brazil, Uganda, Cameroon, and Kenya case study data collection methods and costs.

A variety of tips from a range of experts are provided in supplementary boxes throughout the text, and links to references and further reading are offered.