Benchmarking Immunisation Program Performance in the Africa Region

This paper is the final report of a World Bank study that was intended to demonstrate the application of the benchmarking concept to childhood immunisation in Africa, and to examine the different factors that have contributed to the success or failure of immunisation activities in the region. Because immunisation coverage (in this case DPT3 coverage) is regarded as a proxy for the overall strength of health systems, and because several of the indicators evaluated would have crossover impacts on other areas of health service delivery, the results are believed by the authors to have potentially wide-ranging implications. The study is comparative and uses a three-stage investigation which included 1) a differential diagnosis of immunisation programme effectiveness in 43 African countries; 2) a series of country case studies to explore variability in programme execution; and, 3) a cross-case analysis to generate hypotheses about variation in programme effectiveness. The authors hope that this exercise will help shape the region's engagement in this key public health intervention, as well as enhance collaboration with partners in the field.

Methodology

The first part of this study evaluated the overall programme strength, measured in terms of DPT3 coverage, and the change in effectiveness over time in 43 African countries from 1997-2002. The authors constructed a 3 x 4 matrix to group and compare countries, and to place them in different performance classifications which reflected their current and historical status and their direction of movement.

In the second part of the study, the authors undertook a series of country case studies. Six countries (Rwanda, Ghana, Mauritania, Ethiopia, Malawi, and Cameroon) were selected for in-depth, retrospective reviews of their implementation experiences using a combination of qualitative and quantitative data. Each of these countries was evaluated with reference to several programme implementation variables that had been previously identified from a literature review. These components included the governance/institutional framework supporting the immunisation campaign, management of the activities, strategy, financing, and demand creation.



The third phase was a cross-country analysis that was designed to generate hypotheses about variations in programme effectiveness. In this stage, the programme implementation variables from Phase 2 were further refined and the authors undertook several steps to ensure cross-case comparability, adopted a standard set of measures for each variable, and developed a rating system to determine the degree to which these measures were present in each case. A panel of five analysts independently reviewed each case and scored each measure which was then averaged. A matrix was used to identify patterns in the data.



Results

The first phase of the study revealed substantial variation in programme effectiveness between countries. Average coverage for the six-year period ranged from a low of 27% to a high of 98%. The range for average annual percentage point change was 11.0 to 12.1. When these factors were examined together, of the 43 countries assessed, approximately 33% were classified as "strong" performers; 11% as "historically strong/losing ground" performers; 30% as "historically weak/gaining ground" performers; and, 26% as "weak" performers. Almost two-thirds of the countries, regardless of the historical strength of their programmes, demonstrated some degree of progress, and the percentage of weak/gaining ground countries was greater than that of the strong/losing ground and weak countries.

The results of the second phase of the study, the series of case studies, provided important evidence and insight into the processes and events that determine the relative success of each subject country. The specific factors that have contributed to success of failure in the different countries are documented in these case studies. The authors note that while many of the programmatic decisions that were taken were important determinants in the success of the immunisation programmes, the country performance was also subject to a range of unalterable factors such as population density, geography and culture. The full country case studies are contained in a separate Annex (A).

The third phase is to a great extent a continuation of the second, and led the authors to several important conclusions. The primary finding was that there was no single specific route or combination of activities that led to success, different countries experienced success or failure by taking different paths. The other main finding was that countries needed to be performing relatively well in all five of the programme components that were identified (governance/institutional framework, management, strategy, financing and demand). The two strongest countries, Ghana and Rwanda, ranked high in all 5 areas, though neither country performed equally well on every component, and each did better than the other on certain components.

While a good score in each category was positively related to programme performance (measured in terms of coverage), even exceptionally high achievements in only two or three categories was not enough to raise the programme performance levels. Several other patterns emerged:

• A dominant delivery strategy characterised the successful countries.
• Demand for services was not necessarily a function of the degree of intensity of traditional demand creation activities alone and the quality of service provision played a key role in demand creation.
• The degree of external financing, or the presence of a sector-wide approach (SWAp), did not appear to be correlated with effectiveness.

Key Findings & Recommendations

The synthesis of the information gleaned from the three phases of this study led the authors to arrive at several conclusions and to make a series of recommendations directed to the World Bank, the countries involved, or the global immunisation community. Only a few of the conclusions and recommendations are reported here:



To the World Bank:

1. Considering that immunisation coverage, particularly for DTP3, is often viewed as a surrogate measure of the overall strength of the health care delivery system the Bank should be concerned about the high degree of variability in performance in the region. Additional attention and support should be directed towards underperformers, who may also be at risk of sub-optimal performance on other essential health services.
2. The Bank should recognise that robust implementation of all components of the programme appeared to be critical to success.
3. The Bank should take the lead in 1) facilitating inter-country exchange of good practice in immunisation programme implementation, and 2) developing country capacity to capitalise on external opportunities to enhance programme effectiveness.
4. The Bank should consider the utility and replicability of the benchmarking methodology in additional countries.

To the Countries:

1. Certain combinations of delivery strategies, with different emphases, may be more appropriate in some settings than others. Countries will have to make decisions based on what is regionally appropriate. The efficiency of different combinations should be documented whenever possible so that countries under similar circumstances can make the best evidence-based choices.
2. Information-education-communication (IEC) activities may be necessary, but are not sufficient to increase demand for immunisation services. A more critical element of demand creation may be increased investment in improving the availability and quality of service provision, a supply-side intervention.
3. Countries must prepare for the long-term transition away from external financing towards sustainability in their programming.
4. Countries should consider applying a simplified version of the benchmarking approach and methods to identify performance differences among districts, to identify the reasons for these differences.

To the Global Immunisation Community:

1. The global immunisation community should create more opportunities for countries to meet and share their experiences. Country meetings should be convened in a manner that optimises learning based on performance history rather than commonalities of geography and language.
2. Various incentives could be used by the global immunisation community to promote these and other countries' adoption of innovative solutions to intractable problems such as sub-optimal productivity of health workers.
3. Countries require assistance in figuring out how to best allocate their resources, both domestic and external, across the full range of programme components.
4. The global community should revisit previous efforts to develop a set of indicators of immunisation programme performance that can be used to monitor countries' progress over time.
5. The global community should consider the possibility of developing a benchmarking toolkit that countries can use to assess, explore, explain, and act upon differences in performance at national, regional and/or district levels.