The Second International Conference on National Evaluation Capacities (NEC) was opened on September 12th at the Protea Hotel Balalaika in Johannesburg, South Africa. MCs, Ms. Azusa Kubota & Mr. Indran Naidoo, welcomed the conference participants and introduced the distinguished guests who provided opening remarks: Mr. Ben Mthembu, Chairperson, Public Service Commission of South Africa; Mr. Agostinho Zacarias, UN Resident Coordinator and UNDP Resident Representative in South Africa; and Mr. Sean Phillips, Director General, Department of Performance Monitoring and Evaluation.
Mr. Juha Uitto, Deputy Director, Evaluation Office, UNDP provided a recap of the 2009 NEC conference in Casablanca, Morocco. He touched on the key points as well as recommendations drawn from the previous conference. (All speeches available in the website of the conference)
The tone of the conference, and its theme, “use of evaluation” chosen based on the ideas of participants of the past NEC conference in 2009, was set by the presentation of the Conceptual Framework by the NEC Conference Advisory Group with inputs from UNDP EO and OPSC. Mr. Rachid Benmokhtar Benabdellah (Morocco) presented the framework on behalf of the advisory group and NEC organizers. Ms. Ronette Engela (South Africa) chaired the session.
A brief discussion followed. Issues highlighted for future more in-depth discussions, many of which were addressed in later presentations, were anticipated in this session and included:
- how to maintain balance and independence of the evaluations,
- how to make evaluation more timely, while also monitoring. This depends: if programs take time or have effects on people, monitoring is more suitable as opposed to evaluation.
- dissemination and use for policy: good findings are welcomed by decision-makers but negative ones are useful if the institution doing the evaluation is independent, also financially.
- the importance of champions of evaluation, and independence of individual evaluators.
Following this, Plenary Session 1: Use of Evaluation for Public Policy and Programmes - challenges, factors and opportunities began, chaired by Mr. Patrick Birungi from Uganda. During the session, Ms. Sudha Pillai from India spoke on the role of evaluation in the 5 year planning in India." Mr. Velayuthan Sivagnanasothy from Sri Lanka presented the role of evaluation in influencing the policy from the Sri Lankan perspective. Mr. Indran Naidoo from South Africa described the use question in South Africa, with examples and lessons from the Public Service Commission of South Africa. To conclude this session, Mr. Walter Mauricio Aguilar and Mr. Diego Dorado from Colombia presented from Colombia (via Skype) on the topic of effective evaluations: processes to advocate for changes in public policy.
The discussion that followed focused on the importance of institutionalizing the evaluation function mentioned in the presentations. Some of the main points highlighted were the:
- Need for positive linkages between planning departments and independent evaluation departments in governments.
- Use of evaluation to explain implementation of public policies and inform stakeholders.
- Importance of the participation of civil society in evaluations.
- Importance of joint evaluations, and of recipients and donors working together
- (Political and financial) independence of evaluators as a key to quality of evaluations.
- Search for “champions” to promote effective evaluations in different countries.
and a reflection on the need to evaluate the evaluators, despite lack of clarity on ways to do so.
Following, the first two panel sessions occurred in parallel. Panel 1: Systemic factors contributing to use of evaluation was chaired by Mr. Seydou Yayé, from Niger. Mr. Aristide Djidjoho from Benin presented on the use of evaluation and development in a national assessment in Benin, Mr. Prajapati Trivedi from India spoke on the Indian experience with the performance Monitoring and Evaluation System for government departments, and Mr. Henry Morales from Guatemala talked about “Multi-stakeholder participatory evaluation systems in the field of public policy.”
The discussion that followed focused on the factors enabling environment and institutional frameworks to contribute to the use of evaluation. Some of the main points highlighted were the:
- Government involvement as a key factor in allowing effective follow-up of actions and recommendations from evaluation and dissemination of evaluation among stakeholders.
- Importance of multi-stakeholder participation in evaluation systems.
- Developing National Evaluation Policies, professionalizing the evaluation function, developing of national funds and councils for evaluation, as possible next steps for some countries
- Need to understand differences and interdependence between monitoring and evaluation concluding with suggesting to develop an International Index that evaluates evaluation systems.
In the parallel Panel 2: Impact of audits and budgeting in evaluation, chaired by Ms. Thania de la Garza Navarrete from Mexico, Ms. Selma Maria Hayakawa C. Serpa from Brazil presented on ways to promote accountability and enhance programs and policies through the instrumental use of the evaluations carried out by the Brazilian Court of Audit. Following a video of Mr. Mohammed Chafiki from Morocco was presented on the case of gender responsive budgeting as a tool for public policy evaluation.
The discussion that followed highlighted:
- How audits are exercises that can contribute to the evaluation of policies and integration of evaluation in the budgeting process, so to improve the effectiveness of public policy.
- The need to launch a discussion in the evaluation community on respective roles of audit and evaluation to analyze, from an international perspective, issues such as methodology, standards, data quality, causality, credibility of conclusions, training, competencies of the auditors, etc. Practices on doing so vary depending on cultural and other traditions.
Ms. Nadira El Guermai, from Morocco, chaired the reporting and wrap-up session at the end of the day. This was followed by the opening of the kiosks with a display of publications, posters and evaluation materials prepared by participants. Zambia’s Ms. Prudence Kaoma presented a poster on use of evaluation in decision making for public policies and programmes, and Ms. Marcia Paterno Joppert from Brazil displayed the Monitoring and Evaluation case in Brazil: supply, demand and recent developments. Ms. Ana Morice from Costa Rica portrayed her country’s experience on assessing the impact of strategies to reduce child mortality. Thania de la Garza and Hortensia Pérez from Mexico showed their experience on closing the evaluation cycle focused on follow-up recommendation mechanism for the improvement of public policies.
The second day of the conference began with Plenary Session 2: Use of Evaluation for Public Policy and Programmes - challenges, factors and opportunities, chaired by Mr. Velayuthan Sivagnanasothy from Sri Lanka. Mr. Yonghe Zheng from China spoke of the roles of users in enhancing utility of evaluation with special reference to the international evaluation on the funding and management performance of the National Natural Science Foundation of China. Mr. Albert Byamugisha from Uganda presented on giving national direction through evaluation, with his country’s case of evaluating its Poverty Eradication Action Plan. Mr. Mohamed Benkassmi and Mr. Mohammed Mouime from Morocco described the ONDH information system for the evaluation of human development public policy, and Ms. Hortensia Perez from Mexico presented key elements of evaluation as a decision making tool, referring to the performance evaluation experience in Mexico. Finally, Ms. Junia Quiroga from Brazil showed strengths and weaknesses in using evaluation results in decision making for social programs, with reference to the evaluation system of the Ministry of Social Development and the Fight Against Hunger.
Parallel sessions continued in the second day. Panel 3: Monitoring and how it facilitates use of evaluation was chaired by Mr. Armando J. Vieira Filho, from Brazil. Ms. Hernan Rodriguez Minier from the Dominican Republic described the his country’s experience in the design and implementation of a monitoring system as community methodology to measure progress and impacts of the Millennium Development Goals and National Development Strategy. Mr. Ronald Mangani from Malawi presented on use of evaluation in managing for development results. Mr. Seydou Yayé from Niger talked about the process of Control in Project Advancement (CAP) with reference to the case of the CAP Program to Combat Poverty in Niger (LUCOP). Mr. Bahodir Eshboev from Tajikistan presented on the strategic development goals and priorities of the national monitoring and evaluation system of the Republic of Tajikistan, and Mr. Ekingo Magembe from Tanzania described the importance of monitoring and evaluation in achieving national development policies and programmes’ targets.
The ensuing discussion focused on Monitoring & Evaluation (M&E) challenges that emerged as priorities from the cases, in particular, the need to learn from the experience of other countries, strengthen M&E capacity, use best practices, establish unified M&E networks with development partners, promote with the private sector and investors the use of M&E data, institutionalize M&E activities across government departments, and set targets and baselines to help conducting M&E. In terms of data and information management, the issues discussed reside in the scarce availability and uneven quality of data, especially for MDGs monitoring, the lack of common methodologies to collect data, of standardized and harmonized formats (contributing to uneven quality of evaluation reports) and the need to have in these data that comes from all players, not just from some like the government, so to avoid biased reports. Finally, in terms of sustainability, changes in government and the end or discontinuation of projects also threaten the proper M&E.
Panel 4: Establishing evaluation systems, taking use of evaluation into account was chaired by Ms. Katongo Chifwepa, from Zambia. It included Mr. Ryan Cooper from Chile’s presentation on the ‘Compass Commission, Abdul Latif Jameel Poverty Action Lab’. Ms. Ana Morice from Costa Rica spoke of assessing the impact of strategies to reduce child mortality in Costa Rica. Ms. Shahrazat Haji Ahmad from Malaysia described the Malaysia experience on use of evaluation. Mr. Mohamed Fadel of Mauritania presented on using evaluation in decision making for public policies and development programs. Mr. Darinchuluun Bazarvaani from Mongolia talked about the evaluation of the implementation of the MDG-based comprehensive National Development Strategy and using evaluation results, and Mr. Farkhat Kassimov from Kazakhstan presented on the Kazakhstan experience, challenges, immediate outcomes and future prospects from the case of the country’s government performance evaluation.
The discussion that followed highlighted a variety of aspects:
- The challenges encountered in promoting political commitment that can allow achieving a balance between participation and independence on the evaluations.
- Strengthening national capacities to manage the evaluations as critical factors to deal with staff turnover, lack of political support and involvement of actors, and budget constraints.
- The impact of lack of legal frameworks to guarantee changes implemented after evaluations.
- Issues with financing evaluations, the government commitment to funding evaluations being a way leading to more ownership of the information generated by evaluations.
- The need to institutionalize and strengthen the professional capacity for evaluation - including enhancing mechanisms to communicate results, findings and recommendations.
- The need to synchronize evaluations with planning cycles and budgeting processes. with a final reflection on the fact that when users of evaluation do not participate actively from the beginning in the process, the recommendations have a much lower probability of being used.
In the final Plenary: Towards longer-term initiatives on national evaluation capacity the chair, Mr. Juan M. Cordero (Costa Rica) recap the emerging issues and lessons from Day 1 & 2.
The main points that were raised or revisited were:
- Evaluation is a key component of public policy but may not be interpreted as such in many countries. It is crucial to promote public participation, integrated constitutionally.
- Promotion of use of evaluation should allow broader access and democratization of information, for data to be validated and allow the public to make their own assessment.
- Evaluations should be disseminated by independent bodies to communicate the results in full (both the positive and negative findings and recommendations).
- Evaluation may be used as a tactic in a political context. The sustainability of the evaluation process, in fact, has to do with how it links with the overall political process.
- For evaluation to be used, useful communication strategies must be developed. Findings need to be better synthesized in ways appropriately directed at possible evaluation users.
- Some factors influencing the use of evaluation are directly linked to whether there is an enabling democratic environment within an evaluation culture permeating in the society.
- Institutional frameworks, with the existence of national or sectoral departments with their own funding and agenda, laws and rules for evaluations may promote evaluation use.
- Evaluation report formats in each country should present the government's performance in homogeneous and user-friendly ways.
- Evaluations should be integrated in countries’ legal framework, possibly in developing national evaluation policies; some national contexts, however, may not be ready for that.
- Evaluations depend on the state of the country’s democracy: for example the M&E in a closed country is not very feasible or useful. Transparency goes together with democracy.
- In general, a common feeling of participants is that government should not evaluate itself.
- Closing the communication gap: dissemination is important but is not an end in itself.
- The importance to differentiate types of evaluation: process evaluations (may be faster) and have greater impact (results may not be in time for an administration, but can help the next one, and other countries) to make sure that evaluations fit the appropriate needs.
- The importance of defining the arbiters, because evaluations may depend on the demand of those who have to implement the results and may exclude the evaluated civil society.
- Governments may need support to understand and make the distinction between driving and requesting an evaluation, as well as how to make proper use of it once completed.
- Finally, the discussion ended with opening the question of ‘who evaluates the evaluator’?
The discussion generated potential topics or themes for the next NEC conference (2013):
- The existence of different types of use and their relation to different types of evaluations.
- How to link evaluation with public policy.
- Build national capacities considering differences between countries and fomenting political will to support the use of evaluations.
- The need to build the capacity of actors to respond to the results of the evaluations.
- How to improve information availability and information systems.
- How to define the evaluation agenda and build plans that consider evaluation processes
- Empowering people through the use of score cards of the services provided by the government, as a way to link technical evaluation and the opinion of the beneficiaries.
- Focus on the "how", i.e. best practices to help countries implement evaluation practice.
- The evaluation of the position of countries with respect to the attainment of the MDGs.
In terms of location, it was suggested that the next conference is in the Latin America and Caribbean area, with emphasis to bringing countries and experiences from the Caribbean as well. However several countries expressed an interest to host the next conference: to be followed up.
The following monitoring mechanisms for the NEC conference were proposed:
- A process of reflection started in each country, starting with the synthesis of the conference, calling for different actors within the context of the country to share some collective reflections.
- Experiences by regions are exchanged, starting on an internet portal to be provided by UNDP.
- UNDP will have to publish the experiences that were shared, yet it was also suggested that before the next conference, each country follows up and updates their papers.
Ms. Azusa Kubota and Mr. Indran Naidoo opened the closing session that included closing remarks by Mr. Mashwahle Diphofa, Director General, the Office of Public Service Commission; Mr. Ernest Fausther, OIC, UNDP Johannesburg Regional Service Center; and Mr. Juha Uitto, Deputy Director, EO, UNDP.