Consultancy for Final Programme Evaluation and Beneficiary Satisfaction Survey

FCDO–funded Consortium programme titled “Emergency assistance to crisis-affected communities of Central African Republic, Phase 3” implemented by 4 INGOs: The MENTOR Initiative, International Medical Corps, Cordaid (replaced by ACF in January 2022), and Solidarités International (phased out in March 2022) between April 2019-March 2023.

1. BACKGROUND OF THE PROGRAMME

The MENTOR Initiative (MENTOR) has been present in CAR since 2008. From the very start of the mission, core activities have focused on supporting community-based healthcare services on a large scale. This approach has proven to be effective and fully appropriate in a context where standard health system structures are virtually non-existent outside of main towns and where malaria, a disease that is easily diagnosed and treated if caught on time, is endemic and one of the major causes of mortality. MENTOR is currently operational in eight sub-prefectures in north-western CAR.

In 2015, The MENTOR Initiative, International Medical Corps, Save the Children (phased out to Cordaid in 2016), and Oxfam entered into a Consortium partnership for the first phase of the current programme, with International Medical Corps as lead of the Consortium. Phase 1 aimed to implement a health, nutrition, and WASH programme response reaching an estimated 432,665 persons across four prefectures of Central African Republic (Ouham, Ouham Pendé, Haute-Kotto, and Basse-Kotto) and aimed at preventing and reducing excess morbidity and mortality of the population affected by the ongoing crisis. A final programme evaluation from Phase 1 was undertaken and is available.

The 4 INGOs entered into Phase 2, with The MENTOR Initiative as lead, on October 1, 2016 for a period of 2 and a half years. Oxfam, as the partners specialized in WASH activities, was phased out from February 1, 2018. The health, nutrition, and WASH programmes were implemented in the same 4 prefectures with the aim to build upon what took place during Phase 1 while harmonizing on best practices and developing standardized community sensitization tools. However, the security situation in the CAR remained extremely volatile throughout the entire Phase 2 with frequent crisis and security deterioration. Consortium partners had to adapt and revise the programme to be able to implement pertinent emergency activities where necessary including centralizing the procurement of essential anti-malaria drugs to be able to face potential stock-outs.

On April 1, 2019, the Consortium entered a third phase, for a duration of four years (until March 31, 2023) with the same three health partners, The MENTOR Initiative, International Medical Corps and Cordaid, and Solidarités International as a new specialized member in charge of the WASH component of the programme. The focus of the Consortium remained the same. Phase 3 was also informed by the findings and recommendations of the external evaluation conducted during and at the end of Phase 2. During Phase 3 the Consortium plans to improve its performance in terms of value-for-money, building on the savings expected to be generated following the purchase of capital assets, such as new vehicles (significantly reducing rental costs). The Consortium has expanded its community health strategy through community health workers as IMC has progressively adopted this approach. This phase of the project also aims at providing all health facilities with quality WASH infrastructure and access to safe water. The project has been designed with flexibility, with an emergency financial reserve allocation, to be able to respond to sudden crises given the volatile operational and security context.

1. ENDLINE EVALUATION

2A) GOAL AND OBJECTIVES

A midterm evaluation was conducted in June/July 2021. Its results have informed the objectives of the endline evaluation. The goal of the endline evaluation is to assess and learn from the process and achievements of the programme in the four sub-prefectures, evaluate progress towards the program objectives and results, implementation strategies, assess the extent to which the programme demonstrates good “Value for Money” (using FCDO’s Value for Money approach), analyse consortium coordination mechanisms, its advantages and recommend areas for its potential improvement, document lessons learned, and to inform the partners, FCDO, beneficiaries and other relevant stakeholders about results and findings.

Specific aspects for consideration

1)The evaluation will be guided by the evaluation criteria of relevance, effectiveness, efficiency, impact, and sustainability.

The evaluation should specifically assess:

• Relevance: Assess to what extent local needs and priorities have been addressed, activities and output are consistent with the 4 intended outcomes of the programme.
• Effectiveness: Assess what has been accomplished in relation to expected outcomes and results set in the logframe in each area of intervention and from a central level, specifying the major factors which have contributed to the achievement or not of the intended objectives; assess the extent to which the Consortium coordinates with other relevant actors in each zone and identify where this could be improved upon; assess the Consortium’s ability to adapt to evolving security and political contexts (including the COVID-19 pandemic) in specific intervention zones and CAR in general.
• Efficiency: Assess how inputs (human, financial, material resources) have been translated into results and if results have been achieved at an acceptable cost – see Value for Money below; assess to what extent the Consortium approach and the management of the Consortium programme has provided added value in attaining achievements or has hindered it.
• Impact: Assess to what extent the programme has contributed to reducing excess morbidity and mortality of the targeted population affected by ongoing crises; assess how the programme has leveraged other complementary projects to achieve greater impact. Being a health project, a relevant area of focus shall be to evaluate the appropriateness and consistency of diagnoses and treatment, the rational use of drugs and the diagnostic steps followed by health personnel (this could be achieved by checking whether Community Health Workers and health workers are following their diagnostic and treatment algorithms).
• Sustainability: Assess to what extent the programme will produce benefits in the long-term to the communities and the national health system and identify gaps that could be filled.
• Management: Assess to what extent measures to prevent safeguarding and fraud incidents are implemented in accordance with partners own and international standards. Comment on the consortium’s overall performance including its management of responses to such issues.

2) Accountability How each INGO has been accountable to beneficiaries, national and local authorities, and the donor.

“Value for Money”: In line with FCDO’s policy, the evaluation will analyse the achievement of the Consortium towards identified “Value for Money” indicators and evidence that partners have effectively assessed and considered value for money considerations in the management of the programme. The evaluation should identify areas where improved synergies could translate into better Value for Money in the future.

Due to chronic insecurity in certain intervention zones, Consortium partners have developed and are implementing remote management tools. The evaluation should identify to what extent these tools and processes impede or promote accountability.

3) AccessibilityHow access to health services affected the various population in the 4 sub-prefectures specifically pertaining to health, nutrition, and WASH aspects. Identify if there have been factors preventing or restricting access of certain groups or minorities to healthcare services and potential strategies to address them.

4) Quality Evaluate quality of interventions in terms of indicators chosen and reported against, implementation methodology, competence of staff, team building, gender, and if health care provided through the programme is socially and ethically acceptable.

5) Lessons learnedEvaluate the extent to which the Consortium has acted upon lessons learned during previous phases and the midline evaluation (including evidence and data gaps) and identify further lessons learned, both positive and negative. It will also look at whether the needs of the target population and provision of health services has changed since 2019 and seek to distil how any future funding can meet the critical needs of the population.

2B) EVALUATION TIMEFRAME, METHODOLOGY, AND DELIVERABLES

Design and methodology used in the evaluation will be proposed by the consultant, but should include quantitative and qualitative methods, a desk review of relevant documents, interviews of relevant stakeholders, focus group discussions and observation. Bids for the consultancy should provide a clear description of the design and methodology the consultant will use to answer the key questions, including recognized evaluation methods, proposed counterfactuals if/where appropriate, data collection methods, analytical methods, and approach to synthesis.

Visits to field sites are expected; however, security will be assessed before each field mission and the consultant would be expected to travel from Bangui to field sites only when partners have deemed security is sufficient.

The consultant will be provided with previous M&E documents including logframe, databases of medical data since the beginning of the programme, quarterly reports, midterm evaluation, etc. The consultant will be responsible for the identification and provision of any new primary data needed for the purposes of the independent evaluation, particularly taking into account recommendations from the midterm evaluation. The consultant will need to determine which arrangements would be most cost-effective overall and least burdensome on beneficiaries or programme implementers.

MENTOR requests that all databases containing qualitative and quantitative data collected during the consultancy are to be shared in a commonly used format, together with clear metadata, and which is anonymized and safeguards confidentiality.

Reports should communicate overall findings in an accessible way for non-technical readers, including presentation of data in visually appealing ways, highly structured and rigorous summaries of findings and robust and accessible syntheses of key lessons. Recommendations should be timely, realistic, prioritized, and evidenced-based.

Relevant field visits to assess progress are expected where necessary especially to programme areas that could have not been accessed during the mid-term review due to security constraints. A draft and final evaluation report are expected.

1. BENEFICIARY SATISFACTION SURVEY

3A) Objectives

The survey aims to inform the consortium about how its procedures, activities and implementation are perceived and whether it is meeting beneficiaries’ needs. The information collected will enable the consortium to respond to beneficiaries needs within the scope of the programme. Although consortium partners already have feedback mechanisms in place, the survey aims at providing additional standardized information on the impact of the project and its perception among beneficiaries.

The objective is to engage and strengthen beneficiaries’ capacity to take an active role in shaping the programmes future, and bring about the changes needed to address vulnerabilities and build resilience in the specific context of CAR and within the scope of preventing morbidity and mortality and building dignity by enabling communities to access quality, timely health care.

3B) Methodology

The survey shall focus on qualitative aspects and shall cover beneficiaries and stakeholders at various levels including direct project beneficiaries, disaggregated by type of health care service received, gender, eventual presence of disability, community leaders, local health partners. In order to maximize efficiency, the survey shall be conducted along with the endline evaluation and it is expected to last up to two weeks.

The consultant shall define in consultation with the Consortium Coordination team the most effective and efficient mix of investigation and analytical tools and methodologies to gather and present the data in a final report. Methodologies shall include a review of the beneficiary survey as part of the midterm evaluation, key informant interviews, focus group discussions and community participatory techniques. The contractor may want to consider UNICEF’s Ground Truths Solutions surveys and shall use the Washington question to disaggregate beneficiaries by type of disability to analyze accessibility to and quality of health care provided to beneficiaries, including disabled ones. Results shall be compared to the midterm evaluation beneficiary survey and identify what lessons can be drawn out of it to inform programming and response.

The Consortium, through its partners, shall identify, recruit and pay incentives to surveyors locally who will conduct the data collection in order to overcome the challenges due to local language and culture.

The consultant shall train the surveyors on the agreed methodology to collect data.

The Consortium shall provide logistical support to the teams of surveyors in selected areas.

3C) Deliverables

The output of the survey shall be a Beneficiaries Satisfaction Survey Report detailing the methodologies adopted, sets of questionnaires utilized by type of beneficiary, presentation of findings, sites covered, list of meetings and interviews held, challenges faced during the survey and potential limitations of the findings.

1. COMMUNICATION MANAGEMENT

The consultant’s primary contact in CAR will be the Consortium Coordinator. Secondary contacts include the Consortium manager (based in UK) and MENTOR’s Head of Mission (based in Bangui).

1. CONSULTANT PROFILE

• Higher university degree in relevant field, i.e. Public Health, Development Studies, Humanitarian Response, etc.
• Demonstrable relevant practical experience in conducting evaluations in development and humanitarian settings using qualitative and quantitative research methodologies.
• Proven experience in using participatory evaluation methods, conducting evaluations for complex multi-sectoral, health programmes in challenging settings, experience with FCDO-funded projects and/or Consortium programmes is an asset.
• Strong understanding of CAR context is an asset.
• Strong computer skills including knowledge of appropriate statistical analysis software.
• Excellent spoken and written communication skills in English and French, knowledge of Sango is an asset.
• Strong analytical and communication skills for complex, fast-paced environments.
• A strong commitment to delivering timely and high-quality results.
• Expected to sign and follow MENTOR’s Code of Conduct and the 4 principles and standards elaborated below.

Principles and standards

1. The evaluation and survey should adhere to international best practice standards in evaluation, including the OECD DAC criteria for humanitarian evaluations and FCDO’s Ethics Principles for Research and Evaluation. Bids should demonstrate how they will achieve this.
2. In line with Paris Declaration principles, the consultant should draw on existing data where available, ensure new data collection is complementary to existing systems and that new data are made available to stakeholders as far as possible.
3. Disaggregation of data, including by sex, gender, and disability will be useful throughout the evaluation.
4. The consultant will need to comply with MENTOR’s policies on fraud and anti-corruption and cooperate with any checks required from MENTOR for the duration of the evaluation.
5. For the Beneficiary Satisfaction Survey, we expect the consultant will need to work with a team of surveyors, who’s skills and background will need to be outlined in the proposal.

6. BUDGET

An appropriate budget including detailed expenditure estimated is to be submitted along with bids. Bidders are requested to be very clear about methodology, providing a detailed breakdown of costs for the different significant activities to be undertaken during the evaluation and survey. The budget range for the evaluation is expected to be between $10,000 – $20,000, all included. Bidders are strongly encouraged to compete on the basis of their commercial proposal, demonstrating value for money, as well as technical proposal.

7. PAYMENT

There will be three payments for each evaluation/survey enumerated as follows:

20% of fees for Evaluation/ Survey upon submission of the inception report

50% of fees for Evaluation/ Survey upon presentation of the draft evaluation/ survey report

30% of fees for Evaluation/ Survey upon submission of final report and all documentation as required by The MENTOR Initiative for final payment