External Evaluation of BHA-funded Multisector Assistance Program in Eastern DRC

A. Background
Concern Worldwide is a non-governmental, international, humanitarian organization dedicated to the reduction of suffering and working towards the ultimate elimination of extreme poverty in the world’s poorest countries. Concern has been working in the Democratic Republic of Congo since 1994, implementing emergency response, WASH, food security, nutrition and livelihoods projects. With the support of USAID’s Bureau of Humanitarian Assistance (BHA), Concern is providing multi-sector assistance to vulnerable households in North Kivu and Tanganyika with the following goal: to ensure that disaster and conflict-affected communities in DRC are physically and psychologically safe and better equipped to recover from shocks.

B. Description of the Social, Economic and Political Context
The humanitarian context in DRC has continued to deteriorate due to armed conflict and natural disasters that have led to large displacements especially in the Eastern provinces. These shocks restrict access to essential goods, services and infrastructure for internally displaced persons (IDP), returnee and host populations and disrupt regular and secure access to food, health, water, sanitation and hygiene (WASH), and shelter. In 2023, over 1.2 million people in Tanganyika and 2.4 million people in North Kivu are in need of humanitarian assistance with women, children and people living with disabilities uniquely under threat. Armed groups are prevalent in North Kivu, with kidnappings, armed operations and targeted assassinations driving population movements, particularly in Masisi Territory. The volatile context combined with high levels of inflation, insufficient food production and restricted access to services are directly contributing to the high levels of vulnerability.
The DRC remains the world’s leading country in terms of the number of acutely food insecure people with 27 million people affected. The people in crisis and emergency phases (IPC phase 3 or higher) are mainly found in the east with 31% of the population in Tanganyika in IPC3 or more, and 2.6 million people in IPC 3+ in North Kivu. Nearly 26 million people are in Crisis IPC 3+ between July and December 2023 in all territories of North Kivu and Tanganyika. Conflict and displacement affect already weak WASH infrastructure in returnee, host communities and displacement sites with 31% of people having no access to a clean water source, more than 56% using unimproved toilets and 12% having no access to basic sanitation facilities. This increases the risk of deadly waterborne diseases with regular reporting showing cholera epidemics in Tanganyika (46 cases and 2 deaths reported) and cases of the plague and cholera in North Kivu. IDPs and returnees secure housing needs are classified as catastrophic in Masisi, Nyunzu and Kalemie Territory and an immediate felt consequence is increasing protection risks.

C. Description of the Subject for Review/Evaluation
The overall goal of the BHA-funded project is to equip disaster and conflict-affected communities in DRC both physically and psychologically in order that they might recover more effectively from shocks. The project’s theory of change posts that: ‘If multisector humanitarian assistance is provided to IDPs and a community-based multisector early recovery response is provided in areas affected by displacement and return, then the condition of the most vulnerable households will improve immediately and sustainably, and their capacity to withstand new shocks will improve’. The overarching purpose of the activity has been to meet the need for immediate multisector support and ongoing community resilience through a gender and protection focused response.
Building on the experience of both Concern and fellow consortium member, ACTED, the 24-month program provided integrated multi sector assistance tailored to local needs (with IDPs prioritized in Masisi and returnees supported in Tanganyika) using a community-centric approach. An alert-based emergency first line response was enacted in crisis affected areas with a view to improving the condition of displaced and returnee communities in the immediate. At the same time, complementary early recovery assistance, including Farmer Field Schools and Income Generating Activities, WASH capacity building, transitional shelters, fostered early recovery of all community members affected by displacement and return.

Sub Sectors

• Improved agricultural production and unconditional food assistance
• Water supply, sanitation, hygiene promotion and WASH non-food items (NFI)
• Shelter
• Temporary employment, livelihood restoration and financial services

D. Purpose and Objectives of the Evaluation

The purpose of the evaluation is to assess to what degree Concern’s BHA-funded program has been successful in achieving the established results and objectives as illustrated in the program proposal and to identify future program options to provide sustainable assistance to IDPs and returnees. The evaluation findings will be compared with the baseline data collected during assessments, routine monitoring data, and other information collected as part of the intervention. The collected information will be used to establish good practices and help formulate new interventions in the program areas. More specifically, this end-of-program evaluation seeks to:

• Assess progress made towards the achievement of objectives, indicators and targets and the appropriateness of the targeting strategy used in this context.
• Measure the extent to which proposed activities adhered to planned implementation and achieved intended goals, purposes and outcomes.
• Support the organization’s commitment to accountability to donors, government, public, and beneficiaries through assessing good practices, gaps and incorporating learnings into future programs.
• Highlight the extent to which changes—expected and unexpected, positive, and negative—occurred because of the proposed program and the impact they had on targeted participants, community members and other stakeholders.
• Assess the relevance, efficiency, the effectiveness, and impact of the program using DAC criteria.
• Identify lessons learned and provide practical, innovative, and sustainable program options for assisting returnees with their integration and/or relocation.

Tentative Key Evaluation Questions
The proposed external evaluation will address the following tentative key evaluation questions:

Relevance

• To what extent were Multisector Assistance for Crisis-affected Areas in Eastern DRC humanitarian interventions in-line with the self-identified and international standard needs and priorities of the target groups, including IDPs, returnees, and vulnerable host households?
• Has the program addressed the immediate food security, WASH, livelihoods, protection and shelter needs of the targeted households? To what degree will access to these services be improved on a long-term/sustainable basis?
• Has the program improved resilience of affected populations?

Efficiency

• How did the budget utilization compare with the planned budget?
• What was the comparative cost of the program vs. program outputs?

Effectiveness

• To what extent were the objectives of the program, as reflected in the program proposal, achieved?
• What changes had to be made to the initial proposal in order to adjust to unforeseen circumstances, if any? What are the lessons learned from these experiences?

Impact

• Has the program helped to decrease the prevalence of negative coping mechanisms to which targeted populations resorted due to lack of resources and services to be provided through the program?
• Has the program allowed households to meet their basic needs?

Sustainability

• Did the program fully involve the beneficiaries and other stakeholders in the program implementation process?
• Did the program promote ownership of outputs and outcomes by actively involving beneficiaries in decision-making and action at every phase of the program cycle?

Consortium & Partnerships

• Were there concrete and attainable goals for the consortium itself (beyond the program objectives)? Was there common understanding of these goals?
• Was there a sharing of resources, skills and expertise within the consortium and among partners?
• Was there a dedicated consortium governance structure? What were the successful factors and challenges of the program’s governance structure?
• How and to what extent did the consortium strengthen the capacity of civil society actors (formalized CSOs and/or informal groups)?

Mainstreaming

• Was due consideration given to gender equality and ethnic groups so that program intervention did not increase target groups vulnerability and tension among different ethnic or otherwise vulnerable groups?

M&E and Project management

• What were the strengths, weakness and challenges of the M&E activities of the program, particularly in terms of data quality assurance?
• What are the lessons learned from current gaps identified throughout the program cycle? What are the main action points from this learning to be incorporated into future programmes?

A score will be assigned to each criteria using the following grading scale:
5 – Outstanding Performance
4 – Very good overall performance with a few shortcomings
3 – Good overall performance but with some minor shortcomings
2 – Generally acceptable performance but with some major shortcomings
1 – Barely acceptable performance with many major shortcomings
0 – Totally unacceptable performance or insufficient data to make an assessment.
Crosscutting approaches including Conflict Sensitivity, Protection, Disaster Risk Reduction, the Environment will be considered through the evaluation, and the Evaluation Report will include a section covering these crosscutting approaches, outlining which approaches were considered and how successfully they were addressed in the program.

E. Evaluation Methodology
The evaluation will use a mixed methods performance evaluation with both quantitative and qualitative data. To do so the evaluators will:

• Conduct a review of all quantitative data on program indicators, including comparison of baseline and end line data, and an analysis of these data in reference to the evaluation questions set out above. Furthermore, the evaluators will review the performance monitoring data throughout the award to assess the systematic approach of results based monitoring within the Award. This review will be based on Concern collected data throughout the award.
• The qualitative data collection will be led by the external evaluators. The design of the qualitative approach will address the evaluation questions through a range of approaches, including Focus Group Discussions with beneficiaries, Key Information Interviews (KIIs) with staff and other stakeholders. In addition to addressing the evaluation questions set out above, the qualitative data collection will explore the issues identified through review of the quantitative data. Key areas of analysis will include efficacy of the CRM system, sustainability of food assistance beyond either rounds of assistance, and gender or Persons with Disabilities (PwD) barriers to inform future program design. Focus group discussions with program participants and key informant interviews with project stakeholders will be the primary means of generating qualitative insight into these questions, paying attention to different societal structures, demographics, and gender or age considerations.
• The sampling approach for the primary data collection, given its qualitative focus, will be non-probabilistic and purposive, as it is intended that this primary data will raise a diverse range of perspectives on particular aspects of the implementation of the Award. In particular, the evaluation team is invited to propose a sampling strategy that specifies separate samples for focus group discussions with beneficiaries of each sectoral intervention of the project (food assistance, shelter and settlements, WASH and protection), and this sampling strategy must include consideration of female beneficiaries and persons with disabilities, to ensure their perspectives are fully included in the evaluation. While the evaluation team is invited to propose a sampling strategy, including sample size, based on their analysis of the project and Scope of Work (SOW), it is expected that the evaluation team will define a sample size in an effort to reach saturation against the relevant evaluation questions. Similarly, the evaluation team is invited to propose a purposive sampling approach to interviewing staff and other stakeholders towards saturation of findings against the relevant evaluation questions. While the evaluation team will carry out their sampling in an independent manner, Concern will provide full facilitation and cooperation as needed to realize the sampling strategy and ensure that the evaluation data is collected as intended.

a. Desk-based Research/Preparation
The evaluation team will be expected to make a complete review of program proposal, progress reports, monitoring visits repost, distribution records and other supporting documentation. The evaluators will rely on Concern’s quantitative data collected during the implementation of the Award as per the indicators of the BHA award.
In addition, a report will be published by other humanitarian organizations and will be reviewed to situate the Concern project within the wider context of humanitarian assistance provision in the program area.

b. In the Field
Concern’s M&E team in country will facilitate the evaluation by arranging interviews and discussions with a sample of program participants and non-participants and staff through household interviews and focus group discussions (FGDs). Key informant interviews with other key stakeholders – including program participants, local authorities, humanitarian actors, and the relevant clusters – will also be carried out to corroborate findings from FGDs. The external evaluators will be responsible to propose a sampling methodology to Concern. The M&E team will also collect the case studies/human interest stories (HIS) and pictures to display the program outcomes for internal and external audiences.

c. Post-Field Analysis
A time period will be made available post-field work for the evaluator to analyze and review data collected, to prepare a draft report and to finalize subsequent drafts based on feedback from partners, Concern and other potential peer reviewers.
As part of Concern’s accountability to beneficiaries. Concern will ensure the findings of the evaluation is disseminated to beneficiaries. On receiving the final evaluation, Concern’s MEAL team will structure community consultations in all locations to share with beneficiaries. It is expected this process will take place within a one month window, with delegated levels of responsibility. Area Coordinators will share findings with the respective local authorities within their area of responsibility and the CD will share the results with the Humanitarian Affairs Office. The work plan will be developed by the Programme Quality and Development Coordinator (PQDC) with clear lines of responsibility and timelines for communication.

Limitations
Concern anticipates at this stage the following limitations of the evaluation:

• The time requirement to complete the evaluation within the period of performance of the award will place a time pressure on the evaluators. As the evaluation is a mixed quantitative and qualitative performance evaluation, this may pose a challenge to the external evaluators due to the limited time flexibility. Concern, will thereby seek to have the quantitative data ready to provide the evaluators with sufficient time to analyze the data and identify the areas of focus for the qualitative assessment.

F. Outputs/Deliverables
The evaluator will be fully responsible for the following:

• An abbreviated inception report, detailing methodology for data collection – qualitatively and quantitatively, checklist and questionnaire for data collection, work plan and timelines for submission and review. This inception report will include full detail of proposed sampling approach, including sample sizes, detailing the number of focus group discussions and key informant interviews for each category of respondent.
• A first draft of the Evaluation report with an executive summary and clear recommendations (complete less appendices) for comment from Concern DRC within 1 week of concluding interviews.
• A presentation of findings to the Concern country team and HQ upon completion of the draft Evaluation Report.
• A full final draft of the Evaluation report, integrating the feedback received within one week of receiving consolidated feedback on the draft report.
• The report, in English, should be 10-15 pages long without appendices and should be submitted in electronic format (Word or PDF) to the country program and Desk Officer, and RD and include the following sections:
  o Executive Summary (maximum 2 pages)
  o Brief context and description of the intervention
  o Presentation of evaluation methodology and any limitations encountered
  o Presentation of main findings, conclusions and recommendations using graphs, charts and tables where appropriate
  o Recommendations with a strong focus on the wider lessons learned for Concern to improve future emergency multi-sector programme planning and implementation, and Management Reponses
  o Scoring against the extended DAC criteria.
  o Annexes: including ToR, list of people/groups consulted, interview frameworks/ questionnaires, tools, list of sites visited, abbreviations, maps, charts or graphs that may have been used for this evaluation.

All information should be disaggregated as per the PRIS for the BHA award, including were permissible and logical presenting the disaggregation per location. The analysis should integrate the quantitative data outcomes, with the issues identified to inform future intervention strategies to tackle the barriers of humanitarian interventions to improve the impact of programs.

G. Provisional Timeframe
The evaluation process is expected to take place over an 8-week period. This duration is inclusive of the time required for all work including document review, field visits, debriefing, and final submission of the evaluation report. Submission of the final evaluation is expected to start in March and be completed in June 2024.

Tentative time allocation subject to evaluator work-plan (maximum no. of days)

• Briefing and background reading and preparation of tools and methodology (4)
• Drafting of abbreviated inception report (5)
• Internal and External interviews (18)
• Analysis and compiling of draft evaluation report (19)
• Debrief/presentation of findings (country team and HQ) upon completion of draft (4)
• Complete final report (7)

H. Lines of Communication
The external evaluator will report to the Programme Director and work closely with the Program Quality and Development Coordinator, the National M&E Coordinator, the Program Manager and the Desk Officer in Ireland. For data collection in the field, Concern’s and ACTED’s M&E managers will be the external evaluator’s main points of contact. Each organization’s M&E manager will communicate any feedback, changes or challenges to Concern Worldwide’s program manager in the DRC.

I. Essential Experience

• Technical expertise relevant to evaluating Multisector BHA programmes involving Food Security, Livelihoods, WASH, Shelter and Protection.
• Experience of qualitative data collection methods, including participatory approaches and experience of data collection in the field.
• Ability to assess the quality of programmatic data collection systems, analyse budgets, results frameworks, M+E reports and surveys, and other documents.
• Fluency in French.
• Experience of writing evaluation reports to a high standard, in English.
• Experience of working as part of an agile team.
• Experience within the humanitarian response in DR Congo.

J. Desirable

• An understanding of results-based management approaches.