Concern Worldwide
1. Purpose of the Evaluation
This is the Final Evaluation of the 2 year 2020- 2022 ECHO Pilot Programmatic Partnership (PPP) funded programme in DRC. PPP is a new funding modality developed by ECHO to test new ways of working with humanitarian partners, in line with its Grand Bargain commitments to increase humanitarian aid efficiency and effectiveness.
This country level evaluation will form part of an overall evaluation of the ERNE programme being implemented by Concern Worldwide in 5 countries.
The overall evaluation will be led by a team of external evaluators, henceforth known in this document as the “global evaluators” and will be conducted in 2023 after completion of the ERNE programme by all 5 countries (although the programme was designed to be implemented across all 5 countries over the course of 3 years, DRC funding was not renewed for the third year of the multi-country programme). This country level evaluation will be led by country level evaluator/s recruited through this ToR.
The findings and final report from this country level evaluation will be used by the global evaluators to assess the overall ERNE programme performance against the DAC criteria. A high level synthesis report will also be produced by the global evaluators bringing together key learning from the overall ERNE programme to provide evidence-based recommendations for the future- for both Concern (as implementer) and ECHO (as donor). This country level evaluation of the ERNE programme in DRC will also be used by Concern Worldwide’s country team for learning and in order to feed into the design of future programming.
2. Description of the Context
A complex emergency has persisted in DRC for more than 20 years. Population displacement is frequent and repeated, and mostly driven by armed clashes and intercommunal violence between foreign, self-defence, and other armed groups. DRC is among the poorest countries in the world, ranked 175 out of 189 countries on the 2020 Human Development Index.
In 2020-2022, the population of DRC has continued to face pressing humanitarian needs Armed conflict and violence, epidemics, natural disasters, and the socio-economic impact of COVID-19 have considerably exacerbated already existing vulnerabilities, in a context marked by a structural lack of access to essential services. . With 5.2 million displaced persons, DRC currently hosts the second largest number of internally displaced persons in the world. The country also hosts 527,000 refugees from neighbouring countries. Food insecurity has reached the highest level ever recorded. According to the Integrated Phase Classification (IPC) analysis, the country has the largest number of highly food insecure people in the world. Around 27 million people in the Democratic Republic of Congo (DRC) are experiencing high levels of acute food insecurity (IPC Phase 3 or above) between September and December 2021, of which around 6.1 million people are experiencing critical levels of acute food insecurity (IPC Phase 4). More than 3.4 million children under the age of five suffer from acute malnutrition. The DRC’s risk for 2020 on the INFORM index is rated in the “very high” category, particularly in relation to vulnerability and a lack of coping capacity in the country.
The entire Manono territory is currently in IPC 3 and is expected to remain at this level until June 2022. This is confirmed by the REACH Factsheet (2021) which found that the decrease in one meal during the day was a coping mechanism frequently used by the communities of Ankoro, Manono and Kiambi. These three health zones, and specifically the health zone of Kiambi, where the ERNE project is being implemented, are characterized by high rates of severe acute malnutrition (on average 2% in the territory) (HDX, 2020). In addition, chronic malnutrition is also a public health problem in the territory, with more than 50% of children under five suffering from it (SMART survey 2019 and 2021). These high malnutrition rates are linked to poor child feeding practices, high levels of food insecurity, poor water, hygiene and sanitation (WASH) practices, inadequate access to health services. The context of the Kiambi health zone being isolated, with roads in very poor condition, the absence of a telephone network, and natural barriers, further aggravates the health of children. Thus, according to the Humanitarian Needs Overview (HNO 2022), Manono is one of the territories where the risk of deterioration of the situation is highest, and classified by the nutrition cluster as “high priority.
Concern Worldwide has been working in DRC since 1994. Concern Worldwide teams intervene in the sectors of Water, Hygiene and Sanitation, Nutrition, Food Security and Livelihoods, Shelter, NFI, and Gender & Protection with a focus on the Eastern part of the country.
With programmes in North Kivu and Tanganyika, Concern Worldwide DRC is developing and implementing programmes along the full emergency continuum, ranging from rapid response and humanitarian assistance to early recovery assistance and development interventions. Concern’s objective is to alleviate poverty and hunger and provide high quality assistance through innovative programmes with a strong focus on transforming existing gender norms and integrating protection principles.
3. Description of the ERNE Programme in DRC
With Enhanced Responses to Nutrition Emergencies (ERNE), Concern addresses the scale, efficiency and effectiveness of its nutrition emergency responses through an integrated, 3 pillar programme approach:
- Pillar 1 (results 1 and 2) focusses on life saving interventions by ensuring the provision and increased quality of primary health and nutrition services using the CMAM approach and cash transfers/ vouchers/ in-kind.
- Pillar 2 (result 3) focusses on the preparedness and the local capacity to respond. The aim is to reduce the impact of nutrition and food security shocks by enabling people and systems to anticipate and prepare.
- Pillar 3 (result 4) is focusing on understanding and addressing the underlying elements that make people vulnerable to undernutrition. The aim is to increase nutrition resilience.
In addition, under result 5, Concern will draw out learning from the technical approaches implemented, within the CMAM Surge Task Force and through implementing the ERNE programme under the Pilot Programmatic Partnership (PPP) modality with ECHO.
The ERNE programme in DRC has been implemented in Kiambi Health Zone, in Manono Territory. In Year 2 the programme has expanded its activities from eight to ten Health Facilities of Kiambi Health Zone, covering physically a total of eighteen health areas out of the twenty-six in the Health Zone. ERNE field teams are based in these 18 health areas, where accessibility and security allowed to do so. In these 18 health areas, the team has been supporting daily implementation, all CMAM, primary health care, and nutrition sensitive activities. Remote support in primary health care (provision of essential drugs through the Kiambi HZ supervisors) has been provided in the remaining 8 HFs, where the security situation continues to make accessibility difficult.
Expected outcomes
The programme aims to achieve the following outcomes:
Principal Objective: To contribute to reducing malnutrition morbidity and mortality among children under 5
Specific objective: To increase scale, efficiency and effectiveness of humanitarian preparedness and response for food and nutrition security in targeted countries
Result 1: Increased coverage and improved quality of treatment of acute malnutrition (CMAM) in regions with the highest prevalence
Result 2: Severe deterioration of food security of most vulnerable households affected by crises is avoided
Result 3: Shocks in targeted districts are anticipated and prepared for.
Result 4: Improved behavioural practices impacting nutrition of children under 5 in target communities
Result 5: Learning and evidence from the ERNE programme influences dialogue and policy debate amongst the wider humanitarian community.
4. Evaluation objectives and scope
The objective of this country level evaluation is to provide an assessment of the ERNE Programme in DRC against each of the DAC criteria that will feed into an overall evaluation of the entire ERNE Programme. The selection of criteria reflects the ‘Double Nexus’ nature of the ERNE programme (integrated programming with elements of both emergency response and resilience building). Moreover, the evaluation includes questions investigating the relationship between the PPP funding modality and programme implementation. The evaluation should focus on the design, implementation, activities and results of the ERNE programme in DRC and will cover the time period from June 2020 to August 2022. This evaluation will also be used by Concern Worldwide’s country teamfor learning and in order to feed into the design of future programming, and should therefore offer concrete and targeted recommendations for improving programming going forward.
5. Evaluation questions
Core Final Evaluation questions
Sub-questions
Relevance (including Appropriateness for Emergency pillar>
Were the objectives and the design of the intervention relevant and appropriate to the context and to the needs of recipients?
Was there a clear decision making process in place when deciding to respond?
Were beneficiary needs considered sufficiently when planning the responses?
Are there any examples of adaptations to the evolving context? Can they be considered as successful?
Did the integrated/ Nexus programming Programme Framework hold true?
Did the causal pathway prove to be true?
What assumptions were made? Did these assumptions hold true? Were other factors important? What were these factors?
Did Concern adapt appropriately in response to changes in the context and lessons learnt during the implementation of the programme?
The what extent do ERNE objectives and design respond to beneficiaries’, global, country, and partner/institution needs, policies, and priorities, and continue to do so if circumstances change?
Did the PPP enable appropriate programme adaptations to be made? If so, what type of adaptations took place and did the approvals process facilitate making these adaptations?
Has the PPP supported the increased use of cash-based programming?
Effectiveness
Is there evidence that anticipated results are being achieved at the current stage of the programme?
What is the change observed at the different levels of the programme result chain (CMAM/Health/WASH, CMAM Surge, Nutrition Resilience, EWEA preparedness)? For changes observed, to what extent can they be attributed to the programme?
Was the programme achieving the planned outputs and outcomes and were they on time?
What is the evidence of achieving the response outputs and immediate outcomes?
Were the responses coordinated effectively with external stakeholders (e.g., clusters, other NGOs and agencies, relevant government units, etc.)?
To what extent does ERNE achieve, or is expected to achieve, its objectives, and its results, including any differential results across groups?
Has the multi-year funding target, and enhanced emergency response capacity, and use of innovative approaches ((such as CMAM Surge) offered under the PPP enabled Concern to effectively achieve its ERNE objectives and results?
Has the PPP enabled multi-year arrangements with partner organisations? What have been the enabling/blocking factors? What has been the impact of these arrangements (for example, have they improved relations with local partners)? Has a capacity-building element been incorporated into the partnership arrangements?
Efficiency
Did the intervention use the available resources in the most efficient manner to achieve its outputs?
Did the intervention have sufficient and appropriate resources?
Did the intervention pay competitive prices for its inputs?
Were there any alternatives for achieving the same results with fewer inputs? If/where possible compare the cost units across 5 countries for the same approaches as well as the cost of alternative variants of programming adaptation aiming for the same result
To what extent does ERNE deliver, or is likely to deliver, results in an economic and timely way?
Has the PPP increased predictability of funding and how has this impacted efficiency (for example, through reducing operational costs such as procurement, logistics or administrative costs or increasing staff retention).
Have the reporting requirements under the PPP enabled simplified and harmonised reporting? If not, why not?
Does ERNE use the available resources in the most efficient manner to achieve its outputs?
Did the PPP modality enhance the emergency response capacity of the ERNE team? How so? How timely were rapid responses to shocks?
Impact
Did the intervention achieve the intended impacts?
Is there evidence of the intended change at impact level (positive, negative)? If so, what contributed to this change? If change did not occur, why not?
What are the wider effects of the programme on: individuals; groups (gender, age, disability); communities; institutions; or conflict dynamics? Were they positive or negative?
Were there any unintended consequences as a result of the programme implementation? Were they positive or negative?
To what extent has ERNE generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects?
Did the funding model allow ERNE to help in crises, for example, by reducing a particular problem or risk or by supporting people so that they can cope better?
Connectedness
To what extent are longer-term problems given consideration during implementation of emergency activities within ERNE?
Were the emergency activities carried out in a way that considered longer-term programmes and relevant approaches?
To what extent did the programme support and prepare communities to withstand future emergencies?
Did the PPP enable investment in fragile contexts, emergencies and disaster preparedness?
Coherence
Was there consistency between programme’s activities and the relevant Concern and industry standard policies?
Was the intervention informed by and aligned with the fundamental humanitarian principles? Did the programme adhere to the Red Cross Code of Conduct, Sphere, Core Humanitarian Standard and sector-specific standards, where appropriate?
Was there evidence that the intervention adhered to relevant Concern policies (e.g., Approach to Emergencies, Nutrition, Health, WASH, Equality, Protection, Child Safeguarding Policy, HIV and AIDS)?
To what extent does the PPP modality enable better humanitarian-development coherence in ERNE countries?
How did the programme enable complementarity between humanitarian response and longer-term development interventions?
Coverage
To what extent does ERNE reach the most vulnerable?
Is there evidence that Concern’s interventions reached the most vulnerable?
How has the programme targeted those in greatest need at all levels (geo, ind. etc.)?
How did conflict and gender analysis inform the response design and implementation?
Cross- cutting issues
To what extent did the programme design, implementation and monitoring consider relevant Concern cross cutting issues of Conflict Sensitivity, DRR, Protection, HIV, Partnership, Equality, Environment.
How did the programme consider the needs of women, men, boys and girls, people living with HIV and AIDS, and people with disabilities?
What was the effect of the Covid-19 pandemic on Concern’s ability to deliver its intended results in this programme?
6. Methodology
This evaluation of the DRC ERNE forms part of a group of ‘nested’ evaluations taking place at different times, in different countries and conducted by different teams, as laid out in the table below.
Global Evaluators
- Global Evaluation Synthesis Report
The country level reports feed into the above final report
Country Level Evaluators for DRC (recruited through this ToR)
- DRC/ Ethiopia/ Niger/ South Sudan/ Sudan Country Level Evaluation Report
The country level evaluations will use a mixed method evaluation approach. The Country Level Evaluator will lead the evaluation process, conduct primary data collection, analyse findings and produce the evaluation report.
The Country Level Evaluators will be expected to develop an appropriate work plan for the country level evaluation that is based on the global evaluation approach AND relevant to the country context. The work plan must be as participatory as possible under the current conditions, involving different stakeholders including beneficiaries, partners, Concern staff and other stakeholders.
The prescribed methodology will include the following activities:
- Detailed document review including Country Context Analysis; ERNE Programme Framework; ERNE DRC Programme Toolkit (which should include updated M&E plan); Quarterly and Interim Reports; Annual Review documentation
- Assessment of achievement of indicators under each outcome using existing quantitative data from Concern Digital Data Gathering (DDG) system, summarised in the Zoho Dashboards.
- Key Informant Interviews with Key Programme staff; partner staff; relevant local authorities; key programme participants/community leaders
- Focus Group Discussions with programme participants (appropriately disaggregated).
In the month prior to departure the country level evaluator must attend a meeting with the ERNE Programme Coordination Unit ME Advisor. This is to ensure good understanding of the global programme architecture and ensure access to all centrally maintained programme resources.
7. Expected products/outputs (including critical elements to be addressed in a final report)
Final Report
The outputs of this country level evaluation will include a final country level evaluation report which will contain:
- An executive summary (no more than 3 pages)
- An appraisal of how well the programme has fared against each of the DAC criteria using the following grading scale, where:
- 5 – Outstanding/Exceptional Performance
- 4- Very good overall performance (above expectation)
- 3 – Good overall performance (as expected)
- 2 – Acceptable performance but with some shortcomings
- 1- Barely acceptable performance with some major shortcomings
- 0- Totally unacceptable performance or insufficient data to make an assessment
- Targeted recommendations for improving future programming
The report should follow the template provided in Annex III and should be no more than 25 pages (15 pages focused on findings and conclusions against DAC criteria). [TH1] [TH2] The report submitted by the Country Level Evaluator will be reviewed by the Global Evaluators using Qualitative Comparative Analysis alongside the data validation exercises to increase the robustness of the evaluation process and confidence in the findings at a global level (see Annex IV).
Briefings and Presentations
An in-country briefing with Concern Worldwide country teams should be held to give a preliminary overview of the main findings. A virtual meeting (e.g. Zoom; Microsoft Teams or Skype) should also be held with country team to discuss feedback on the draft Evaluation report, before the report is finalised and signed off by the Regional Director. A PowerPoint presentation (or similar) must be prepared for each briefing and shared with the DRC Desk Officer, DRC evaluation focal points and ERNE ME Advisor.
Evaluation Data
The Country Level Evaluator will provide as part of the report a list of references to all quantitative and qualitative data sources, along with any data sets used in data analysis included in the final report, as well as notes, recordings, images, etc. from key informant interviews and focus groups for validation and quality assurance.
8. Reporting lines
The Desk Officer for DRC, will lead the overall coordination of the Country Level Evaluation. They will be supported by Programme Coordination Unit ERNE M&E Advisor. Both will be key contacts for logistics, training and preparation for this evaluation.
In-country reporting will be to the ERNE Programme Manager, and liaise with the Programme Quality and Development Coordinator and Programme Director.
Other members of Concern Worldwide including the Programme Coordination Unit (PCU) and Programme Approaches and Learning Unit (PALU) will also provide support to the country evaluators on an ongoing basis during the evaluation.
9. General conditions of the consultancy
- Concern will provide accommodation for the consultant in team guesthouses while in Kalemie and Kiambi. In Goma, the consultant will be in a hotel.
- Meals and other incidentals will be the responsibility of the consultant.
- The consultant will conduct his/her work using his/her own computer.
- Other miscellaneous costs directly related to the task that can include photocopying of questionnaires, etc. shall be covered by Concern. Any costs need to have prior approval from Concern.
- The movement of the consultant and team to and from the field will be facilitated by the Concern/ACTED.
- The cost related to travel from/ to field of the survey teams will be covered by Concern.
10. Composition, skills and experience of the Evaluation team
The roles, responsibilities and skills and experience of the country evaluator(s) are laid out below.
ERNE Country Level Evaluator
The ERNE Country Level Evaluator/s will have the following responsibilities:
- Lead liaison with PCU and in-country stakeholders.
- Lead planning and preparations for country programme data collection and collation, ensuring all permissions necessary for conducting data collection and collation have been granted and are adhered to (including any ethical clearance/permissions).
- With CWW country programme staff and programme implementers identify key informants for interview and focus group participants, coordinate and support stakeholder engagement
- Conduct in-country data collection (observations, KIIs and FGDs etc.) and collate the data
- Facilitate key in-country meetings/workshops with CWW country programme staff and other key stakeholders were necessary
- Conduct ERNE country programme evaluation data analysis and interpretation
- Write ERNE country programme evaluation report
The ERNE Country Level Lead Evaluator should have the following skills and experience:
- 5 years’ experience of conducting evaluations of country programmes
- Technical expertise relevant to the ERNE country programme
- Proven experience of qualitative data collection methods, including participatory approaches
- Proven experience of working remotely with a distributed team, including providing support to colleagues at a distance
- Ability to assess the quality of programmatic data collection systems, analyse budgets, results frameworks, M+E reports and surveys, and other documents
- Strong knowledge of Core Humanitarian Standards (CHS)
- Good spoken and written French & English. Swahili is a plus.
- Experience of writing evaluation reports to a high standard, in English
- Experience of working as part of an agile team
- Excellent knowledge of the DRC context, especially in terms of security and culture, is required
11. Plan for Evaluation implementation (including timelines)
The evaluation of the ERNE programme in DRC will be carried out from 22nd August 2022.
Duration of evaluation will be approximately 4 weeks and will include:
- 0.5 week desk review of documents ;
- 2 weeks field work to include initial orientation with teams, all primary qualitative data collection, preliminary analysis and briefing with country teams on initial findings and recommendations ;
- 1 week for data analysis and to produce draft report ;
- 0.5 week for virtual presentation and incorporating feedback in Final Evaluation Report.
During the evaluation the country level evaluators will have access to the Programme Coordination Unit team, including the Global ERNE ME Advisor to ensure consistency of process and to talk through and agree solutions to challenges that arise.
In advance of departing for the evaluation mission the Country Level Evaluator/s, with support from the country team, will develop an evaluation plan, including a timeline and methodology. This will be submitted to the ERNE DRC Programme Manager, DRC Desk Officer and ERNE ME Advisor for review and approval no later than two weeks before planned travel to DRC.
Final report structure:
-
- Executive Summary – Project background, with a project overview and a description of the theory of change – Methodology, with details on: – Logic model – Data collection methods – Data collection tools – Sampling – Quality control – Data analysis – Limitations of the evaluation – Evaluation results.
The evaluation report will include an appraisal of how well the programme has fared against each of the DAC criteria using the following grading scale, where:
- 5 – Outstanding/Exceptional Performance
- 4 – Very good overall performance (above expectation)
- 3 – Good overall performances (as expected)
- 2 – Acceptable performance but with some shortcomings
- 1 – Barely acceptable performance with some major shortcomings
- 0 – Totally unacceptable performance or insufficient data to make an assessment
- Conclusions and recommendations
- Lessons learned
- Annexes, including the ToRs, the evaluation matrix and full detail of tools used during the evaluation.
How to apply
Please submit an Expression of Interest by 24/07/2022 to [email protected] outlining exact availability in line with the approximate timeline. The expression of interest should contain:
1. Technical offer:
- Up to date CV of the consultant(s) and cover letter explaining how the consultant(s) meets the skills and experience required.
- Technical proposition detailing proposed methodology and resources needed (max 3 pages).
- At least one example of an inception report and one example of a final report from similar work, which demonstrates evidence of the skills and experience, required.
2. Financial offer:
- A list of all expenses expected to be incurred by the consultant including a daily rate.
- Costs of transport in-country and accommodation will be covered directly by Concern and should not be included.