Evaluation consultant for Nachbar in Not Program – Emergency Assistance to Ukraine and neighbouring countries

CONTEXT AND BACKGROUND:
On 24 February 2022, a new phase of the International Armed Conflict (IAC) in Ukraine started with the intensification and spread of the hostilities, from then onwards affecting the whole country and causing major concern for the protection of civilians and essential civilian infrastructure. This came after months of rising political and military tensions and eight years of conflict in and around Eastern Ukraine where an estimated 2.9million people were already in need of humanitarian assistance and 1.4million were internally displaced.

To date, by September 2023, millions of people have been displaced, inside and outside Ukraine and a total of approximately 17.6 million persons in Ukraine need humanitarian assistance, affecting all regions of the country. With low chances of the IAC ending soon, the humanitarian situation inside Ukraine continues to deteriorate, both for still unfolding immediate needs given the volatile IAC situation, as well as for the mid-to long-term need to support recovery.

In spring 2022, the Austrian Federal Government decided to provide EUR 41.96 million from the Austrian Foreign Disaster Relief Fund (“Auslandskatastrophenfonds/AKF”) to the NACHBAR IN NOT (NIN) Foundation for funding of Humanitarian Aid Projects in response to the humanitarian crisis in Ukraine and neighboring countries. To that end, the Austrian Development Agency (ADA) and NIN signed a Grant Agreement covering the period 15 June 2022 – 14 June 2024. (“NIN-ADA Program”). This grant is also referred to as the “AFK-ADA Funding”.

The NACHBAR IN NOT Foundation, with funding appeals to the Austrian public, has, since its establishment 31 years ago raised mainly private donations from the Austrian public (around EUR 310 million since 1992 for various humanitarian crises). It is set up of a Board of Directors and eight permanent member organizations (permanent “cooperation partners”) who implement the funds raised by the Foundation. They are the Austrian CSOs: Austrian Red Cross, CARITAS, Malteser Hospitaldienst Austria, Diakonie Katastrophenhilfe, CARE Österreich, Arbeiter Samariter Bund (ASBÖ), Hilfswerk International and Volkshilfe. In addition, the NIN Foundation can entrust temporary “project partners” with the implementation of the funds (i.e. additional CSOs).

This time, in response to the war in Ukraine, for the first time, NIN received government funding from the Austrian Federal Government: the EUR 41.96 million for the “NIN -ADA Program” indicated above.

This NIN-ADA Program, which is the focus of this evaluation, is part of the wider “NIN Ukraine Response” of the Foundation, which at the time of writing (September 2023) has raised over EUR 98 million (EUR 56.2 million private donations and EUR 41.96 million “AKF-ADA Contribution” from the Austrian Government). Out of this total, EUR 73.4 million have been contracted by September 2023, and these EUR 73.4 million include EUR 41.5 million of the government funding, which means that 100% of the operational budget of the government funding has already been contracted. With those EUR 41.5 million, 21 humanitarian assistance projects have been or are being implemented at the time of writing. These projects are implemented by altogether 9 organizations: by 7 of the 8 NIN members and 2 additional Austrian CSOs as temporary project partners (Jugend Eine Welt and World Vision Austria).

The Projects under the “NIN-ADA Program” take place in Ukraine and Moldova. As stipulated in the Grant Agreement between NIN and ADA, their focus is on the following 8 thematic areas: 1) Water and Sanitation, 2) Food and Nutrition, 3) Cash and Voucher Assistance, 4) Protection, 5) Shelter, 6) Psychosocial Support, 7) Medical Assistance, and 8) Education in Emergencies. These thematic areas also constitute the programmatic frame of the “NIN-ADA Program”. The large part of the projects is implemented in various different parts of Ukraine[1], 5 projects partially or completely take place in Moldova.

PURPOSE, OBJECTIVES AND INTENDED USERS

The overall purpose of this evaluation is twofold:

• Accountability: To give account of the use of public funds and the results achieved by the program to the donor (i.e. ADA and the Austrian Government) and to the Austrian public at large.
• Learning: For the Nachbar in Not Foundation (the Board, the Cooperation and Project Partners) to support the organizational accountability and learning and contribute to the continual improvement of NIN’s performance in addressing the humanitarian needs and providing humanitarian solutions and protection for people affected by disasters and conflicts.

The objectives of the evaluation are summarized below:

• To evaluate the NIN-ADA Ukraine Response Program’s performance in terms of appropriateness and effectiveness.
• To determine the reasons for achieving intended or any unintended results of the individual projects funded by the NIN-ADA program and draw lessons for the remainder of the “NACHBAR IN NOT Emergency Action Ukraine” (NACHBAR IN NOT UKRAINE HILFSAKTION) and for similar future emergency response actions of such a scale, complexity and expected length.
• To identify key lessons and potentials for strengthening response capacities of implementing partners and local communities.
• To identify potential areas for civil society advocacy and collaboration on protection of women and girls affected by the conflict.

The Intended Users of the Recommendations of the Evaluation are:

• Donor /Government Representatives:
  • The Austrian Development Agency (ADA) as the donor representative and the operational unit of the Austrian Development Cooperation.
  • The Austrian Foreign Office as the body responsible for administering the Foreign Disaster Aid Fund (AKF).
• The Nachbar in Not Foundation:
  • The Board
  • The Implementing Partners – The permanent member CSOs and the temporary project partners and their local partner organizations.

SCOPE:
The evaluation aims to focus on the following four thematic areas of the NIN-ADA Program – Cash and Voucher Assistance, Shelter, Protection[2] and Psychosocial Support [3].

For NACHBAR IN NOT, those are the most interesting and most important ones in terms of both accountability and learning: Cash is considered the preferred option for similar future interventions, however, for several of the local implementing partners this modality is new. Shelter is chosen because it is a large component of the intervention; Protection and Psychosocial Support are chosen because a) a large part of the target group are women and children and the aim is to receive findings on the extent to which their needs in those areas have been met (accountability); b) Psychosocial Support and dealing with and alleviating trauma is a key issue in many conflicts; thus findings of this evaluation can be used by the NACHBAR IN NOT members to focus and plan interventions in similar contexts.

In terms of geographical coverage and sampling of projects, the Evaluators will be requested to suggest a sample of projects in both Ukraine and Moldova as part of the Inception Reporting. Most of the 21 projects under the “NIN-ADA Program” are multi-sectoral. At the time of writing 11 of the projects include components of “Provision of Cash and Voucher Assistance”, 12 of the projects include components which contribute to “Provision of (Emergency) Shelter”; 4 of the projects include components aimed at “Strengthening Protection” of vulnerable groups affected by the war and displacement and 6 of the projects include “Provision of Psycho-Social Support” as one result area. These numbers may change as projects are adapted to changing needs and to the evolving situation on the ground; for instance, in 2023 some project interventions were modified to shift their focus to provision of Cash and Voucher Assistance in place of originally planned and provided material assistance.

As outlined in Chapter 5 below and with a view to the conflict environment in Ukraine: on-site data collection is strongly preferred as much as the security situation allows, however, it is understood by the contracting party that a hybrid approach to data collection may be required in areas which are inaccessible due to security risks.

It is expected that as part of the discussion of the inception reporting, the evaluators will then work with NIN to develop a final sample of projects to be evaluated, to ensure key NIN considerations are accounted for and thus included in the sample, and the final sample will be approved by NIN.

The main criteria guiding the evaluation will be:
– The Appropriateness of the interventions (Quality Criterion 1 of the Core Humanitarian Standard (CHS)).[4]
– The Effectiveness[5]of theinterventions, with the extent to which desired results have been achieved to date.

Both criteria are key for the NACHBAR IN NOT Foundation which commissions this evaluation, to know whether “we are doing the right thing” and “whether the intervention is achieving its objectives”.

Findings and recommendations concerning both criteria will be important for both NACHBAR IN NOT and the back donor (ADA) and will be key to provide accountability and make suggestions in terms of learning.

EVALUATION QUESTIONS:
Ad 1. Appropriateness:

• To what extent has the Humanitarian Response provided under the NIN-ADA Funding in the areas of a) Cash and Voucher Assistance, b) Protection Psychosocial Support and d) Shelter been appropriate and relevant to the needs of the affected population?
• To what extent have these focus areas been in line with the needs of the affected population?
• To what extent have the respective projects been designed based on solid context analyses[6] and needs assessments?
• To what extent have the specific needs of women, girls, boys and men affected by the war and displacement been considered and responded to by the project interventions?
• How appropriate to strengthening local partners’ capacities was the design of interventions in the four sectors which are the focus of this evaluation? How were adaptations to changing needs been made on project level? To what extent has the program frame allowed for adaptations?
• To what extent is there a need for further focusing (in terms of sectors, target groups and geographic prioritization)?

Ad 2. Effectiveness:

• To what extent have the program and the individual projects (to date) achieved their intended results, with a focus on the four Result Areas of a) Cash/Voucher, b) Protection, c) Shelter, d) Psychosocial Support?
• From among all the vulnerable groups in this conflict and displacement crisis, to what extent have especially vulnerable women and children benefitted from the results to date?
• What factors were decisive for achievement/non-achievement of the intended results in the 4 sectors indicated above?
• To what extent have interventions contributed to strengthening local capacities?

DESIGN AND APPROACH:

The evaluation will give attention to the integration of gender, protection and accountability to affected populations in the program’s response, and on differential effects on men, women, girls, boys.

The evaluation will use secondary qualitative and quantitative data complemented with primary data collection as necessary and feasible. The methodology should demonstrate impartiality and lack of bias by relying on a cross-section of information sources and using a mixed methods approach to ensure triangulation of evidence. The evaluator is expected to work on two main levels: (a) organizational level (learning about the NIN implementers’ service delivery capacity (ability to deliver services in accordance to standards) and (b) also related to beneficiaries (using participatory approaches to ensure the voice of beneficiaries is included in the design).

After an initial desk review in the inception phase[7], for data collection, there is a strong preference for the evaluators to conduct interviews or focus group discussions onsite, in Ukraine and Moldova, wherever possible. Where not feasible, due to the security situation, applying hybrid methods is possible. The Evaluation Team will be expected to collect the data themselves (i.e. avoiding approaches using third-party data collectors).

Tentative WORKPLAN & Deliverables:
1. Signature of Contract – 15 December 2023

2. Kick off (Vienna or online) – 8/9 or 11/12 January 2024
Initial briefing, defintion of main questions and finalization of ToR
Estimated working days: 2 (lead expert)

3. Inception Report (max. 10 pages plus annexes) – 31 January 2024
Desk Review and Inception Report with Annexes (eg. evaluation matrix, approach, selection of projects to be evaluated)
Estimated working days: 8 (lead expert)

4. Feedback to /Clearing of Inception Report – 28 February 2024
Adaptation of Inception Report according to Feedback.
Estimated working days: 1 (lead expert)

5. Presentation and Discussion of Inception Report and Planning Field visits (preferably in Vienna) – 11/12 March 2024
Presentation to NIN and cooperation partners
Estimated working days: 2 (lead expert), 4 days (local teams)

6. Field Phase – March/April 2024
Data Collection/ Project Visits in Ukraine and Moldova
Estimated working days: 20 (lead expert), 40 days (local teams)

7. Draft Final Report, incl. Results Assessment Form – 15 May 2024
Estimated working days: 5 (lead expert)

8. Feedback to /Clearing of Final Report – 10 June 2024

9. Final Report, incl. Results Assessment Form – 20 June 2024
Adaptation of Final Report according to Feedback
Estimated working days: 1 (lead expert)

10. Debriefing /Presentation of Results (Vienna) – end of June 2024
Estimated working days: 1 (lead expert)

EVALUATION MANAGEMENT ARRANGEMENTS:

Evaluation Management lies with the NACHBAR IN NOT (NIN) Office in Vienna. NIN’s assigned evaluation manager will represent NACHBAR IN NOT during the evaluation and will work together with a Reference Group which has been established for this evaluation.

The evaluation manager is responsible for: guidance throughout all phases of the evaluation; co-ordination of the review process of key documents delivered by the evaluators; approval of all deliverables.

The Consultant is responsible for: 1) conducting the evaluation; 2) the day–to–day management of the evaluation (including travel organization, security and duty of care for the whole team (e. g. proper insurance); 3) reporting of progress or challenges to the evaluation manager; 4) the development of recommendations; 5) the production of all deliverables in accordance with contractual requirements.

REQUIREMENTS FOR THE EVALUATORS:
The Evaluation Team is expected to be made up of one senior evaluator as team lead and national evaluation experts who bring to this evaluation the following expertise and experience as required:

• Evaluation Expertise and Experience:

For the Team Lead: Extensive experience as evaluator in a humanitarian/development context, proven by at least three evaluations as team lead; substantial experience as team leader in conflict settings and track record of working together with local evaluation teams; advanced degree in social sciences or relevant related field; strong experience in qualitative data collection and analysis; proven experience applying direct and online data collection methods.

• Thematic and Geographical Expertise/Experience: Solid knowledge of Ukrainian context, strong skills in evaluation design, qualitative data collection and analysis, drafting and editing in English. Strong communication and time management skills as well as cultural sensitivity are required, as is a Commitment to Quality and Quality Assurance. Track record of applying a human-rights based and gender-responsive approach to evaluations.
• Language Skills: Excellent spoken and written English- (a must for the Team Lead), with Ukrainian or Russian being considered an asset. Ukrainian /Russian and very good English for Local Evaluation Team Members in Ukraine and Moldova.
• Gender Balance: Not least due to the fact that the large part of project participants are – mostly Ukrainian – women and children who have been affected by the war and displacement, it is paramount that the (local) team is gender-balanced.

The evaluators may come into contact with beneficiaries who have protection concerns and/or are traumatized. Therefore, all evaluators must sign a Code of Conduct for safeguarding and also ensure that the evaluation process has safeguards so that questions and discussions do not re-traumatize beneficiaries.

None of the evaluation team members must have been involved in the design or implementation of the program or any of its projects.

Legally registered consultants and companies are eligible to apply.

[1] Projects are currently implemented in more than 15 Oblasts, from Lviv, and Transcarpathia in the West to Kharkiv, Zaporizhja, Donetzsk in the East.

[2] For this program Protection is defined as: “Strengthening the safety, dignity and rights of the vulnerable population (affected by the war and displacement)”. See also the EC definition:

https://civil-protection-humanitarian-aid.ec.europa.eu/what/humanitarian-aid/protection_en

[3] This means that it is neither expected nor planned to fully assess all projects nor the program’s contribution to all 8 thematic result areas.

[4] The definition of Quality Criterion 1 of the CHS: is: “Communities and people affected by crisis receive assistance appropriate and relevant to their needs”. – For the purpose of this evaluation, the wording “Appropriateness” is used for this criterion, in keeping with the ALNAP definition of “The extent to which humanitarian activities are tailored to local needs, increasing ownership, accountability and cost-effectiveness

accordingly” (replacing the relevance criterion used in development evaluations.) -(Source: ALNAP ,- https://www.alnap.org/system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016.pdf, p. 113)

[5] „Is the intervention achieving its objectives?” (ADA Evaluation Guidelines, p.16, in reference to the OECD DAC Evaluation Criteria”; Similarly: “The extent to which an activity achieves its purpose, or whether this can be expected to happen on the basis of the outputs.” (Source: ALNAP ,- https://www.alnap.org/system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016.pdf, p. 113).

[6] https://phap.org/theme-context-analysis

[7] Data available on the level of the NACHBAR IN NOT Foundation includes number of beneficiaries (gender-disaggregated) per Result Area, allocations of projects to the thematic sectors/result areas and geographical allocation of projects on Oblast/Rayon level. In addition, the Austrian partners of NACHBAR IN NOT and their local partners in Ukraine and Moldova would collect different kinds of data based on their specific expertise and internal policies.