EVALUATION INTERNATIONAL MULTI-SECTORAL RESPONSES UKRAINE

CONTEXT
The humanitarian response in Ukraine and neighboring countries, was led from the start of the invasion by its national and local governments, while civil society and local communities have played a dominant role in the actual implementation of aid activities. Existing and well-established non-profit organizations as well as improvised citizen/private volunteers’ initiatives mushroomed throughout the country. Together they managed to respond to an important share of the needs.

In addition, there was a large-scale response by the international community. Governments committed important material and financial resources, and the general public has donated vast sums of money. UN agencies significantly strengthened their presence in Ukraine and while most international humanitarian organizations (INGOs) were not working in Ukraine at the start of the invasion, many established a presence in the first weeks and months after.

SV’s initial response to the Ukraine crisis was to support international partners. Two of them, PIN and PAH, were new to SV but had been working in Ukraine since 2014. The third was SV’s worldwide partner Intersos that newly established itself in Ukraine. Two others, MAG and Mukwege Foundation, were also new to Ukraine but already SV partners in other contexts.At a later stage SV started to work with Dutch volunteer initiatives and with a series of Ukrainian NGOs and volunteer networks. From January 2023 SV has focused its support largely on Ukrainian civil society organizations. SV established a small representative office to support this choice, while at the same time SV took over responsibility for a large-scale medical supply program from one of the Dutch volunteer initiatives.

While the Ukrainian context was new for SV, SV has prepared for the response and has taken a whole series of decisions concerning organizational set-up, partners and project choices. The main objective behind each question for learning is how effective and relevant SV’s aid provided, directly or through partners, was for those who were in need of assistance.

Against this background, SV’s evaluation plan for the Ukraine response differs from regular project evaluations. Since project evaluations often do not offer much learning in terms of governance, processes, decision-making, and strategic approaches, this evaluation plan takes a programmatic approach – to combine project-based learning with more strategic level learning. For the evaluation to offer learning opportunities on this level, a general learning objective has been identified, and due to the scope of the Ukraine response, this learning objective is divided into different thematic focuses and evaluation-sub questions (the evaluation plan will be made available after expressed interest). As part of the overall meta evaluation required by SHO, different teams will be evaluating different sets of projects. This will allow the evaluation plans to cover a wider range of topics and identify learning opportunities at a more strategic level. Consistency will be ensured by including the identified sub-questions in different ToRs. This enables the implementation of project-based evaluations, while also feeding into the larger learning objective.

As part of this larger evaluation plan, this particular ToR will focus on themulti-sectoral responses by SV’s International NGO partners.

PRESENTATION OF THE PROJECTS TO BE EVALUATED – INTERSOS
RRM, Shelter, Health – March 2022 to November 2022

• Code: UKR-22.5
• Type of evaluation (see methods section): light-touch medium-term effects evaluation (follow-up)1
• Title: Provision of material assistance (NFIs and HKs) and health support to the IDPs in Vinnytsia and Poltava Oblasts, through integrated protection approach
• Duration: March 2022 to October 2022
• Project Locations and target groups: 16.400 IDPs in Vinnytsia and Poltava Oblasts, and RRM locations
• Total budget: 1.078.386 EUR
• Sectors of intervention: RRM, Health
• Local partners: Myrne Nebo and Hub Vokzal (Kharkiv), Light of Hope (Poltava), Svitlyy Spravy (Mykolaiv)
• Types of activities: 3 rounds of distribution of NFIs and hygiene kits, various RRM activities (NFIs, hygiene kits, medications, drinking water, etc.)
• Project stakeholders: INTERSOS, SV, beneficiaries, hospitals, local authorities
• Project specific objectives: Provision of material assistance (NFIs and HKs) and health support to the IDPs in Vinnytsia and Poltava Oblasts, through integrated protection approach
• Expected results / indicators: please refer to the project proposals and details (provided in inception phase)

Shelter/health, partnership – March 2022 to October 2022

• Code: UKR-22.6
• Type of evaluation: light-touch medium-term effects evaluation
• Title: Support to health facilities and shelters in Lviv Oblast for IDPs fleeing the conflict in Ukraine
• Duration: March 2022 to October 2022
• Project Locations and target groups: 35.677 IDPs in Lviv, Ivano-Frankivsk and Ternopil Oblast
• Total budget: 385.321 EUR
• Sectors of intervention: Shelter, health
• Local partner: 100% Life
• Types of activities: Shelter and services in Lviv, mobile team for services in other collective centers (medical/PSS consultations, hygiene kits, etc.), rehabilitation children’s hospital Lviv
• Project stakeholders: INTERSOS, 100% Life, SV, beneficiaries, hospital, local authorities
• Project specific objectives: IDPs fleeing Eastern and Central Ukraine are in safe and secured environments while transiting in Lviv and provided with Primary Health Care support, Protection activities, Legal consultations and NFI / Children have a safe and secure space were seeking refuge during the alerts threat
• Expected results / indicators: please refer to the project proposals and details (provided in inception phase)

Shelter, RRM – November 2022 to November 2023

• Code: UKR-22.09
• Type of Evaluation: End-term
• Title: Provision of emergency assistance to IDPs in Lviv and Eastern Oblasts of Ukraine
• Duration: November 2022 to November 2023
• Project Locations and target groups: 13.262 IDPs in Kharkivska, Zaporizhka, Dnipropetrovska,Donetska, Lviv, Donbas
• Total budget: 2.000.000
• Sectors of intervention: Shelter, RRM
• Local partner: 100% Life
• Types of activities: NFIs, winterization (collective centers), RRM, collective center support (e.g. PSS, meals, etc), , mobile teams
• Project stakeholders: INTERSOS, local partners, SV, beneficiaries, local authorities
• Project objective: to contribute to increased access to essential services in underserved conflict-affected areas and areas with high numbers of IDPs, providing a critical, life-saving intervention to protect the most vulnerable from the consequences of the conflict
• Expected results / indicators and activities: please refer to the project proposals and details (provided in inception phase)

PRESENTATION OF THE PROJECTS TO BE EVALUATED – PIN

• Code: UKR-22.2
• Type of Evaluation: light-touch medium-term effects evaluation (follow-up)
• Title: Provision of emergency assistance to IDPs in Lviv and Eastern Oblasts of Ukraine
• Duration: March 2022 to August 2022
• Project Locations and target groups: x IDPs in collective centers
• Total budget: 757.500 EUR
• Sectors of intervention: Shelter
• Local partners: Posmishka
• Types of activities: NFIs and educational materials to collective centers
• Project stakeholders: PIN, local partner, SV, beneficiaries, local authorities
• Project objective: To reduce the impact of the armed conflict on the people of Ukraine and ensure access to relevant needs-based humanitarian assistance
• Expected results / indicators and activities: please refer to the project proposals and details (provided in inception phase)

PRESENTATION OF THE PROJECTS TO BE EVALUATED – PHA

• Code: UKR-22.3
• Type of Evaluation: light-touch medium-term effects evaluation (follow up)
• Title: Emergency multisectoral assistance to conflict affected people residing in Ukraine
• Duration: March 2022 to November 2022
• Project Locations and target groups: 6.831 IDPs in Dnipro, Zaporyska, Donetska, and Lviv
• Total budget: 750.000 EUR
• Sectors of intervention: Shelter, FSL
• Local partners: NGO Light of Culture (Dnipro), Posmishka UA (Zaporyska)
• Types of activities: Food kits, MPCA, house repair, NFI
• Project stakeholders: PHA, SV, beneficiaries, local authorities
• Project objective: To reduce suffering and improve well-being of conflict-affected people residing in Ukraine through increasing ability of conflict-affected people to cover their basic needs
• Expected results / indicators and activities: please refer to the project proposals and details (provided in inception phase)
• Code: UKR-22.10
• Type of Evaluation: end-term evaluation
• Title: Emergency multisectoral assistance to conflict-affected people residing in Ukraine
• Duration: December 2022 to December 2023
• Project Locations and target groups: 3960 IDPs in Dnipropetrovska, Kharkivska and Donetska Oblast
• Total budget: 1.000.000 EUR
• Sectors of intervention: Shelter/NFI, MPCA, WASH, Protection
• Local partners:
• Types of activities: MPCA, winterization (cash – fuel), small WASH infrastructure rehabilitation, PSS
• Project stakeholders: PHA, SV, beneficiaries, local authoritiesProject objective: To reduce suffering and improve well-being of conflict-affected people residing in Ukraine through increasing their ability to cover their basic needs with MPCA, Shelter, WASH and Protection assistance.
• Expected results / indicators and activities: please refer to the project proposals and details (provided in inception phase)

PRESENTATION OF THE PROJECTS TO BE EVALUATED – OPEN DOORS UKRAINE

• Code: UKR-23-0018
• Type of Evaluation: end-term
• Title: Help Ukrainians build a roof above their head
• Duration: January 2023 to September 2023
• Project Locations and target groups: 3715 returnees in Irpin in Kyiv oblast (province), Trostyanets (Sumy oblast) and Balakliia and Oskil (Kharkiv oblast)
• Total budget: 246.540 EUR
• Sectors of intervention: Shelter
• Local partners:
• Types of activities: repairs (schools/houses), generators, furnishing IDP shelters
• Project stakeholders: ODU, SV, beneficiaries, local authorities
• Project objective: Improving living conditions in war torn Ukraine
• Expected results / indicators and activities: please refer to the project proposals and details (provided in inception phase)

EVALUATION OBJECTIVES

The purpose of the evaluation:
The purpose of the evaluation is to learn from the first 1,5 years of the Ukraine response, assess the impact of the projects, learn from the response of international partners, and define recommendations for SV and the partners on how to make future responses more relevant and effective.

General objective:
Evaluate the effectiveness and relevance of SV and its partners’ diverse assistance approaches and organizational set-ups, such as direct implementation, subgrants and partner support, and in-kind donations, and assess SV’s role in enhancing their effectiveness for the benefit of the affected population. To what extent did the modality of the project approach, organizational set-up and/or type of partnership have 1) an impact on the way the needs of the affected population were responded to, 2) an (unintended) impact on the lives of the affected population, the work and relationships of local partners and other local stakeholders?

Specific questions on multi-sectoral responses for all sectors, including related sector questions where relevant:

Relevance, effectiveness, timeliness and (unintended) effects:
1.
How well did the response address the needs identified by of the affected population? To what extent did the program activities effectively respond to immediate humanitarian needs in a timely manner? How well did they adapt to the changing context and emerging needs during different program phases?
RRM: To what extent did the RRM contribute to a more agile, flexible, relevant and local response? What lessons can we gather from both internal and partner perspectives regarding its innovative nature?
2.
What positive/negative changes were observed in the lives of the affected population as a result of the response?
Shelter: Did the shelter program contribute to the overall well-being and protection of the affected population? And how did the different approaches to shelter between our partners have an impact on the affected population (e.g. in terms of appropriateness to the needs, adaptation to the context, cost-efficiency, etc. )?
3.
What were the key factors that contributed to the success or challenges of the response in delivering assistance? And what role did the affected population have in shaping or commenting on the successes and challenges of the response?
Shelter: What lessons can be learned from shelter project experiences, and how can improvements be identified for future programming?
RRM: What effects did procurement challenges have?
4.
How has SV’s expertise/mandate and partnerships (including SV’s presence) affected the quality of the partners projects?
5.
What has been the impact of the partner’s presence and response on local stakeholders? (e.g. on local partners, other local organizations, authorities, relationship between communities and authorities, etc.)
6.
For analysis: Has there been a difference in relevance, effectiveness and timelines in projects implemented by partners with a strong presence in Ukraine vs partners that were new to the context?

For each project and/or partner, learnings from the practices and effectiveness of accountability to the affected population (information sharing, complaints and feedback mechanisms etc.) and their participation in the response and decision-making should be defined, and the following partnership questions integrated:
1.
How did SV’s approach impact the decision-making power of its partners concerning the design, implementation and modality of delivery?
2.
How did (local) partnerships promote the inclusion and meaningful participation in the decision-making of local actors and marginalized groups within the affected population?
3.
How did the partners consider the local embeddedness/representativeness of local partners, and impact power dynamics and relationships between different local stakeholders? And how has this affected the response?

EVALUATION METHODOLOGY
The evaluation team, consisting of a local and/or international consultant(s), is expected to employ a methodology for unbiased results, designed to contribute to the general learning objective by semi-structured evaluation questions for all projects, in order to aggregate the findings.

The evaluation can be divided into two types. Both types of evaluations follow the evaluation questions above, but could use different methods. The first concerns a light-touch evaluation for projects that have ended in 2022 (UKR-22.5 & UKR-22.6 INTERSOS, UKR-22.2 PIN and UKR-22.3 PHA) which focuses more on the processes of the start-up phase, how relevant and effective the projects were, and include case studies of the experiences of, and impact of the project on, beneficiaries and local partners. Since beneficiaries will often be difficult to trace, KIIs with local stakeholders, outcome harvesting with partner staff and/or local stakeholders, and case-study approach might be more appropriate.

The second is a final evaluation for projects that have ended (or are about to end) in 2023 (UKR-22.09 INTERSOS, UKR-22.10 PHA and UKR-23-0018 ODU), focusing on the effectiveness of the projects, taking a strong people-centered approach. It is required that this evaluation approach revolves mostly around consultations and participation of beneficiaries and local stakeholders. Methods could include KIIs, FGDs and/or a survey. Furthermore, the partnership questions could be evaluated in a different way, such as KIIs with local partners and a facilitated reflection workshop.

Since there are projects with similar activities, such as PSS and support to collective centers by INTERSOS and PHA, and aims (with different activities), such as shelter support through NFIs in collective centers, MPCA or direct repairs, a degree of comparison should be included in the evaluation concerning the relevance, timeliness and effectiveness of the projects. During the data analysis process, it is important to pinpoint specific details that pertain to the partner, project, and/or local partner, where possible. This ensures a clear differentiation in how each project and partner contributed to the overall outcomes. Additionally, if there are noted variations between partner and projects, it is essential to provide a thorough explanation for these particularities.

Before the collection of data by the chosen methodology, the evaluator should formulate a causal chain hypothesis, identify possible other factors that influence the results, and validate this causal chain, while checking the effects of other factors. Furthermore, the evaluation conclusions should
focus on providing insight into the reasons ‘why’ a particular finding was observed rather than solely describing them.

Data collection methods could include:
For the inception report:
1.1.
Desk-based literature review: meta-analysis and summary of reports addressing humanitarian context, needs, missed opportunities, etc; analysis of project proposals and workplans, baseline and existing MEAL data, performance indicators and budgets and other broader regional and contextual background material
1.2.
Secondary data analysis: analysis of data already collected by the project staff

For the evaluation:
1.1.
Semi-structured interviews: key informant interviews with staff, local stakeholders, direct beneficiary groups, and the affected population. Interviews can take place in person or remotely. Since the response has changed significantly since the start and initial beneficiaries have moved away, including different groups will allow a reconstruction of the first months and the reflections of future developments.
1.2.
If desired, a small survey around central evaluation questions, to solicit a wider range of perspectives and uncover/test cause-effect correlations.
1.3.
(Participatory) Focus group discussions (in person and/or remotely): facilitated small group meetings on specific issues and topics to solicit comments and responses around key issues arising from interviews and secondary data analysis. Preferably using more participatory methods.
1.4.
Direct observation of project interventions and relationships between different stakeholders for 2023 projects.
1.5.
Meetings and consultation with a wide range of other key stakeholders such as partner and SV staff but also external stakeholders who were of priority in the implementation of a coordinated intervention. An outcome harvesting and/or after-action review session with partner staff is encouraged.
It is not required for the consultant to visit all project locations, but there should be a clear strategy for selecting the focus areas in the application and inception report.

The method of analysis depends on the chosen methodology and should be included in the inception report.

STAKEHOLDER INVOLVEMENT
The principal intended users of the evaluation are SV, the partners, local partners, as well as SHO as the project funding source. The partners are expected to share relevant findings with the affected population and local stakeholders.

The partners and SV staff will provide the evaluation team with all the necessary background documentation and data available from the projects to conduct desk-based assessments in preparation for the fieldwork phase. Partner staff will accommodate the evaluation team’s needs and facilitate the fieldwork including access to the relevant communities and project sites for data collection and setting up meetings with beneficiaries, community leaders, local authorities, relevant partner staff and other key stakeholders. The evaluation team will coordinate with SV MEAL advisor and Ukraine Team Lead and should discuss any technical and methodological matters that arise.

The evaluation team is expected to hold a post-fieldwork meeting with SV and partner staff and other stakeholders to present and discuss preliminary findings. Draft reports are expected to be shared with partner staff, HQ staff and SV staff to incorporate feedback and ensure factual correctness before final versions are signed off prior to completion of the final evaluation report. The final version of the evaluation report will be provided to both SV and shared with the partners.

EXPECTED DELIVERABLES
The evaluation team is expected to deliver the following key deliverables:

• Inception Report (of a maximum of 5 pages) produced by the lead consultant within two weeks of contract being signed detailing the evaluation team’s research design and intended methodologies (for data collection and analysis), activity workplan, team responsibilities and overall analysis of the desk-based review and secondary data from partners;
• Project Evaluation Report with detailed findings and recommendations specific to this project;
  • Summary of Evaluation Report for wider distribution (including in-country sharing with local stakeholders, such as beneficiaries, local authorities, etc.)
  • The Project evaluation report must be written in English and not exceed 25 pages (excluding annexes). The report must be written in Word and submitted in one stamped paper and one electronic copy. The draft report will need to be presented to SV for final input and fact checking before the final version is submitted.
  • The written report should be based on (but not limited to) the following framework:
    1. An executive summary (2 pages maximum)
    2. A methodology section outlining the inception report content and planning process, make-up of the team (e.g. sectoral expertise, local knowledge, gender balance) and its appropriateness and independency for the evaluation and any potential biases.
    3. A constraints, limitation (including reflexivity) and challenges of the evaluation section
    4. A context analysis of the overall humanitarian response in Ukraine and how the projects are situated within it.
    5. A findings section in line with the set evaluation questions discussed in detail
    6. A recommendations and lessons learned section following the analysis of the findings and the discussion with key stakeholders of the preliminary findings. It will be structured with specific subsections for each evaluation question, partner and project, ensuring that findings are tailored to each context/question providing general conclusion for each and any observed differences are explained.
    7. List of Annexes, including to the minimum: the present ToR, the inception report, sample of the data collection tools used, a list of sources of information used/documents reviewed, any relevant graphic material that could not be placed in the report.

QUALIFICATIONS
The selection will follow a competitive bidding process, which will be based on a request for questions (inclusive both of a technical and financial offer) published online on relevant website(s). Following a
pre-screening of the received offers, those meeting the minimum requirements will be ranked on both the technical and financial offers, following also a panel interview of the shortlisted consultants/firms.

SV will make every effort to ensure there is no conflict of interest on the basis of the consultants selected in order to guarantee independence and objectivity. The consultants have to possess a multidisciplinary mix of evaluative and programmatic skills, qualifications and experience.

Specific skills and experience sought:

• Significant demonstrable experience in the evaluation of humanitarian projects and/or sociology and anthropology work, including field-based data collection, analysis and reporting and experience of conducting desk-based evaluations emergency response projects within a conflict environment
• Extensive knowledge of humanitarian contexts, preferably with knowledge specifically on Ukraine and/or lived experience.
• Relevant academic qualifications and training in the related fields.
• Significant experience with qualitative research methodologies, particularly involving semi-structured interviews, focus groups and qualitative analysis.
• Excellent communications and presentation skills.
• Ability to write clear and sensible reports (examples of previous work shall be submitted).
• Fluency in English and Ukrainian is essential.
• Understanding of national humanitarian mechanisms and context is preferable
• Ability to manage the available time and resources and to work on tight deadlines
• Independence from SV, partners, project stakeholders or other parties involved
• National consultants/institutes are particularly encouraged to apply.

No data can be used by the consultant concerning this study without the written permission of the partners. The consultant acting as service provider will make sure to present itself as such for all discussions held within the framework of the consultancy.

The consultant immediately agrees to respect all specific security instructions of the partners and most recent Government directives as well as to operate in line with the partners’ PSEA and child protection policies.