Project name: Life-saving humanitarian aid for the conflict-affected population and refugees in Ukraine and neighboring countries, with a focus on the elderly, women, and children (later in text: Ukraine humanitarian response)
Project location: Ukraine, Moldova
Project duration: March 2022 – April 2023
Donor: German Federal Foreign Office
Implementing organization: ADRA Germany with ADRA partners from Ukraine and Moldova
BACKGROUND
As an aid organization ADRA Germany e. V. provides support for people in need worldwide – regardless of ethnic origin, gender, religion or belief, disability, age or sexual identity. The state-approved non-governmental organization was founded in 1986 and is part of a global network with over 118 independent country offices and around 7,500 full- time employees. The organization has been conducting development cooperation and humanitarian aid projects for over 30 years. ADRA Germany is a member of the German relief network (ADH) and carries out projects financed with own funding, ADH funding as well as with institutional donors (GFFO, BZM, EU etc.). The main office with around 70 employees is located in Weiterstadt near Darmstadt, with a second office in Berlin.
Ukraine humanitarian response
ADRA Deutschland e.V., together with its local partners, the ADRA offices in Ukraine and Moldova, is implementing since March 2022 a cross-sector, life-saving humanitarian aid project financed mainly by the GFFO. Belonging to the global ADRA network with shared values and support systems, each ADRA office is an independent organization. The project builds on the existing humanitarian presence of local partner organizations and is in line with the UN Flash Appeal, which aims to address the key needs of IDPs and left-behind affected people in Ukraine and refugees crossing the border. The interventions of the project are focusing on 3 key areas:
- survival packages (food and non-food items (NFI) in kind or cash) for IDPs and left-behind affected persons in Ukraine and refugees in neighboring countries,
- protection, psychosocial support and legal counseling for IDPs and left-behind affected persons in Ukraine and refugees in neighboring countries,
- safe shelter components for IDPs and left-behind affected persons in Ukraine and refugees in neighboring countries. In accordance with the Grand Bargain, the project is contributing to localisation, which means a high degree of autonomy of the local offices towards implementation.
In the context of displaced persons, according to UNHCR statistics, there are 5.8 million internally displaced persons in Ukraine by January 2023. Refugees outside the borders of Ukraine in neighboring countries are: 109,000 are registered in the Republic of Moldova, 486,000 in the Czech Republic, 1.5 million in Poland, 107,000 in Slovakia, 34,000 in Hungary and 50,000 in Bulgaria.
The communication between ADRA Germany and the offices in Moldova and Ukraine was supported by on-side visits as well as regular online meetings with project managers and project staff. While ADRA Germany was already cooperating with ADRA Ukraine in the years 2014-2017 along the conflict line with activities such as PSS, light repair (financed by GFFO), the cooperation with ADRA Moldova on a large-scale project and with an institutional donor was new.
The project has been planned in two phases:
Phase 1: 1-6 months – Focus on immediate needs to provide life-saving assistance through multisectoral interventions in Ukraine and neighboring countries to ensure that affected target groups have access to food, other essentials, and shelter, as well as psychosocial support and legal counseling. Conducting feasibility analyses for cash interventions in Ukraine determined the modality required, and the intensity of the emergency response continued to build over six months.
Phase 2: 6-12 months – Focus on providing rents and shelter rehabilitation and improvement support to ensure they meet minimum standards and requirements. Accompanying emergency assistance via cash and in-kind distributions due to emerging emergencies in this context and activities adapted in response.
Project Objective: Provide emergency life-saving humanitarian assistance to vulnerable internally displaced persons and populations left behind in the conflict region in Ukraine, and support women, children, and the elderly fleeing the war and seeking refuge in the neighboring Republic of Moldova.
Activity overview:
Ukraine: Multi-purpose cash assistance, Evacuation, Mobility assistance, Psychosocial individual and group support. Psychosocial support (referral mechanisms), Winterization kits distribution, Transit centres support, Cash for rent, Light repair
Moldova: Multi-purpose cash assistance, Evacuation, Psychosocial individual and group support, Psychosocial support (referral mechanisms), Winterization kits distribution, Transit centres support
PURPOSE OF THE EXTERNAL EVALUATION
The following project components are part of the evaluation.
A: Multi-purpose cash assistance B: PSS Component
C: Shelter Component
The purpose of the Evaluation with tentative date in April 2023 is to assess the impact, effectiveness, efficiency and accountability1 of the GFFO project to develop lessons learnt for use in future work, and programme design and development. The study will also serve as an analytical tool on the connectivity to follow-up project. The project evaluation will be externally presented Ukrainian clusters and humanitarian actors as well as to the donor German Federal Foreign Office. ADRA will also present the report to the affected population, local authorities and other relevant stakeholders.
The lessons learnt / recommendations will be incorporated into our institutional learning and their outcome will be used for future projects.
The aims of this evaluation are to:
- Examine the relevance and effectiveness of the project in relation to the project goals and objectives
- Identify good practices;
- Identify gaps and areas of unmet needs both in activities and from a cross-cutting perspective;
- Examine the appropriateness of project activities for vulnerable groups; and
- Produce “lessons learnt” for review of implementation approach and for planning future projects.
The evaluation of the Ukraine humanitarian response will be oriented on OECD DAC criteria and conducted in order to determine the level of
- Relevance – the extent to which the objectives2 of the Intervention are consistent with beneficiaries’ requirements, country needs;
- Coherence – how well does the intervention fit into the humanitarian context in Ukraine and Moldova
- Effectiveness– the extent to which the Intervention objectives were achieved, or are expected to be achieved
- Efficiency – how well are resources being used, that can be claimed by the Intervention.
- Evaluation Questions
Key evaluation questions for the Evaluation include the following:
In reference to the outputs:
- To what extent did ADRA interventions contribute to meeting the needs of vulnerable people in the context of financial assistance for daily needs of food, medicine, rent, utilities, and clothing?
- To what extent did ADRA interventions support the well-being of beneficiaries through psychosocial support in the context of their ability to psychologically cope with the difficulties of wartime?
- To what extent did ADRA interventions contribute to the target population getting safe shelter in the context of financial assistance for rents and through light repair of buildings?
In reference to the OECD DAC criteria:
- Relevance: Do the goals of the project address the priorities and needs of the partners and the target groups? Have gender and diversity aspects been taken into consideration during planning and implementation? How visible are the project and the German contribution in comparison to other the contribution of other actors?
- Coherence: To which degree has the project taken contributions of other actors into consideration? Has there been duplication/coordination?
- Efficiency: To which degree have the project goals been achieved? What are reasons for partial achievements or non-achievements? Have there been unintended impacts (positive/negative) and if so what has been done? Which external factors have positively/negatively contributed to the efficiency? Is the monitoring system sufficient to capture changes in the political context, risks and problems during implementation? Can the project easily react on this and do steering adjustments?
- Effectiveness: Is the project management effective? What is the relationship between invested funds and achieved goals?
- Connectivity: Is the project able to connect to other actors towards a long term perspective? Which are the opportunities and risks for connectivity of the project?
The final evaluation should also reflect relevant Core Humanitarians Standards, focusing on (but not limited to) CHS 1, 2, and 3 in evaluating the project’s impact:
- CHS 1 Quality Criterion: Humanitarian response is appropriate and relevant;
- CHS 2 Quality Criterion: Humanitarian response if effective and timely;
- CHS 3 Quality Criterion: Humanitarian response strengthens local capacities and avoids negative effects.
Finally, the evaluation should also reflect on ADRA and their future role in the Nexus in Ukraine by seeking answers to these guiding questions:
-
- Did the project plan and implement an adequate transition and exit strategy that ensures longer-term positive effects and reduces risk of dependency? – How well are the project’s outputs linked to more long-term focused objectives?
The results of the evaluation will be used for potential project adjustments, learning management and future project interventions in humanitarian context. ADRA will utilize the evaluation findings for both improving the quality of work for accountability purposes and for lesson learning, and more specifically for improving compliance with the Core Humanitarian Standards (CHS) and other standards relating to accountability and transparency of our work, as well as technical quality of our programmes.
The key evaluation questions listed above are to be considered as guiding questions only and the evaluation team is not limited to them. The refining and further elaboration of the questions should be done by the evaluation team, which will propose a matrix of detailed evaluation questions. The final evaluation questions will be discussed and agreed upon through consultation during the inception phase.
Methodology
The consultant/evaluation team will develop methodology for Evaluation that will be using specific experimental or quasi-experimental methods to answer the evaluation questions. Develop the study design in consultation with ADRA Ukraine program staff and ADRA Germany staff. The design of data collection tools (Questionnaire, focus group questions and informed consent forms) must be validated by ADRA Germany.
The consultant/evaluation team should use the online and/or onsite way of interviews. Datasets which are already collected should be primarily used for the quantitative analysis together with the accountability mechanisms data. The consultant/evaluation team should analyze the accountability and feedback data in order to determine the perception of the activities by beneficiaries together with interviews.
Targeted beneficiaries of the Evaluation (quantitative analysis – random sampling): 1.000 (Ukraine), 500 (Moldova);
1.500 in total
Intervention segments: cash assistance, legal support, cash for rent, light repair
Target beneficiaries of the Evaluation (qualitative analysis – random sampling): 70 (Ukraine), 30 (Moldova); 100 in total
Intervention segments: mobility assistance, evacuation assistance, winterization kits distribution, psychosocial support
Timeline of the Evaluation
The Evaluation should have one measurement point and report, and to be conducted in the last quarter of the project (March – May 2023). The task should include production of the inception report, data collection and reporting which should be done in approximately 25 days. More detailed schedule is presented in part 5.
Requirements related to gender and age
Evaluation and study design, methodology, data collection and analyses should adequately capture the situations and experiences of people identifying with all genders. The Consultant/evaluation team should collect appropriate sex- disaggregated data and ensure that an evaluation is gender-sensitive or “engendered.” To engender an evaluation is to view the evaluation process, dynamics, design, and the key elements of each evaluation stage through a “gender lens” to make sure that the evaluation and associated data collection and analysis practices are fully informed by an awareness of how gender shapes and is shaped by both programs and evaluations.
Engendering an evaluation means that all stages of the evaluation reflect:
- an awareness that the degree and meaning of program participation, program results, and potential sustainability are shaped by gender;
- a recognize on that explicit attention to gender issues must be integrated into the evaluation if gender equality objectives are to be addressed; and
- a commitment to examining the extent to which gender equality was achieved as a result of the strategy, project, of approach that was implemented.
The evaluation must follow the instructions of data disaggregation according to the sequence 0-17, 18-49 and 50+ in order to make a relevant comparison of the results according to gender and age.
EVALUATION MANAGEMENT
- Roles and responsibilities
For the evaluation, ADRA Germany will contract an independent third-party consultant/evaluation team firm to collect reliable, accurate, valid, and timely data.
Responsibilities of Consultant/evaluation team / Consulting firm:
- Consultant/evaluation team firm is fully responsible for the implementation of this assignment, including methodology design, data collection, and reporting.
- Consultant/evaluation team firm will liaise with ADRA staff throughout the process, providing regular updates and seeking their input and advice where necessary.
- Consultant/evaluation team firm will prepare a detailed task timeline and draft methodology (inception report) and work with ADRA to finalize the methodology
- Consultant/evaluation team firm is responsible for methodology design, including sampling and tools for data collection
- Consultant/evaluation team firm will adhere to the timeframe of the consultancy.
- Consultant/evaluation team firm will deliver periodical and final report.
- All the data collected during the research/study period should be protected in line with provisions of the General Data Protection Regulation 2016/679, and national regulations on the protection of personal data in Germany
Responsibilities of ADRA Germany:
- Providing all background materials and data for the research consulting firm;
- Ensuring the participation and support of implementing partners and beneficiaries;
- Monitoring adherence to specified deadlines; facilitating access to required information;
- Providing guidance throughout all phases of execution, approving all deliverables, and facilitating, access to any documentation (or any person) deemed relevant to the process;
- Share deliverables with stakeholders;
- Perform quality control, as required for all deliverables;
Responsibilities of ADRA Ukraine & ADRA Moldova:
- Providing information about the project, implementation stages, success and difficulties;
- Facilitating the consultant’s interview process with selected project beneficiaries;
- Enabling insight into data on the implementation of activities;
- In cooperation with ADRA Germany, providing support to the consultant for other information that is necessary for the successful implementation of the evaluation
DELIVERABLES AND REPORTING REQUIREMENTS
The consultant/evaluation team should submit the following deliverables during the evaluation process:
- Draft methodology for the Evaluation (including evaluation design) – (max. 4 pages) The proposal outlining the methodology of the planned evaluation and the work plan are part of the documents which needs to be submitted before the kickoff workshop.
- inception report (methodology, sampling, tools for data collection, approach to data analysis and details of Evaluation Work Plan (draft implementation schedule)) The inception report should set out the planned approach to meeting the evaluation aims and objectives, methodologies to be used and questions to be answered through the reviews and planned interviews. It should provide a description on how data will be collected and drafts of suggested data collection tools such as questionnaires and interview guidelines. The proposal outlining methodology and work plan should be used as basis for the inception report. The inception report should be submitted after the kickoff workshop.
- Draft and final report of Evaluation submitted in English; final report will incorporate ADRA’s feedback into the draft report;
- Datasets – quantitative data/datasets created or obtained in performance of this task; datasets) must be delivered in non-proprietary, machine readable format; dataset must be organized and documented for use by those not fully familiar with the intervention or evaluation.
Content of the evaluation report(s)
ADRA expects that the final reports will include the following sections, at a minimum:
- Cover Page
- Acronym List
- Executive Summary
- Project Background
- Purpose and Scope of the Evaluation
- Literature Review (relevant sources cited in APA style)
- Evaluation Questions
- Evaluation Methodology
- Evaluation Results
- Conclusions (successes and challenges)
- Lessons learned
- Recommendations
- Bibliography
- Attachments (evaluation SoW/ToR, data collection instruments, photos, charts, graphs, additional analyses)
EVALUATION SCHEDULE
The below evaluation schedule can be modified and adjusted according to the Intervention overall implementation plan. In order to perform all scheduled tasks, Consultant/evaluation team firm shall be hired for the period February 2023 – May 2023. The consultant/evaluation team shall prepare and implement the work plan in consultation with and support by the relevant personnel of ADRA Germany and partner organizations, in accordance with the following dynamics:
Drafting of the work plan and preparation of detailed work methodology and submitting the Inception report with a detailed
work plan and methodology for the evaluation – March 15, 2023
Data collection – April 1, 2023 – April 30, 2023
Evaluation – draft report – May 15, 2023
Evaluation – final report – May 31, 2023
A total of a maximum 25 days is envisioned for the purpose of this consultancy to undertake and produce the final evaluation. Expressions of interests should include a rough plan and outline of the proposed approach. Specific Dates are subject to mutual agreement.
EXPERIENCE AND SKILLS REQUIRED
The consultant/evaluation team/evaluation team should have the combined following qualifications:
- Demonstrable experience (minimum 5 years) in evaluation of humanitarian programs in conflict and displacement settings
- Experience in the design and implementation of multi-methodological evaluation approaches and tools (quantitative and qualitative)
- Russian and Ukrainian language skills are desirable
- Sound data analysis skills with the use of appropriate statistical software for data analysis and visualization both quantitative (e.g. SPSS, Stata) and qualitative (e.g. Atlas.ti)
- Familiar with the OECD-DAC evaluation criteria for Humanitarian Action, ALNAP, Sphere standards and related international quality (e.g. CHS) and evaluation standards
- Experience in leading program/project evaluations (quantitative and qualitative) in international development and/or relief settings
- Excellent organizational and report writing skills (demonstrated through work samples)
- Excellent verbal and written communication skills in English
- Ability to work effectively and independently in a cross-cultural environment
- Willingness to travel to project sites
- Experience of working in Ukraine and/or post-Soviet region is desirable
The consultant/evaluation team must adhere to humanitarian principles, ADRA Protection Policy, and ethical values of ADRA.
Selection criteria
In addition to the quality of the content of the offer, the total price and the scope of the services offered, previous experience, references and target group orientation are also taken into account when the offer is awarded. The basis for the evaluation is as follows:
- technical approach (methodology and implementation plan) = 40 points
- key personnel qualifications = 35 points
- fees and associated costs = 25 points
All documents and data collected are confidential and, in the case of personal data, are to be treated in accordance with the data protection regulations (in particular the EU GDPR and the Federal Data Protection Act) and are to be used exclusively for the evaluation. No later than one year after completion of an evaluation, the contractor will destroy the data and documents permanently and in compliance with data protection.
How to apply
The application should include:
- A cover letter addressing the selection criteria above.
- Applicant’s CV, highlighting experience relevant to this evaluation.
- A sample report from a prior assignment with content directly relevant to this evaluation.
- Technical proposal illustrating the applicant’s understanding of the TOR, draft evaluation framework and plan including any logistic support required.
- Financial proposal: The financial proposal should provide cost estimates for services rendered including daily consultancy fees.
- Contact details of at least two references from among recent employers or clients
We ask that you submit an offer to Eldar Komar, MA (he/him) at [email protected] by March 3, 2023 (CoB). Please indicate „Evaluation Ukraine” in the subject line. Only shortlisted candidates will be contacted. If you have any further questions, you are welcome to contact us by email. Please no phone calls!
Your application is collected for the purpose of recruitment and personal administration. Unless you direct otherwise (for example if you would like the application kept on file for future vacancies) the application forms (and attachments) of unsuccessful applicants will be destroyed after 6 months. It is the agency policy to protect, and keep secure, all personal data collected. All personal data is processed for the purpose of recruitment, and, in the case of successful applications, for the satisfactory administration of their employment, and for no other purpose.