People in Need
Contract title: Protection program activities Evaluation
Project code: UKRP000285
Tender no.: CZEC001596
Procedure: Negotiated with publication
Location: Ukraine
International NGO People in Need is searching for the provider of following services:
Evaluation of of the Protection programme activities implemented under GAC & SDC funded projects during the period from April 2022 until May 2023 in Ukraine.
The evaluation shall focus on Protection programme activities implemented within both projects. The purpose of this evaluation is to assess PIN’s performance and delivery of the Protection interventions under the GAC and SDC funded project according to OECD evaluation criteria. The evaluation shall help PIN to improve its future programming through lessons learned and best practices generated through this project.
People in Need (PIN) is an international NGO with a proven track record of providing quality humanitarian assistance to the most vulnerable in the Middle East, Asia, Africa, and Europe. PIN was founded in 1992 and since then has delivered emergency relief and development cooperation, human rights support, social integration, and education and awareness programs to vulnerable communities in more than 40 countries with a goal to help people and communities build resilience in conflict and disasters and support them in recovery.
The evaluation shall focus on Protection programme activities implemented within both projects. The purpose of this evaluation is to assess PIN’s performance and delivery of the Protection interventions under the GAC and SDC funded project according to OECD evaluation criteria. The evaluation shall help PIN to improve its future programming through lessons learned and best practices generated through this project.
The maximum budget forseen for the evaluation is 40 000 EUR.
The contract awarded to successful supplier(s) is a Contract for providing services which will commit PIN to purchase the specified quantity of services as defined in the contract at the agreed prices / rates.
Project Outline
GAC (Global Affairs Canada) funded project “Core gender and age-based interventions in priority sectors of need in western and eastern Ukraine“ (April 2022 – May 2023) aims to contribute to enhancing financial stability and social wellbeing of Vulnerable Displaced Population (VDP) in western and eastern Ukraine (Rivne, Kharkiv, Poltava, Kirovohrad, Lviv, Chernihiv, Volyn and Ivano-Frankivsk oblasts). The project is led by PIN and implemented in collaboration with two partners. The activities include:
o Multipurpose Cash Assistance (MPCA) to the IDPs;
o Age, gender and diversity-informed shelter and WASH upgrades in the collective centres;
o Establishing Child Friendly Spaces (CFS) within selected collective centres and in central communal spaces, where children can participate in structured, facilitator-led social emotional learning and recreational activities aimed at supporting the development of key life skills and coping strategies;
o Psychosocial Support (PSS) to the displaced population in each location targeted through the intervention, using a combination of locally hired and capacitated psychologists and a mobile team approach.
SDC (Swiss Agency for Development and Cooperation) funded project „Improving meaningful access to emergency assistance for vulnerable conflict-affected persons in Ukraine“ (September 2022 – May 2023) complements other humanitarian response activities of PIN in Ukraine through delivering protection interventions, as well as increasing local preparedness and recovery by strengthening the capacity of local organizations who would sustain the impact of this intervention. The activities include:
o Provision of emergency PSS services through PSS Mobile Teams;
o Provision of cash transfer top-up under Special Needs Fund (SNF) designed to support emergency needs for cases with specific additional needs due to age, gender, or other diversity characteristics;
o Capacity building to local partners.
Evaluation purpose and objectives
The evaluation shall focus on Protection programme activities implemented within both projects. The purpose of this evaluation is to assess PIN’s performance and delivery of the Protection interventions under the GAC and SDC funded project according to OECD evaluation criteria. The evaluation shall help PIN to improve its future programming through lessons learned and best practices generated through this project.
Evaluation objectives include:
1) to evaluate the achievements of the Protection interventions in relation to the goal, objectives and results defined in project proposals and “Logical Frameworks”;
2) to evaluate impacts of the project on targeted groups and communities, as well as overall effectiveness and appropriateness of the project activities, challenges in implementation of the Protection interventions, and potential for sustained outcomes; as well as to identify external and project-related factors (if any) limiting the achievements of the programme;
3) to identify best practices, lessons learned, strengths, and challenges in the design of project activities, and their implementation, coordination and monitoring.
Key evaluation questions
Following research questions are suggested for the evaluation of the project:
Appropriateness:
1.To what extent did the projects’ protection activities (CFS, PSS services, SNF transfers) meet the needs of the beneficiaries? What can be done in terms of approach and design of activities to make them more appropriate to the needs of the target groups, PSS staff and families benefitting from the provided support and services?
Effectiveness:
2. How effective the project has been in increasing psychosocial wellbeing of the individuals receiving PSS, ability of people with disabilities and chronic illnesses to meet their special needs with provided cash transfers, as well as in achieving other goals and targets of the protection activities as defined in the project proposal?
3. How effective the project has been in ensuring and monitoring quality and impact of the provided psychosocial services, and addressing any issues identified during the implementation? What can be done to strengthen monitoring and control mechanisms, as well as indicators for measuring project achievements?
4. Has PIN’s coordination with partners, other actors and stakeholders been effective in terms of information exchange, as well as ensuring effective referrals and complementarity of interventions? What can be done to strengthen coordination between international/national NGOs and governmental services and improve mechanisms for effective referrals?
Impact and sustainability:
5. Were there any unintended (positive or negative) consequences of the Protection activities at the individual and community level? How can the positive impacts be strengthened in the future programming?
6. What was the project’s impact on wellbeing of the vulnerable displaced populations and their participation in social life? How and to what extent did it contribute to social cohesion within the targeted communities? How the existing protection activities could be complemented to better address the needs of individuals and communities?
7. What was the impact of the project on integration and adaptation of the internally displaced children? What can be done to strengthen child protection activities?
8. What is the project’s impact on supported local organisations and their ability to deliver quality and continuous protection services in their communities after the project?
Evaluation methodology
The mixed methods evaluation will utilize data from primary and secondary sources. Desk review will include review and compilation of existing information, reports and data collected during the project implementation, including project proposal, project reports, assessment and monitoring data and reports produced by PIN’s and partner’s M&E and Programme teams, as well as findings from Community Feedback and Response Mechanism. Additionally, review of secondary data shall include open source information, including external data from UN agencies, relevant Clusters, and other I/NGO partners to enable comparison and review/interpretation of findings in light of overall context and developments.
For primary data collection both quantitative and qualitative methods shall be used as required, considering evidence gaps identified during the secondary data review. The data collection methods may include: Key Informant Interviews (KIIs), Focus group discussions (FGDs), case studies, as well as surveys with specific groups of stakeholders, and/or other methods as proposed by the evaluator.
Qualitative study
Qualitative information may be collected through KIIs and FGDs with:
o project management and staff involved in delivering cash-based assistance, protection and capacity building activities;
o partner staff;
o members of local authorities, coordination centres representatives, local volunteers;
o children, parents, supporting professions specialists, teachers, adults;
o psychologists, CFS staff;
o community groups as relevant;
o other stakeholders as relevant.
Guiding questions for the qualitative data collection shall be designed to provide deeper insights and explain and validate monitoring results related to both achievements of the planned project outcomes, as well as beneficiary feedback regarding appropriateness, quality, usefulness and sustainability of the assistance. Based on perceptions and feedback from different stakeholders, interviews and discussions shall provide information for evaluation of the project and activity design, help to identify challenges in implementation process, factors contributing to success of the project, and any intended and unintended consequences and effects of the project. As such, the qualitative evaluation will capture lessons learned and best practices.
Quantitative data
While considerable amounts of monitoring data are collected by PIN and the partner during the project (please see details below), depending on information gaps identified through review of secondary information and data, Consultant may propose additional surveys with specific stakeholders where quantitative data may be necessary or helpful in answering research questions listed in ToR (can be obtained on request from [email protected].
Following PIN data and monitoring results will be shared with the external evaluator for baseline-endline comparison (where relevant) and to support evaluation of project achievements against targets, as well as provide other necessary information:
Where relevant, PIN conducts baseline data collection with an aim to assess situation of individuals prior accessing protection services and to establish baseline values of the outcome indicators that are further measured during subsequent endline or post intervention monitoring surveys.
Following data will be shared with the evaluator for analysis (see also Data Sources in Annex A & B):
o pre- and post-training assessments;
o data from assessments of individuals’ psychosocial wellbeing and behaviour indicators before and after the intervention (including Strengths and Difficulties Questionnaire (SDQ) scores of children attending CFS);
o Interview transcripts (in Ukrainian) from KIIs conducted by PIN enumerators with parents/teachers/schools’ representatives where group sessions with children took place;
o data obtained through phone-based post-intervention monitoring survey with trainees (in case the trainee confirms the readiness to participate in the interview);
o reports (in Ukrainian) of the monitoring visits in the established CFSs;
o programme data as relevant and required, incl.: beneficiary registration data, training attendance data, etc.
Data analysis
PIN expects all quantitative data to be rigorously analysed and representative within the reasonable limits and constraints of the context. Qualitative data should also be rigorously analysed and should primarily focus on developing a deeper understanding about the relevance of the programme and providing recommendations for: improving or strengthening the relevance, effectiveness, efficiency, positive impact and sustainability of the results of the program.
Data analysis will include: triangulation of data related to the research questions obtained from different primary and secondary sources. Where relevant, results shall be disaggregated by gender, age, disability, residency status or other disaggregates depending on information needs relevant to research questions and methodology proposed by the evaluator.
Specific Tasks
– Refine the evaluation objectives and primary research questions in consultation with PIN’s team.
– Conduct desk review of secondary data, including PIN’s and partner’s existing project monitoring and evaluation data, to identify gaps in data coverage and knowledge.
– Develop a methodology and evaluation tools to address the specific objectives and individual research questions of the evaluation.
– Collect primary data to establish and quantify PIN’s and partners’ performance against criteria outlined above.
– Provide draft reports to programme management presenting evaluation findings, lessons learned, as well as recommendations.
– Incorporate PIN’s feedback into draft reports and prepare final report. The final report should both describe the results of the evaluation, and provide actionable recommendations for improving
PIN’s programme.
Expected Deliverables
The required deliverables for this consultancy are:
1) Inception report, outlining proposed methodology, agreed by PIN;
2) Data collection tools;
3) Cleaned datasets and transcripts of interviews and focus group discussions (translated into English where appropriate);
4) Draft Final Report which should address the research questions and provide recommendations
5) Final reports, integrating PIN team’s comments on draft report. The report must be clear and concise, and the following sections must be included as a minimum: Executive Summary (maximum 3 pages), Methodology, Analysis of Findings, Lessons Learned and Recommendations, Annexes: TOR, a list of individuals interviewed, templates of data collection tools used, a description of the methods employed, a summary of survey results (if appropriate) and any other relevant materials.
Duration
GAC-funded project and the initially approved SDC project implementation period is scheduled to end in May 2023. Evaluation will start by the end of the project, and is expected to take approximately 10 weeks from June until mid-August 2023.
Ethical considerations
When dealing with personal data, PIN follows Guidelines on Processing Personal Data and The Principles of Work with Personal Data. PIN’s MEAL manual contains guidance, specifically related to data collected for the monitoring and evaluation purposes (relevant extract of the document will be provided to the consultant contracted for the service upon request). In line with these documents, the evaluation team will make clear to all participating stakeholders that they are under no obligation to participate in the evaluation survey or other data collection activities. All participants will be assured that there will be no negative consequences if they choose not to participate. The monitoring team will inform respondents about acquiring their data from PIN and obtain informed consent from the participants or from caregivers when approaching children as respondents. In case any special category data (e.g. about health, ethnicity, religion etc.) are collected that was not collected by PIN during beneficiary registration and/or no written consent was obtained, the evaluation team shall obtain signed consent from the participant or caregiver. The research team will ensure prior permission is received for taking and use of visual images for specific purposes. The evaluation team will ensure the visual data is protected and used for agreed purposes only. In particular, the Evaluator will employ robust data security measures (related to storing and sharing datasets containing personal information) to further ensure participants’ confidentiality and anonymity.
Logistics
No logistical support is provided by PIN to facilitate travel into or while working inside Ukraine. Consultant is responsible for the local travel and other arrangements for its team inside Ukraine.
Qualification criteria
Criteria which applicants must meet in order to progress to the next round of evaluation. If an applicant does not meet any of the Qualification Criteria, they will be excluded from the tender process immediately. These criteria are scored as ‘Pass’ / ‘Fail’.
Each applicant must meet and properly prove the following minimum qualification criteria with relevant documentation:
1) Technical proposal (maximum 10 pages) including:
– Organization profile and how it meets requirements for the assignment (incl. list of team members and info on their experience relevant to the task);
– Evidence of Consultant’s experience in Ukraine crisis, or similar contexts; list of locations (including oblasts, hromadas) where the Consultant has field presence or access to inside Ukraine;
– Consultant’s approach and methodology for required Evaluation including: proposed research questions (in line and, where relevant, in addition to those outlined in the ToR); proposed data collection methods; details on data analysis; sampling approach; quality assurance procedures; etc.;
– Details on data protection measures undertaken by the firm;
– Risk management;
– References to applicable sector standards, relevant policies and standard operating procedures, such as: Safeguarding, Protection, Ethics, Code of Conduct, Data Protection, Gender Equality and Social Inclusion, Conflict of Interest, sector standards, etc.
2) Overview/list of past evaluations, assessments or research projects undertaken by the Consultant in the last two years. For every contract/assignment, following information included:
– Contracting agency (donor/NGO/UN agency/International Organisation)
– Country and region of the assessed/evaluated programme or research activity
– Programme sector(s) which were assessed or evaluated during assignment, if applicable (e.g. Protection, Food Security, Livelihoods, Water, Sanitation and Hygiene, etc)
– Programme modality or modalities of activities which were monitored, assessed or evaluated during assignment (e.g. in-kind distribution, voucher/cash distribution, capacity building/training)
– Role/tasks of key experts proposed for this evaluation in the listed research project (e.g. contract/team management, training of enumerators, research activity planning, data analysis & reporting, etc.)
– Brief (max three lines) description of the purpose and scope of the assessment, evaluation or research assignment
– Contact details of the contracting agency’s focal point that can be contacted for reference.
3) CVs of key personnel involved in undertaking the evaluation (lead consultant/researcher(s), sector/technical expert(s), field researcher/team leader inside Ukraine).
4) Two most relevant examples of past assessments or research by the company and experts proposed for the task (can be anonymized or redacted for privacy). These should preferably be relevant to the sectors (Protection/PSS) as outlined in ToR, not necessarily from Ukraine.
5) Detailed, itemised cost proposal, including daily fees and any other associated costs.
Evaluation criteria
The evaluation committee will evaluate and award the contract on the basis of the following evaluation criteria:
1) Company’s and proposed team’s qualification – 20%
2) Relevant experience – 20%
3) Quality of technical proposal – 20%
4) Quality of performance and outputs of the previous assessment / research projects – 20%
5) Financial offer/ price – 20%
1) Company’s and proposed team’s qualification (20%)
– Relevance of company’s profile
– Expertise of the proposed team (incl. management, lead consultants, technical experts, field staff) relevant to assignment, qualification requirements and their role in the team (e.g. expertise and experience in designing evaluation methodology and tools, conducting qualitative/quantitative data collection, analysis and reporting, leading and managing evaluations, training or overseeing enumerators, etc.)
– Operational and sector standards followed in implementation of the assignment (incl. policies on Safeguarding, Protection, Ethics, Code of Conduct, Data Protection, Gender Equality and Social Inclusion, Conflict of Interest, and sector-specific standards as appropriate)
Means of verification: Technical Proposal (Organisational profile and info about the team, references to the standards followed by the organisation and applicable to the evaluation); expert CVs.
2) Relevant experience (20%)
– Company’s and/or lead and/or technical experts’ experience in conducting evaluation of Protection sector projects/ interventions (incl. psychosocial support activities; etc.)
– Company’s and lead and/or technical experts’ experience in conducting similar research for NGOs/ UN/ International Organisations in humanitarian contexts; familiarity with the context in Ukraine
Means of verification: Technical Proposal (Overview/list of past evaluations, assessments or research projects undertaken by the Consultant and involved experts in the last two years); expert CVs.
3) Quality of technical proposal (20%)
– Demonstrated level of understanding of the research objectives, key areas and questions. Additionally proposed research questions (and/or sub-questions) meaningfully complement those listed in the ToR
– Proposed data collection and analysis methodology: quantitative and qualitative data collection methods and sampling strategies, their relevance to the evaluation objectives and research questions, level of detail regarding qualitative data collection; data processing and analysis methods/approach; considerations regarding data protection
– Time and resources allocated for the assignment (feasibility of timeline and activity plan, availability of sufficient staff and other resources for implementing the planned activities)
– Evidence of understanding the risks relevant to the context and proposed measures for risk mitigation
Means of verification: Technical proposal (data collection and analysis methodology, research questions, timeline, risk mitigation measures)
4) Quality of performance and outputs of the previous assessment / research projects (20%)
– Relevance of submitted assessment or research samples in terms of scope, type of assignment, sectoral coverage, geographic focus
– Presentation of information: structure of the report in terms of logic and clarity; use of visual data presentation (choice and presentation of charts, graphs, tables, maps, photos…)
– Quality and applicability of recommendations, their link to conclusions
– Quality of performance based on references and PIN’s previous experience
Means of verification: submitted sample reports, references (from referees selected by PIN from those mentioned in Overview/list of past assignments), PIN’s experience with the company.
Submission deadline: 18/05/2023
Questions/ Clarifications deadline : 11/05/2023
For more information about the tender requirements and required scope of services please contact [email protected] to receive complete tender documentation.
How to apply
Submission deadline: 18/05/2023
Questions/ Clarifications deadline : 11/05/2023
For more information about the tender requirements and required scope of services please contact [email protected] to receive complete tender documentation.