Diakonie Katastrophenhilfe
- Background
Türkiye hosts the world’s largest refugee population, including approximately 3.1 million registered Syrian refugees, and faces significant economic challenges such as high inflation. The February 2023 earthquakes displaced 3.3 million people, with 2 million living in tents and containers. As of February 2024, over 675,000 individuals remain in 392 formal container sites across 11 provinces, with limited data on those in informal settlements. Many face barriers to accessing services, such as shortages of appointments and financial constraints. Despite the efforts of the Turkish state, as well as national and internation organizations, many vulnerable people are still not being reached, e.g. persons with special needs, unregistered refugees, survivors of gender-based violence. Diakonie Katastrophenhilfe (DKH) and Support to Life (STL) are implementing a 12-month ECHO-funded project titled “Rebuilding Life for Earthquake Survivors: Addressing Core Needs and Protection” in Hatay, Adıyaman and Kahramanmaraş provinces of Türkiye. The action aims to improve access to rights and services for earthquake-affected refugees through protection information dissemination, psychoeducation sessions, GBV prevention and response, case management, individual psychological counseling, Cash for Protection targeting GBV survivors and multi-purpose cash assistance.
DKH seeks for an experienced consulting firm to implement Protection Outcome Mapping (POM) Study to evaluate the specific objectives of the MHPSS integrated protection intervention. The study will be conducted as part of the project’s monitoring and evaluation framework. This study is designed as an external evaluation tailored to protection and MHPSS sectors, going beyond standard OECD evaluation criteria.
- Objectives
The main purpose of this study is to evaluate the effectiveness and impact of the project’s protection and MHPSS interventions as well as to assess the degree of which protection mainstreaming incorporated throughout the implementation. The Protection Outcome Mapping shall deliver information on the achievements on impact related indicators of the action. The relevant indicators are reflected below:
Outcome Indicator One
% of beneficiaries reporting that humanitarian assistance is delivered in a safe, accessible, accountable and participatory manner.
Coverage Outcome Indicator One
All activities implemented through community based, specialized protection, MPCA
Data sources Outcome Indicator One
Post activity survey: community-based protection including awareness, information dissemination, GBV key messaging, psychoeducation
Post assistance survey and PDM for Cash for Protection: Specialized protection including case management and individual psychological counseling
PDM of the MPCA activities (obtained through third party monitoring)
Outcome Indicator Two
% of individuals reported improved wellbeing and mitigated protection risks
Coverage Outcome Indicator Two
Specialized protection activities including case management and individual psychological counseling
Data sources Outcome Indicator Two
Post assistance survey and PDM for Cash for Protection: Specialized protection including case management and individual psychological counseling
Outcome Indicator Three
% of beneficiaries referred to specialized MHPSS services
Coverage Outcome Indicator Three
Cases referred to specialized MH services from either case management or individual counseling activities
Data sources Outcome Indicator Three
Psychiatric referral tracker and follow-up
- Conceptual Framework
The conceptual framework for this study integrates key elements from the ALNAP’s guide on monitoring and evaluation of protection, and the IASC common monitoring and evaluation framework for MHPSS. This integrated approach allows for a comprehensive analysis of the impact of action interventions in the earthquake-affected areas of Türkiye, namely in Hatay, Kahramanmaraş and Adıyaman.
The framework incorporates key principles from ALNAP’s guide on evaluating protection in humanitarian action. This includes an outcome-oriented approach that focuses on assessing changes in the lives of affected people, the use of contribution analysis to examine how action interventions contribute to protection outcomes, and a commitment to mixed-methods and participatory approaches. These principles ensure that the study goes beyond simply measuring outputs to understanding the real impact of interventions on refugee communities.
From the IASC MHPSS M&E Framework, the study incorporates key goal impact indicators such as functioning, subjective well-being, extent of prolonged distress, coping strategies, social behavior, and social connectedness. It also aligns with the IASC framework’s outcome areas, which range from ensuring that emergency responses do no harm to improving access to appropriate mental health and psychosocial support. This integration allows for a holistic assessment of both protection and MHPSS outcomes, recognizing the interconnected nature of these domains in humanitarian contexts.
The framework is tailored to the specific objectives and interventions of the DKH-STL action. It focuses on assessing how activities such as protection information dissemination, psychoeducation sessions, case management, and multi-purpose cash assistance contribute to improving access to rights and services and mitigating protection and GBV risks for earthquake-affected refugees. This action-specific focus ensures that the study produces findings that are directly relevant to improving ongoing and future interventions.
Cross-cutting themes are woven throughout the framework, including gender and age sensitivity, disability inclusion, localization, and consideration of the humanitarian-development nexus. These themes reflect critical considerations in contemporary humanitarian protection work and ensure that the study captures the diverse experiences and needs within the refugee population.
The analytical approach of the study combines several key elements. It involves reconstructing and assessing the action’s theory of change for protection outcomes, applying systems thinking to analyze protection risks and interventions within their broader context, using contribution analysis to assess the action’s role in observed changes, and employing an intersectional lens to understand how multiple factors influence vulnerability and resilience.
By integrating these various elements, the conceptual framework provides a robust foundation for a comprehensive and nuanced evaluation of the protection situation and the impact of interventions. It allows for a deep understanding of the complex realities facing earthquake-affected refugees while also producing practical insights to inform future programming and policy.
- Analytical Approach
The analytical approach for this Protection Outcome Mapping Study integrates several key methodologies to provide a comprehensive assessment of the action’s impact on protection outcomes for earthquake-affected refugees in Türkiye.
At its core, the approach uses Theory of Change (ToC) analysis to reconstruct and evaluate the action’s logic model, identifying both intended and unintended outcomes. This is complemented by systems thinking, which contextualizes the action within the broader socio-ecological environment, examining how interventions interact with various elements of the protection ecosystem.
Contribution analysis forms a critical component, developing and testing contribution stories for key protection outcomes. This method allows for a nuanced understanding of how the action has influenced observed changes, while also considering alternative explanations.
The study incorporates intersectional analysis throughout, examining how factors such as gender, age, and disability intersect to influence vulnerabilities and resilience. This ensures a nuanced understanding of how protection risks and intervention effectiveness vary across different groups. To capture a wide range of outcomes, including unexpected ones, the approach includes elements of outcome harvesting. This is supplemented by comparative case analysis, which helps identify contextual factors influencing intervention success across different scenarios. For a comprehensive insight into the Outcome Harvesting modality, the https://outcomeharvesting.net/resources/ website can provide additional perspective for advisors.
Participatory methods are integrated throughout, involving beneficiaries and local stakeholders in data collection. This ensures that affected populations’ perspectives are central to the analysis. The approach also includes temporal and contextual analyses to understand how protection risks and outcomes have evolved since the earthquake, and how the broader political, economic, and social context influences these outcomes.
Finally, a capacity and resource analysis will assess the presence of existing/available services and their capacities and resources (human resource, physical space, technical competencies) around protection and MHPSS, provided by governmental and non-governmental institutions (such as women shelter, social service center, migrant health center, community mental health center, hospitals, etc.)
This multi-faceted approach combines rigorous evaluation methodologies with systems thinking and participatory elements, allowing for a comprehensive understanding of the action’s contribution to protection outcomes while considering the complex, dynamic nature of the post-earthquake protection environment in Türkiye.
- Scope of Work and Methodology
Description of the Task (Phase One)
Conduct a thorough analysis of the following datasets. This includes analyzing:
- Post assistance and Post Activity Survey Results
- PDM results for Cash for Protection
- FGD and monitoring visit reports carried out by STL
- Other relevant reports and studies developed by STL
Role of STL & DKH (Phase one)
STL IM Officer carries out a data consolidation and shares with the third-party contractor
Role of Third-Party Contractor (Phase One)
Analyze and interpret the provided dataset
Description of the Task (Phase Two)
Develop a detailed methodology, including data collection tools for both quantitative and qualitative methods and carry out the primary data collection:
- Field based protection mainstreaming assessment to collect good practices and areas for improvement with regards to dignity, access, participation and safety.
- Mapping of protection and MHPSS services provided by governmental and non-governmental institutions.
- Semi-structured online surveys with social workers and psychologists.
- KIIs (with local authorities, DG ECHO representative, inter-agency coordination mechanism and STL field staff).
- FGDs with beneficiaries that are not part of the specialized protection activities (e.g. case management) ensuring representation of diverse demographic groups.
- Development of 2-3 in-depth case studies through collaboration with DKH/STL visibility and communication departments.
- Mid-term review report (developed and delivered by DKH & STL)
Role of STL & DKH (Phase Two)
Review questionnaire and tools developed by the third-party contractor
Facilitates field access for data collection
Role of Third-Party Contractor (Phase Two)
Develops detailed methodology, sampling and tools
Submission of inception report
Carries out data collection
Analyze qualitative and quantitative data collected at phase 1 and 2
Description of the Task (Phase Three)
Preparation and finalization of the report.
Role of STL & DKH (Phase Three)
Review and provide feedback on the report
Role of Third-Party Contractor (Phase Three)
Develop and finalize the report
Description of the Task (Phase Four)
Presentation of the findings including recommendations
Role of STL & DKH (Phase Four)
Dissemination of the report to relevant stakeholders and donor working group
- Data Management Criteria
In the scope of this work, the following data confidentiality and data management responsibilities are expected from the third party:
Informed Consent: The third party and its dependent and/or temporary staff (enumerators) should obtain the informed consent of all stakeholders on this issue.
Data Responsibility: In its technical proposal, the third party should demonstrate its data management capabilities and qualified staff structure.
Data Security: The consultants must follow strict data security protocols. This includes:
Encryption of data both in transit and at rest.
Use of password-protected devices and systems.
Compliance with GDPR or relevant local privacy laws.
Ethical Considerations: The consultants should avoid collect the sensitive personal data (e.g., personal health information), if they collect the sensitive data they must anonymize the data
Centralized Storage: The technical proposal should define where data will be stored (e.g., on a cloud platform like Google Cloud or an internal server) and ensure access control measures are in place.
Access Control: Limit access to the collected data based on roles, ensuring that only authorized personnel can view or edit sensitive information.
Backup and Recovery: Consultants must ensure regular data back-ups and have a data recovery plan in place in the event of technical failure or disaster. During the completion phase of the procured service, the third party must hand over the raw data to the contract holder.
- Expected Deliverables
In the scope of this work, the following deliverables are expected from the third party:
- Inception report: The Third Party is expected to submit an inception report detailing the scope of proposed work, methodology as well as review of background documents and existing information.
- Workplan: The workplan includes the detailed timeframe of the work with responsible persons, deadlines and milestones.
- Data collection tools and methodology: The Third Party is expected to develop all the tools and methodology for this work, share with the DKH and STL teams for review. The finalised data collection tools and methodology are expected to be shared as a deliverable.
- Raw data: The data collected by the Third Party in the fields needs to be shared with STL in compliance with data protection regulations.
- Final Report: The Third Party is expected to prepare a final report composed of all the information about the conducted work findings and recommendations. The final report should not be more than 50 pages, and it should include annexes if needed.
- Presentation of the findings including recommendations: A presentation summarising the work, findings and recommendations is expected from the Third Party.
- External publication: DKH and STL would like to publish the outcome of this work through an external publication, therefore Third Party is expected to prepare a short publication (max 10 pages) with key information and recommendations for the external audience including humanitarian actors and local/central authorities in Turkey, international organisations and donors.
- Key Evaluation Questions
The evaluation criteria and questions within this study aim to assess both the impact of the specific interventions outlined in the project proposal. They are designed to provide information that can feed into reporting on the outcome indicators mentioned. The evaluation should use a mixed-methods approach, combining quantitative data on the indicators with qualitative insights on impact and effectiveness.
Evaluation Criteria and Related Questions
Criterion One: Protection Mainstreaming and Do No Harm
- To what extent do beneficiaries perceive assistance as safe, accessible, accountable and participatory?
- What measures and good practices are in place with regards to beneficiary feedback and complaints, disability inclusion, gender mainstreaming, and participation?
- How effectively has the action integrated protection mainstreaming principles across all interventions?
Criterion Two: Psychosocial wellbeing
- What changes in wellbeing do beneficiaries report as a result of the interventions?
- How have the interventions contributed to improving social connectedness and support networks among beneficiaries?
Criterion Three: Protection risk mitigation
- How have the protection and MHPSS interventions contributed to mitigating protection risks for beneficiaries, including GBV risks?
- To what extent have interventions improved beneficiaries’ access to rights and services?
Criterion Four: MHPSS Service Provision and Referral Systems
- What proportion of MHPSS beneficiaries are referred to specialized services when needed? How effective is this referral process? What is the referral success rate for action beneficiaries to specialized mental health services?
- What specialized mental health services (psychiatric and community-based psychosocial services) are available in each action location?
- How accessible are these services in terms of geographic location, cost, and cultural appropriateness?
Criterion Five: Quality, Appropriateness and Effectiveness of the Services
- How satisfied are beneficiaries with the quality, timeliness, and appropriateness of protection and MHPSS services provided?
- How effective have the cash-based interventions (Cash for Protection and MPCA) been in supporting protection outcomes?
Criterion Six: Contextual Factors and Adaptability
- What factors have enabled or constrained the impact of the protection and MHPSS interventions?
- How has the action adapted its approach to address evolving protection needs in the earthquake-affected areas?
- What issues/concerns of the target population remain unaddressed and why?
Criterion Seven: Intersectionality and Inclusion
- How effectively has the action addressed the diverse needs of different groups (e.g., based on gender, age, disability status)?
- What unintended outcomes (positive or negative) have resulted from the interventions?
- Timeframe
The study will take place between September-December 2024:
- 20 October: Deadline for Application
- 21-25 October: Selection of the Applicant
- 28-31 October: Contracting
- 4 November: Kick off Meeting
- 4-30 November: Finalization of Phase 1 and Preparation of Phase 2 (including submission of the inception report)
- 2 December – 31 December: Completion of Phase 2
- 15 January: Draft Report dissemination for feedback
- 31 January: Submission of final report and presentation
- Qualifications and Application Process
- Demonstrated experience in conducting complex evaluations in humanitarian contexts, particularly in protection and MHPSS sectors.
- Strong expertise in mixed-methods research, including quantitative and qualitative data collection and analysis.
- Familiarity with protection mainstreaming principles, MHPSS frameworks (particularly the IASC MHPSS framework), and cash-based interventions in humanitarian settings.
- Experience working in Türkiye or similar contexts, with knowledge of the refugee situation and earthquake response.
- Strong analytical and report writing skills.
- Ability to work in Turkish and English; knowledge of Arabic would be an asset.
- Demonstrated understanding of ethical considerations in protection-related research.
- Evaluation of Bids and Selection Criteria
This tender employs a 70/30 weighting between technical and financial bids for final selection. The technical assessment is based on 8 criteria, each contributing a specific percentage to the overall technical score. The most significant criterion is the assessment of proposed staff, accounting for 50% of the technical score. This emphasizes the importance of a qualified team with relevant experience. Other criteria evaluate the contractor’s project understanding, methodology, and implementation plan. The below list details each criterion and its weigh in the technical assessment.
Interpretation of the objectives in the ToR and critical examination of tasks: 10%
Description and justification of the contractor’s strategy and methodology for delivering the services: 5%
Design of the conceptual and analytical framework in line with the ToR: 5%
Proven relevant experience and sample of previous work(s): 10%
Presentation and explanation of the implementation plan: work steps, milestones and schedule: 5%
Approach and procedure for coordination with DKH and STL: 10%
Personnel assignment plan (who, when, what work steps) including explanation and specification of the expert working days: 5%
Assessment of proposed staff (qualifications, language, general Professional experience, protection experience): 50%
How to apply
Application Procedures
- Interested consulting firms should submit a technical and financial proposal by October 20, 2024. Please note that non-compliance with the list of documents shall make the applicant ineligible for further selection process.
- The technical proposal should include:
- Proposed methodology, approach and strategy
- Team composition and CVs of key personnel
- Relevant experience and sample of previous work
- Proposed timeline
- The financial proposal should detail all costs associated with the study in Euros in the form of distributed over the expected number of the engagement dates of the various team members (Team Leader, Key Experts and enumerators).
- Proposals should be submitted electronically to [[email protected]].
- Shortlisted candidates may be invited for an interview or to provide additional information.
- The selection process will take place between October 21-25, 2024.
- The selected applicant will be notified by October 25, and contracting will occur between October 28-31, 2024.
- If you have any questions please send us an email before the 12th of October at [email protected]
- For further information and required annexes, please refer to this link:https://ewde.sharefile.eu/public/share/web-s9ccf3d6d2bb447b1a1f07133e1c47c10