External evaluation of Cash Assistance in Re-emerging Markets in Syria CARMA 3

1. INTRODUCTION AND BACKGROUND

Recent needs assessments highlight the continuing protection risks faced by vulnerable Syrian households (HHs). The economic crisis has further exacerbated the situation, aggravating peoples’ dire livelihoods, food insecurity, protection needs, and negative coping strategies. In response, CARMA3 aims to provide life-saving and -sustaining assistance with cash as the modality at its core. Multipurpose cash transfers will address HHs basic needs gap. Protection services built on the learning from CARMA2, incl. case management (CM), Cash for Protection (CfP), provision of specialised in-kind services and goods tailored to individual needs and referral. Awareness raising sessions on protection concerns will be offered. CARMA3 will target Aleppo, Hasakeh, Rural Damascus, Damascus, Homs, Hama, Deir-ez-Zor and Dara’a, reaching 31,240 beneficiaries. Caritas Austria (CAUT), as the lead, will implement the action with Caritas Switzerland (CACH) as well as the national partners Caritas Syria (CS) and the Greek Orthodox Patriarchate of Antioch and all the East, Department of Ecumenical Relations and Development (DERD-GOPA). The joint challenges of scaling up support whilst continuing to invest in protection activities has required CAUT to review the efficiencies that can be made with partners. The international consortium has been reduced: CACH leads on basic assistance (BA) and CAUT takes responsibility for protection. Capacity building (CB) for partners will continue to be prioritised and drive more effective performance in CARMA3. CARMA3 will continue to set standards for cash programming in Syria, including CfP, proving its cost efficiency and hence allowing for further expansion and scaling. On a sector level, CARMA3 will commit to increased efforts and resources towards harmonisation across cash actors especially moving towards joint general vulnerability assessment frameworks and pushing deduplication through future common beneficiary platforms.

1. RATIONALE

The purpose of this final evaluation is institutional learning to improve future interventions designs. The final evaluation’s findings and recommendations will be used to focus on learning about emergency response and use of the cash modality for basic assistance and protection in the context of the Syria crisis response as well as replicating success stories and sharing lessons learned with partners and the community of practice. The final evaluation covers the 12 months of CARMA3 intervention. If feasible, the final evaluation will take place in Syria in various project sites and with both implementing partners as well as with all the funding/technical partners, covering the full scope of the programme across all results.

1. EVALUATION OBJECTIVE

The main objective of the final evaluation is to assess the short and medium-term impact of the project intervention on the lives of individual beneficiaries and their families, achieved through the delivery of a package of different emergency response activities and trainings for national partners.

The final evaluation is expected to

• Provide an assessment of the overall impact of the CARMA3 project against the specific and overall objectives as well as results as defined in the project’s logical framework.
• Deepen insight into assumptions and approaches underpinning the project, and how these compare to the priorities and concerns of beneficiaries.
• Assess the relevance of the project with regard to design and coherence, including the intervention logic and its assumptions.
• Assess the extent to which the project was effective in delivering the planned outputs and outcomes.
• Assess the extent to which the project has managed to achieve a positive impact on the lives of the project’s beneficiaries (disaggregated by women, men, girls and boys).
• Provide recommendations for possible future interventions of the same type.

This final evaluation will be conducted in line with the revised OECD/DAC evaluation criteria of 2019[1].

1. EVALUATION QUESTIONS
   1. Relevance (25%)

• To what extent were project design and approach appropriate to the situation?
• To what extent are the outcomes and indicators of the project still pertinent?
• Are the planned outputs of the project appropriate to meet the outcome and the overall goal (impact) – as part of the analysis of the logframe matrix / project strategy?
• Are the assumptions and risks still valid?
• To what extent have multiple sources of information (including affected people and communities, local institutions, etc.) been consulted when assessing needs, vulnerabilities and context?
• To what extent are the interventions (activities, targeting, transfer modalities) appropriate: culturally, social-economically and environmentally?
  1. Effectiveness (15%)
• To what extent has the programme already produced its expected outputs or will be likely to achieve them? To what extent has the programme already achieved its outcomes or will be likely to achieve them? What were the major factors influencing the achievements or non-achievements?
• To what extent is the programme likely to achieve real impact (change) for the target groups?
• How is project-generated (additional) disposable HH-income utilised or invested by target HHs (e.g. education, food security, others, etc.)?
• Did the programme contribute to capacity building as planned? To what extent is knowledge of the project teams exploited and contributing to the achieved results (benefits of knowledge management)?
• Was the main target group involved in the project planning phase? Are the statements of the main target group on the attainment of goals identical with the opinions of the actors having provided humanitarian assistance (e.g. employees of the respective organisation)?
• What have been the key challenges and/or risks hampering implementation of the project? How have these been addressed by the programme management? Which steering and/or mitigation measures have been taken? Have these been the right ones? How effective was the decision-making process?
• Were (local) coordination structures established? How did the organisations harmonise and coordinate their interventions with other partners? Was there a lead agency nominated? Which factors hindered and supported the coordination? Has the coordination led to more efficiency and impact of the intervention?
  1. Efficiency (15%)
• Was the project implemented in the most efficient way in terms of time and resource allocation, planning, coordination, and communication?
• To what extent has the project been managed and implemented as planned? In case: what issues occurred and why?
• Are constraints and risks regularly identified and analysed, and plans adapted accordingly?
• Was the transferring modality implemented in the most efficient way (cash through FSPs) compared to alternative modalities or transfer mechanisms?
• Were the budget and available financial resources realistic for the achievement of the intended objectives and outputs (to include staffing, training, capacity building)?
• To what extent were all items/equipment purchased and used as planned under this programme?
• Are programme management and decision-making processes within the programme team and steering committee efficient? What role does participation play in decision-making processes within the programme team?
• Are there clear processes in place to support monitoring and use of the monitoring-results for management and decision-making?
• In the CARMA project, what are core issues and benefits of working with multiple/different partners in order to create impact?
  1. Impact (25%)
• Are the results/outcomes of CARMA in line with the overall and specific objectives as formulated in the project’s logical framework and specified in the proposal? If any, what were unintended impacts of the programme?
• What has happened as a result of the project? What do beneficiaries consider the most significant changes brought about by the project in their lives (immediate impact)? How do beneficiaries evaluate the impact of the CARMA intervention in proportion to their overall needs?
• What were the positive and negative, intended and unintended effects?
• How was the money (cash grants to beneficiaries) used? What was the money used for (or in which sectors)? Were the investments of cash short-term or long-term?
• How where capacity building measures integrated into work practices of national partners?
• To what extent has the programme affected HH dynamics, female participation and/or inclusiveness in HH decisions? Who is in charge of the cash on HH level? How is it distributed on HH level and who is spending the cash (gender differences)?
• Have the cash transfers affected HH behaviours, such as use of coping strategies, changes in food consumption, changes in education or health practices? Is the transfer value sufficient to meet project objectives (specifically, meeting HH basic needs)?
• Did beneficiaries consider the vulnerability criteria fair and transparent and understand the selection process? Are any groups missed out in their opinion?
• What evidence is there of the impact that the project has had to date on persons of concern with special needs, women, the relations between the IDPs, returnees, and the host communities and the local economy? Which changes are evident and attributable to the project?
• Has the project reached the key groups who are at greatest risk? Have all of those in need of protection received protection during the interventions?
• To what extent has the CARMA intervention contributed to wider sector learning and/ or harmonisation?
  1. Sustainability (20%)
• To what extent will the benefits of a programme or project continue after the project has been completed? What are the major factors which influence the achievement or non-achievement of sustainability of the project?What steps are already and/or need to be taken in order to secure sustainability of the programme?
• To what extent did CARMA contribute to strengthening existing systems or to systemic change (e.g. Damascus-based Cash Working Group and Interagency referral system)?
• To what extent did CARMA contribute to building capacities and expertise of national partners?
• What are key considerations to keep in mind when developing a second phase of the project? What are key considerations to keep in mind when developing an exit strategy for the project?
• To what extent are the target groups and counterparts able to adapt sufficiently to external changes and shocks? How stable is the situation in the surrounding field regarding social justice, economic efficiency, political stability and ecological balance?

1. METHODOLOGY

It is expected that the final evaluation will employ quantitative as well as qualitative data, making use of different methods, such as analysis of documents, structured interviews, focus group discussions (if feasible), semi-structured interviews (face-to face or remote modality), etc. All data collected needs to disaggregated by sex and age of beneficiaries; be sensitive to gender disparities: seek to include persons with disabilities, and be prepared to probe about unintended outcomes of the project.

The methodology to appear in the proposal should include but not limited to

• Review of key project documents including: proposal, project reports, PDMs, previous evaluation reports from CARMA1 and CARMA2 and other related documents.
• Overview of any changes based on PDM findings.
• Review of CARMA3 process and outcome monitoring data as well as other data collected in the course of the CARMA intervention (e.g. primary data for outcome and process monitoring, etc.)
• Review of the effectiveness of feedback, complaints and response mechanism.
• Interview key staff associated with the project and other stakeholders with each meeting documented (face-to-face or remote modality).
• Meet and interview project beneficiaries at community level, structures and document all discussions (only if feasible).
• Undertake physical verification visits to project sites or structures to affirm level and results exhibited (only if feasible).

Consultant(s) will be asked to specify methodology for the final evaluation in their technical proposal submitted with the application.

Given potential political instability and uncertainty in the region at the time of field presence, this plan may need to be adapted flexibly and stakeholder meetings may have to be planned and organized in a way to meet legal requirements and minimize exposure to risks. Further remote modalities for meetings can be utilized and CARMA3 consortium members can provide support as needed.

1. DELIVERABLES
   • Describe the conceptual framework reflecting the ToR

Inception Report

In the inception report, the evaluator(s) will describe the design of the final evaluation and will elaborate on how data will be obtained and analyzed. The report should at a minimum:

• Highlight any reservations regarding the feasibility and/ or limitations
• Outline the methodology and provide a timeline for the data collection and review
• Contain tools/ templates for data collection during data collection (if feasible and/ or necessary)
• Include selection of locations for data collection and sampling methodology
• Provide an analysis of consulted documentation
• Describe utilisation of software for data analysis and visualisation to enhance quality and efficiency of the final report
• Present outline of the final report

The project management has the possibility to give feedback on the inception report and proposed design. Data triangulation and quality control are very important and need to be discussed in the inception report. The inception report needs approval by CAUT before entering the research & evaluation phase.

CAUT is aware that the limited timeframe for the evaluation can pose a challenge to building the trust/rapport required to get to in-depth conversations; the evaluator(s) is/are invited to reflect on this in the inception report.

Final Draft Evaluation Report

The report includes a draft executive summary and clear findings and recommendations to be read and commented by all project partners. The findings and recommendations of the draft final report have to be structured according to the evaluation questions. To be approved by CAUT.

Final Evaluation Report [2]

The final evaluation report should be clear, well-structured (along the DAC criteria and according to the evaluation questions) and should not exceed 30-35 pages.

An outline of the report’s structure needs to be agreed upon during the inception phase, but should include at least:

• Cover page, table of contents, list of abbreviations
• Reflection on research ethics and limitations
• Executive summary including recommendations (three to five pages)
• Description of the review objective(s), methodology and activities
• Discussion of findings (presented per the specific tasks – Relevance, Efficiency, Effectiveness, Impact and Sustainability)
• Conclusions and Recommendations
• Annexes (e.g. list of reference documents, list of interviewed persons, minutes of meetings)

Findings and recommendations should be based on a critical analysis. To be approved by CAUT.

For more details please refer to the Terms of Reference an associated documentation at the link provided

https://wolke.caritas.at/s/pw987RRCdBFkR2w

Password: bppzaH6QmD