External evaluation of Cash Assistance in Re-emerging Markets in Syria CARMA 2

1. INTRODUCTION AND BACKGROUND

Recent needs assessments highlighted multiple protection risks that vulnerable Syrian households face and their intersectionality. The economic and C19 crisis further exacerbated the situation, aggravating peoples’ dire livelihoods, food insecurity, protection needs, and negative coping strategies. In response, CARMA2 provides life-saving and life-sustaining assistance with cash as modality at its core. Multipurpose cash transfers (incl. cash for winter) to address the basic needs gap, Protection services such as Individual Protection Assistance through unconditional Cash for Protection and provision of specialised services and goods tailored to individual needs. CARMA2 has targeted more than 6000 households in Aleppo, Hasakeh, Rural Damascus, Damascus, Homs, Hama, Deir-ez-Zor and Dara’a over a period of 20 months. Caritas Austria (CAUT) leads a consortium to implement the action together with Caritas Switzerland and Diakonie Katastrophenhilfe as well as the national partners Caritas Syria and the Greek Orthodox Patriarchate of Antioch and all the East, Department of Ecumenical Relations and Development (GOPA-DERD). CARMA2 aims to set standards for cash programming in Syria, and on a sector level, has committed increased efforts and resources towards harmonisation across cash actors especially joint general vulnerability assessment frameworks and pushing deduplication through future common beneficiary platforms.

1. RATIONALE

The purpose of this evaluation is institutional learning to improve future interventions designs. The evaluation’s findings and recommendations will be used to focus on learning about emergency response and use of cash based interventions for basic assistance and protection in the context of the Syria crisis response as well as replicating success stories and sharing lessons learned with partners and the community of practice. The full project evaluation covers the 20 months of CARMA2 intervention. If feasible, the evaluation will take place in Syria in various project sites and with both implementing partners as well as with all the funding/technical partners, covering the full scope of the programme across all results.

1. EVALUATION OBJECTIVE

The main objective of the evaluation is to assess the short and medium-term impact of the project intervention on the lives of individual beneficiaries and their families, achieved through the delivery of a package of different emergency response activities and trainings for national partners.

The end of project evaluation is expected to

• Provide an assessment of the overall impact of the CARMA2 project against the specific and overall objectives as well as results as defined in the project’s logical framework.
• Deepen insight into assumptions and approaches underpinning the project, and how these compare to the priorities and concerns of beneficiaries.
• Assess the relevance of the project with regard to design and coherence, including the intervention logic and its assumptions.
• Assess the extent to which the project was effective in delivering the planned outputs and outcomes.
• Assess the extent to which the project has managed to achieve a positive impact on the lives of the project’s beneficiaries (disaggregated by women, men, girls and boys).
• Provide recommendations for possible future interventions of the same type.

This evaluation will be conducted in line with the revised OECD/DAC evaluation criteria of 2019

1. EVALUATION QUESTIONS
   1. Relevance (25%)

• To what extent were project design and approach appropriate to the situation?

• To what extent are the outcomes and indicators of the project still pertinent?

• Are the planned outputs of the project appropriate to meet the outcome and the overall goal (impact) – as part of the analysis of the logframe matrix / project strategy?

• Are the assumptions and risks still valid?

• To what extent have multiple sources of information (including affected people and communities, local institutions, etc.) been consulted when assessing needs, vulnerabilities and context?

• To what extent are the interventions (activities, targeting, transfer modalities) appropriate: culturally, social-economically and environmentally?

  1. Effectiveness (15%)

• To what extent has the programme already produced its expected outputs or will be likely to achieve them? To what extent has the programme already achieved its outcomes or will be likely to achieve them? What were the major factors influencing the achievements or non-achievements?

• To what extent is the programme likely to achieve real impact (change) for the target groups?

• How is project-generated (additional) disposable HH-income utilised or invested by target HHs (e.g. education, food security, others, etc.)?

• Did the programme contribute to capacity building as planned? To what extent is knowledge of the project teams exploited and contributing to the achieved results (benefits of knowledge management)?

• Was the main target group involved in the project planning phase? Are the statements of the main target group on the attainment of goals identical with the opinions of the actors having provided humanitarian assistance (e.g. employees of the respective organisation)?

• What have been the key challenges and/or risks hampering implementation of the project? How have these been addressed by the programme management? Which steering and/or mitigation measures have been taken? Have these been the right ones? How effective was the decision-making process?

• Were (local) coordination structures established? How did the organisations harmonise and coordinate their interventions with other partners? Was there a lead agency nominated? Which factors hindered and supported the coordination? Has the coordination led to more efficiency and impact of the intervention?

  1. Efficiency (15%)

• Was the project implemented in the most efficient way in terms of time and resource allocation, planning, coordination, and communication?

• To what extent has the project been managed and implemented as planned? In case: what issues occurred and why?

• Are constraints and risks regularly identified and analysed, and plans adapted accordingly?

• Was the transferring modality implemented in the most efficient way (cash through FSPs) compared to alternative modalities or transfer mechanisms?

• Were the budget and available financial resources realistic for the achievement of the intended objectives and outputs (to include staffing, training, capacity building)?

• To what extent were all items/equipment purchased and used as planned under this programme?

• Are programme management and decision-making processes within the programme team and steering committee efficient? What role does participation play in decision-making processes within the programme team?

• Are there clear processes in place to support monitoring and use of the monitoring-results for management and decision-making?

• In the CARMA project, what are core issues and benefits of working with multiple/different partners in order to create impact?

  1. Impact (25%)

• Are the results/outcomes of CARMA in line with the overall and specific objectives as formulated in the project’s logical framework and specified in the proposal? If any, what were unintended impacts of the programme?

• What has happened as a result of the project? What do beneficiaries consider the most significant changes brought about by the project in their lives (immediate impact)? How do beneficiaries evaluate the impact of the CARMA intervention in proportion to their overall needs?

• What were the positive and negative, intended and unintended effects?

• How was the money (cash grants to beneficiaries) used? What was the money used for (or in which sectors)? Were the investments of cash short-term or long-term?

• How where capacity building measures integrated into work practices of national partners?

• To what extent has the programme affected HH dynamics, female participation and/or inclusiveness in HH decisions? Who is in charge of the cash on HH level? How is it distributed on HH level and who is spending the cash (gender differences)?

• Have the cash transfers affected HH behaviours, such as use of coping strategies, changes in food consumption, changes in education or health practices? Is the transfer value sufficient to meet project objectives (specifically, meeting HH basic needs)?

• Did beneficiaries consider the vulnerability criteria fair and transparent and understand the selection process? Are any groups missed out in their opinion?

• What evidence is there of the impact that the project has had to date on persons of concern with special needs, women, the relations between the IDPs, returnees, and the host communities and the local economy? Which changes are evident and attributable to the project?

• Has the project reached the key groups who are at greatest risk? Have all of those in need of protection received protection during the interventions?

• To what extent has the CARMA intervention contributed to wider sector learning and/ or harmonisation?

  1. Sustainability (20%)

• To what extent will the benefits of a programme or project continue after the project has been completed? What are the major factors which influence the achievement or non-achievement of sustainability of the project? What steps are already and/or need to be taken in order to secure sustainability of the programme?

• To what extent did CARMA contribute to strengthening existing systems or to systemic change (e.g. Damascus-based Cash Working Group and Interagency referral system)?

• To what extent did CARMA contribute to building capacities and expertise of national partners?

• What are key considerations to keep in mind when developing a second phase of the project? What are key considerations to keep in mind when developing an exit strategy for the project?

• To what extent are the target groups and counterparts able to adapt sufficiently to external changes and shocks? How stable is the situation in the surrounding field regarding social justice, economic efficiency, political stability and ecological balance?

1. METHODOLOGY

It is expected that the evaluation will employ quantitative as well as qualitative data, making use of different methods, such as analysis of documents, structured interviews, focus group discussions (if feasible), semi-structured interviews (face-to face or remote modality), etc. All data collected needs to disaggregated by sex and age of beneficiaries; be sensitive to gender disparities: seek to include persons with disabilities, and be prepared to probe about unintended outcomes of the project.

The methodology to appear in the proposal should include but not limited to

• Review of key project documents including: proposal, project reports, PDMs and other related documents.
• Review of CARMA2 process and outcome monitoring data as well as other data collected in the course of the CARMA intervention (e.g. primary data for outcome and process monitoring, etc.)
• Interview key staff associated with the project and other stakeholders with each meeting documented (face-to-face or remote modality).
• Meet and interview project beneficiaries at community level, structures and document all discussions (only if feasible).
• Undertake physical verification visits to project sites or structures to affirm level and results exhibited (only if feasible).

Consultant(s) will be asked to specify methodology for the end of project evaluation in their technical proposal submitted with the application.

Given potentially continuing COVID-19 restrictions at the time of field presence, this plan may need to be adapted flexibly and stakeholder meetings may have to be planned and organized in a way to meet legal requirements and minimize exposure to health risks. Further remote modalities for meetings can be utilized and CARMA2 consortium members can provide support as needed.