Evaluation of Cash Assistance (CVA) to refugees from Ukraine in Poland

1) Background

Starting from June 2022, Diakonie Katastrophenhilfe (DKH) and Diakonia Poland, with technical assistance of Support to Life (STL), provided multi-purpose cash assistance (MPCA) to vulnerable families, who have been displaced from Ukraine to Poland. Implemented in coordination with the humanitarian actors in Cash Working Group and local authorities, the purpose of this assistance was to cover the immediate needs of vulnerable families in the aftermath of displacement. The project targeted most vulnerable refugees mainly in 3 provinces and 7278 families (19632 individuals) were reached through the registration centers established in Kalisz, Ledziny, and Sosnowiec.

The MPCA was provided to eligible families after the registration and verification process. Parishes, social services offices, local NGOs and social media were used for the identification and referral of the most vulnerable refugee families. To apply for the assistance, refugees registered themselves into the system through a website and made an appointment to complete their registration at the temporarily established Registration Centers. In those registration centers, all the information provided in the self-registration has been verified by the registration clerks. After the verification process, eligible refugees received their assistance through Conotoxia debit (visa) cards. Monthly cash transfers were made to each card for 3 months with the head of household receiving 700PLN and each additional member 600PLN.

While winter is approaching, the project will include an additional top-up for winter needs and existing beneficiaries with ongoing vulnerabilities will be provided a one-off winterization assistance.

2) Evaluation

Summary

Description: This ex-post evaluation will be conducted in the aftermath of two consecutive (MPCA and Cash for Winterization) projects implemented by Diakonie Katastrophenfile and Diakonie Poland. The evaluation questions will be based on the following criteria: partnership, efficiency and effectiveness, best practices, replication.

Location: Poland (Kalisz, Ledziny, Sosnowiec) + Online interviews

Duration and timing: 40 working days. 01 December 2022 – 25 February 2023

Summary outputs:

• Inception report
• Executive summary
• Full report covering evaluation of the contract process with financial service provider for internal learning evaluation of the cooperation with the financial service provider, internal Evaluation Report (including detailed description of scope, purpose, methodology, key findings, best practices and detailed recommendations for the Diakonie Katastrophenhilfe and implementing partners), with dedicated separate evaluations of both projects (Cash and Winterization)
• Publication (3-4 pages, on best Practice for the general public, including the description of the Project Implementation (external report)
• 4 case studies (with photos) demonstrating project impact (two for MPCA and two for winterization).

Evaluation team / profile: The evaluation team can consist of 2-3 persons (or more, if deemed necessary and if justified). The team should be led by one team leader, who will act as the main contact point between the evaluation team and DKH. Evaluators should have prior evaluation experience in humanitarian emergencies, preferably in sectors of multi-purpose cash assistance, cash for basic needs and/or other cash-based interventions.

Purpose

The overall objective of the evaluation is to draw lessons from the MPCA and Winterization components of the project in order to enable replication of good practices in the future, and to address any potential weaknesses in implementation. The evaluation will focus on all steps of the project life cycle starting from initial design, contacting of service providers, setting up the operation, implementation and all the way up to final reporting. The evaluation is expected to go beyond a post-distribution monitoring exercise and achieve an overall evaluation of the whole program, presenting concrete recommendations for improvement that would feed into future interventions.

Evaluation Questions

Organizational CVA Preparedness Evaluation

1. What are the strengths, weaknesses, opportunities and threats (SWOT) of DKH planning and set up process (preparedness)? Were there adequate tools and mechanisms (e.g. internal processes, FSP procurement and contracting, framework agreements, operational procedures, roles and responsibilities, and similar) in place to enable swift program planning and response? (Including recommendations and lessons learnt for future responses)
2. Were implementation tools and instruments developed and provided for staff in time?
3. Did DKH provide adequate support?
4. Evaluation of collaboration with Financial Service Provider (FSP), especially with regard to feasibility of future upscaling.
5. What are the good practices, key learnings and potential recommendations across all categories and questions (incl. identification of practices to be replicated and scaled up)?
6. What concrete steps would need to be taken at DKH HQ and global level for increased organizational preparedness for CVA delivery in emergencies?

Partnership Evaluation

1. What are the strengths, weaknesses, opportunities and threats of the partnership between the different stakeholders involved in this project (DKH, Diakonie Poland, STL, local parishes, etc.)?
2. How well did the partnership work according to the partners (DKH, Diakonie Poland, STL, local parishes etc.)? What worked well what can be improved?
3. Review of partnership pilot to work with our partner STL – learnings and recommendations.
4. What is the added value of working through local actors and networks (e.g. registration, targeting, verification, duplication, implementation, MEAL, communication etc.)?

Projects Evaluation

1. Did the projects address beneficiaries most urgent needs and priorities at the right time?
2. Were each project’s objectives achieved?
3. Were recipients satisfied with the project in general?
4. Were beneficiaries treated with respect and dignity by all parties throughout the project?
5. What needs did beneficiaries cover with the cash? How did their situation improve as a result of the response?
6. Are there appropriate and accessible systems of accountability in place which beneficiaries are using?
7. Are there feedback and complaint management procedures in place?
8. Is new learning (incl. feedback) systematically captured and acted upon during implementation?
9. Was the project successful in addressing service gaps and in avoiding duplication?
10. Did people receive the correct amount of cash? What costs were borne by the beneficiary in receiving and using the cash?
11. Were any cases of misappropriation/ fraud by agency staff, local elites or authorities involved in targeting or distribution?

Methodology

The evaluation should make use of primary and secondary data collection methods. Primary methods include focus group discussions and key informant interviews with beneficiaries, Diakonia Poland, STL, DKH HQ staff and the financial service provider Conotoxia. Secondary data can be obtained from existing internal reports and surveys, as well as relevant external documents. Random or purposive sampling methodologies may be used as appropriate, depending on specific evaluation questions. Further details can be agreed upon at a later stage

Tasks

• Planning of evaluation activities
• Desk research
• Field research (on the spot and/or remotely)
• Analysis -Reporting

Evaluation Team – Roles and Responsibilities

The evaluation team can consist of 2-3 persons (or more, if deemed necessary and if justified). The team should be led by one team leader, who will act as the main contact point between the evaluation team and DKH. Ideally, the evaluation team should have one Ukrainian-speaking member to facilitate direct engagement with community members, or arrange translation services and at least one evaluator should be female.

Evaluators should have prior evaluation experience in humanitarian emergencies, preferably in sectors of multi-purpose cash assistance, cash for basic needs and/or other cash-based interventions. The evaluators should be able to demonstrate that they have working knowledge of the Core Humanitarian Standard (CHS) and will be able to act in accordance with humanitarian principles, avoiding risks of doing harm at every stage of the evaluation. A permission for travel in the EU, for a presence in Poland and if necessary in Berlin are required. Interviews with staff from Berlin can be realized remotely.

The evaluation team must demonstrate:

• Previous experience with and extensive knowledge of cash and voucher programming (including conditionality and restrictions), and thus best practice globally and regionally of designing and implementing Cash Transfer Programs
• Experience with conducting similar kind of evaluations;
• Experience with participatory evaluation and qualitative and quantitative methods;
• Familiarity with and clear comprehension of SPHERE standards, humanitarian principles, Core Humanitarian Standards (CHS) and a conflict-sensitive approach to evaluations
• Excellent English skills; – speaking and writing; Ukrainian skills would be an asset, if not, interviews must be conducted with the support of an interpreter;
• Ability to travel to project sides in Poland