Save the Children
Context
Save the Children is an international non-governmental organization that aims to improve the lives of children worldwide. This evaluation will be carried out for the Domestic department of Save the Children Netherlands.
This impact evaluation is for TeamUp, a psychosocial movement-based intervention for children aged 6-17 years, affected by conflict, displacement and adversity. The program was developed by War Child, Save the Children and UNICEF Nederland in 2016 in the Netherlands as a response to the worldwide Syrian refugee crisis. Since then, TeamUp has expanded rapidly and is currently active in more than 26 countries worldwide in a variety of development and humanitarian settings.
TeamUp applies a non-verbal and embodied-learning approach, consisting of a wide variety of group movement-based games, sports, creative movement, routines, body awareness and breathing exercises. The intervention aims to enhance the psychosocial wellbeing of children and teenagers by improving their sense of safety, strengthening their social connectedness, promoting self-regulation skills and encouraging a positive outlook towards their future.
This impact evaluation takes place in the Dutch asylum reception centers, where TeamUp is currently active in around 40 locations for diverse cultural backgrounds and age groups in the range of 6-17 years. The program has been implemented in this setting for five years and has funding until at least 2024. An estimated 4500 children will participate in the weekly TeamUp sessions in 2023. In the Dutch refugee setting, TeamUp is carried out by TeamUp facilitators, who are volunteers coming from a variety of relevant educational backgrounds e.g. education, sports, mental health, etc.). All facilitators are carefully selected and are trained and supervised by dedicated TeamUp trainers. A TeamUp session is carried out by 2-5 facilitators and targets a total of ten children on average.
The project is funded by the EU foundation of Asylum, Migration and Integration (AMIF).
Timeframe
March 2023 – December 2023
Objectives and research questions
The overall objective of the evaluation relates to the extent to which TeamUp’s intended outcomes and impact are achieved:
· Assess how and why TeamUp contributes to the psychosocial wellbeing of children and teenagers, ages 6-17 years, in Dutch refugee settings via the four key components as defined by the Theory of Change (sense of safety, social connectedness, self-regulation and positive outlook)
· Identify how TeamUp’s intended outcomes for children can be better achieved
· Contribute to the overall evidence base of the TeamUp program
· Give children’s and their parents’ perspectives and lived experiences a stronger voice in the further development of TeamUp’s Theory of Change
Key research questions
How, why and under which circumstances does TeamUp contribute to promoting the psychosocial wellbeing of children in Dutch asylum reception centers?
Subquestions
1. How and under which circumstances do TeamUp’s intended outcomes for children occur?
2. What are the key factors that cause the outcomes?
3. Does TeamUp contribute to these changes? If so how and why?
4. How can TeamUp be made more relevant to age, gender and cultural background of children living in Dutch asylum reception centers?
Methodology
Contribution analysis will form the overall approach for this evaluation, which employs a variety of methods to examine the research questions.
Literature and document review: review relevant literature on methods used, relevant published and grey literature, and review results from earlier TeamUp studies, evaluations and monitoring efforts. TeamUp’s Theory of Change will form the overall framework for this contribution analysis.
Data collection: use qualitative and participatory and quantitative data collection methods as appropriate to respond to the evaluation questions. Due to the complexity of the context (see Annex 1), the impact evaluation will rely less heavily on quantitative methods but more on qualitative and participatory methods such as participatory action research, process tracing, outcome- and narrative based methods. Key stakeholders such as children and parents should be given an active role in designing, collecting and analyzing the data. By using such participatory methodology, evidence building to support and refine TeamUp’s contribution claims will need to follow an iterative process that may require several rounds of data collection and engagement with key stakeholders.
Analysis and report drafting: compile and synthesize primary and secondary findings into a presentation and report, including the deliverables listed further below.
Deliverables
· Evaluation proposal based on this terms of reference. Maximum budget should not exceed €60.000 including VAT but excluding travel and hotel expenses. Travel and hotel expenses should be included in the proposal. Budget needs to be specified based on daily or hourly rates.
· Upon kick-off of the evaluation in the presence of relevant program team members, the identified evaluator/s will prepare an inception report which details their understanding of the assignment and how the research questions will be addressed. The report will include the design, methodology, research questions, sampling approach and size, risk matrix, data collection tools and quality assurance metrics and ethical considerations pertaining to data collection activities.
· After feedback from the program team, final evaluation inception report, to be submitted to an ethical review board including all necessary documentation.
· Train the program team participating in the evaluation in contribution analysis and overall approach to the data collection activities.
· Advise on how to best carry out the internal process evaluation (see Annex 3) at key stages during the evaluation, so that it aligns with the main outcomes and impact evaluation.
· Facilitate learning workshop with the program team to exchange preliminary learnings from the impact evaluation and the process evaluation (see Annex 3) around mid-point of the evaluation.
· Upon completion of first round of data collection: share draft report outlining preliminary findings with relevant program team members for feedback.
· Upon completion of data collection activities: prepare final findings report outlining primary and secondary findings.
· Final Report (max 40 pages including graphs and figures) including literature review, findings organized under each research question, conclusions, lessons learnt and recommendations.
· It is desirable to contribute in 2024 to write a publication on the findings in collaboration with relevant program staff.
Key Stakeholders and Respondent population
1) Children who participate in TeamUp. Most common cultural backgrounds include Syrian, Turkish, Afghani and Eritrean
2) Parents and caregivers of the children, who also live in the asylum reception centers together with the children
3) TeamUp facilitators coming from diverse backgrounds themselves
4) TeamUp program staff, which includes recruitment focal points, trainers, and regional coordinators
5) Relevant staff of partner organization COA, the central organization that manages asylum reception centers in the Netherlands. TeamUp is carried out with their permission, collaboration and support
The ways in which these stakeholders are involved will be decided on together with the consultants and other relevant the program team
Collaboration with Program team
The evaluation team of external consultants will be supported by an internal TeamUp team of MEAL staff, relevant program staff and a team of data collectors, and researchers of the War Child Research and Development team. An evaluation reference group will advise on the evaluation at key stages.
The consultants will be hired by Save the Children and managed by the TeamUp MEAL Specialist who will manage the overall evaluation and help coordinate the data collection activities in the field.
Indicative timeline 2023
When
What
February
Decision of external party based on the Terms of Reference
March
Refine and finalize evaluation design including (sub)questions, methodology and stakeholder engagement
March
Submission for Ethical review (process will depend on approach and methods chosen)
April – July
Data collection including analysis midterm
September – October
Final data collection and start analysis
November / December
Analysis, conclusions and final report including reflection on learnings and challenges during the evaluation
As the above planning is indicative, the final planning will be decided upon together with the program team after finalizing the evaluation design and inception report.
Dissemination
The final report will be shared with the donor AMIF, TeamUp staff, the partner COA (Netherlands official central organization that manages asylum centers), Save the Children, War Child. In 2024 an article will be written on the findings resulting in a publication.
Ethics and child safeguarding
Save the Children upholds high standards for ethics and child safeguarding. All staff, partners and consultants should adhere to Save the Children’s Child Safeguarding Policy and Code of Conduct. Consultants who do not have training on child safeguarding will be trained.
As children will be actively engaged in disclosing information relating to their psychosocial wellbeing and sense of safety, this evaluation will involve (at least minor) child safeguarding risks. Depending on the final methods and approach chosen, will likely require ethical approval from an external ethical committee (Save the Children’s ethical review board or an IRB). This will be decided as part of finalizing the evaluation design.
At the least, a risk assessment will be carried out and risks (foreseen and unforeseen) will be managed throughout the evaluation.
Requirements
The consultant should meet the following requirements:
· A track record of successfully completed similar assignments (e.g. research studies, evaluations, publications, etc.). This includes a track record of contribution analysis approach and impact evaluations using theory-based evaluation methods such as described in the methodology section. Specification of how the mix of methods ensure relevance for the key stakeholders and strength of evidence is an asset.
· Experience in designing and applying participatory methods with children
· Knowledge and understanding of programs to support child protection, psychosocial wellbeing and resilience of children.
· Excellent interpersonal skills and an ability to communicate effectively with a wide range of audiences in terms of status, geography and cultural background.
· Ability to work in English, Dutch an asset.
· Registration as a business or independent contractor.
How to apply
The consultant will be hired and managed by Save the Children’s MEAL specialist in The Hague, Netherlands. Interested candidates are requested to provide the following information:
· CV and cover letter, in English, setting out how they meet the requirements above and including links to 3 relevant examples of previous work
· A short description of a proposed methodology and financial proposal (3-5 pages)
· All applications should be submitted to the TeamUp Netherlands MEAL Specialist [email protected] by February 24 2023
For any questions relating to this consultancy, please contact Tom Zwollo: [email protected]
Save the Children upholds an active policy of security for children and does not accept or tolerate any form of abuse. This subject is part of our selection procedure. Save the Children takes part in the Agency Misconduct Disclosure Scheme. As part of the selection process, Save the Children will request standardized references in line with the Inter-Agency Misconduct Disclosure Scheme and it will be required to present a recent Certificate of Conduct. When applying to this opportunity you agree to consent with this procedure.
At Save the Children we are proud of diversity. For this reason we invite everyone to join our movement, regardless of race, sexual orientation, religious or cultural background, character, or other visible or invisible traits. We celebrate our differences because this ensures that we grow together.
Selection criteria
The received offers will be assessed upon the following criteria:
Quality of the proposal and evaluation design, including a sound mix of proposed methods that lead to strength of evidence
40%
Track record with conducting relevant impact evaluations involving children in development and humanitarian settings
30%
Value for money
20%
Track record in designing and implementing participatory methods for children in evaluations
10%
Annex 1: Additional information on background and context
This impact evaluation takes place after more than five years of implementation of the TeamUp program in the Dutch asylum reception centers’ context and will continue at least until 2024. It is being implemented in a variety of asylum reception centers, such as centers that have a more stable nature, transitory locations, and emergency reception centers. In recent years, the context of Dutch refugees has changed dramatically as waves of refugees arrived to the Netherlands, including large numbers of refugees from Afghanistan, Turkey, and the Ukraine. Ukrainian refugees receive special status in the Netherlands including housing organized by municipalities, and receive added services compared to other refugees. For this reason, Ukrainian refugees do not participate in TeamUp sessions as they do not live in asylum reception centers.
As a response to the high influx of refugees, the Dutch government set up temporary emergency asylum reception centers in 2022. Although these were set up to open for maximum of six months, many have stayed open for a longer time. In 2023, a larger portion of TeamUp locations will take place in emergency settings.
Sufficient staffing and adequate living conditions for newly arrived refugees have come under a lot of strain, which impacts the services of COA and very likely the stress of children and their parents living in Dutch reception asylum centers. Late 2022 the Dutch government made it mandatory for municipalities to provide reception services to refugees, with the aim to be able to better distribute refugees across the country and improve their living conditions. It is to be seen how this plays out.
This evaluation takes place within this rapidly changing and complex context. In addition, there is considerable complexity involved to measure outcomes and impact in the Dutch refugee setting because:
· TeamUp has a step-in step-out approach so children can attend TeamUp sessions whenever they like
· Children and their families move regularly from one reception center to the next
· The composition of TeamUp teams often changes because the trained volunteers come and go, even more so in the current climate where many sectors are understaffed – team changes can have an impact on the quality of implementation
· There are many actors and organizations active in asylum reception centers that can have influence on psychosocial wellbeing of children
· Within asylum reception centers, the settings are also diverse. There are for instance regular reception centers, family locations and emergency reception centers, and centers for unaccompanied minors
This evaluation will rely on methods that suit the complexity of the context.
Annex 2 TeamUp study scope and evidence base
TeamUp program was developed to become an example of an evidence-based intervention. Research is done on its efficacy and effectiveness, and during implementation, the program’s feasibility and quality of care (attendance, adherence, competence) are assessed (see table 1 below).
To ensure that the evaluation builds on findings so far, a review is needed of existing monitoring, evaluation and research data, especially in the Dutch reception center context.
For the TeamUp program worldwide, there have been a several evaluations and research done so far to assess the process and intended outcomes as explained in the Theory of Change (see Annex 4).
• 2018/2019: Process evaluation in Dutch reception centers: implementation quality, processes and stakeholder perceptions of implementation and outcomes; (1)
• 2019/2020 Formative evaluation of TeamUp in Dutch schools: implementation quality, stakeholder feedback and potential outcomes
• 2020 Development and rollout of the TeamUp monitoring framework using a standardized set of indicators and tools to measure them (quantitative and qualitative)
• 2020 Development of TeamUp’s Theory of Change including visual, narrative and literature review
• 2020 Quasi-experimental study in Ugandan schools. Publication expected 2023.
• 2023 Randomized-controlled trial in Burundi for TeamUp to become an Evidence based intervention
Table 1: road to impact, as envisioned by War Child Research and Development
Where the Uganda quasi-experimental study in 2021 and upcoming RCT of TeamUp in Burundi seek explain the effectiveness of TeamUp and answer the What (does TeamUp reach the intended effects and if so to what extent), it has not been assessed How and Why the intended changes take place. This evaluation aims to help fill this gap.
For academic research and for this impact evaluation, the Theory of Change is the framework for the assessment.
Besides deeper research and evaluations, TeamUp’s quality of implementation and potential outcomes are continually monitored through its monitoring and evaluation system, which in the Netherlands, is done annually.
• Monitoring of the quality of implementation is done mostly through the three so called quality of care indicators attendance (how many children attend and how often do they attend?), adherence (is TeamUp being implemented according to the methodology?) and competence (do TeamUp facilitators have the competencies that are needed to do TeamUp well?)(2). Standardized instruments have been developed to measure these three indicators.
• Monitoring of outcomes is done through child friendly focus group discussions and 1on1 conversations about satisfaction, outcomes and children’s suggestions for improvement.
Literature cited:
1 Bleile, A.C.E., Koppenol-Gonzalez, G.V., Verreault, K. et al. Process evaluation of TeamUp: a movement-based psychosocial intervention for refugee children in the Netherlands. Int J Ment Health Syst 15, 25 (2021)
2 Jordans MJD, Kohrt BA (2020). Scaling up mental health care and psychosocial support in low-resource settings: a roadmap to impact. Epidemiology and Psychiatric Sciences 29, e189, 1–7. https:// doi.org/10.1017/S2045796020001018
Annex 3 Process evaluation (out of scope for this Terms of Reference)
Simultaneously, a process evaluation will be carried out internally, with the aim to assess the quality of implementation and identify ways to further make practical improvements in the program. The process evaluation will be led internally and is out of scope for this Terms of Reference, however insights from the impact evaluation will inform the process evaluation and vice versa.
Background
Competent facilitators and stable teams, leading to qualitatively good sessions according to the TeamUp method, are a prerequisite to achieve any outcome for children. For this reason, this evaluation includes a part on the quality of implementation. The process evaluation and monitoring data shows that the quality of implementation of TeamUp in Dutch reception centers is often sufficient or good, however this varies per team. In 2022, the stability of teams has become a bigger challenge than before in part due to the job market in the public sector (employers have many options to choose from, leaving less room for volunteers). To improve the stability of teams and increase the quality of TeamUp sessions, a pilot will be started with paid facilitators in 2023 in some locations. Data collection during the pilot will be integrated into the process evaluation.
The process evaluation will look at the so-called quality of care indicators, which can be viewed as the minimum indicators for quality programming:
· Attendance (how many children attend and how often do they attend)
· Competence (do TeamUp facilitators have the required competencies in order to do TeamUp well)
· Adherence (is TeamUp being implemented well enough according to the methodology to produce its expected effects)
Besides using the standardized TeamUp tools to measure these indicators, this process evaluation will follow the same line of questioning as the outcomes and impact part of the evaluation. Specifically, the internal process evaluation seeks to answer the following research questions:
1. What are the key factors that need to be in place to have quality TeamUp sessions?
1.1 Which preconditions are needed to have competent TeamUp facilitators and stable teams leading to quality TeamUp sessions?
2 What elements of TeamUp need to be adapted to become more effective and efficient in our service delivery and increase and relevance for the children without affecting current success?
2.1 How can quality TeamUp sessions best be achieved?
2.2 What are the contributing factors and challenges to achieving adequate child attendance, facilitator competence and adherence? What are the underlying contextual factors and mechanisms that are at play?
3 Would a mix of paid and volunteer facilitators as alternative scaling model be more effective to sustain and grow the TeamUp program within the Dutch refugee context? If so how and why? What are the contributing factors and challenges?
Annex 4 Theory of Change TeamUp visual
The full Theory of Change narrative includes a literature review and detailed description of each outcome and explanation of how TeamUp contributes to this outcome. It can be shared upon request.