Jesuit Refugee Service
1. PROGRAM OVERVIEW
Program name & location : Sustaining Education for Burmese Refugees, Mae Hong Son and Kayah State. Thailand, Mae Hong Son Province, Muang
Program duration : July 2018 – June 2022
Program sector/s : Education
Implementing agency : Jesuit Refugee Service, Asia Pacific (JRSAP)
Donor funding source : Caritas Australia, Australian Aid, Private Donor
Total budget : AUD380,000 (July 2021 to June 2022)
Total local downstream partners and name : Zetaman and KnED/STEDF
2. PROGRAM BACKGROUND
This project supports access to education for children residing in refugee camps in Mae Hong Son, Thailand and returnees in Kayah State, Myanmar. Through education and social cohesion activities, the project provides education services to refugee children, supports education activities with parents and adults, and strengthen capacity of the refugee education department, Karenni Education Department (KnED). The refugee children gain learning so they can continue their education upon return or resettlement, with the aim of smooth transition of teachers and students to their return sites in Myanmar.
JRS in Thailand collaborates with the Committee for Coordination of Services to Displaced Persons in Thailand (CCSDPT) to ensure coordinated efforts among partners in camps. Meanwhile, JRS in Kayah state, together with Loikaw Diocese partners and KnED/STEDF, addressed the education needs of returnees, IDPs and host community, with peace and reconciliation integral to the process. The project also supports in- and out-of-school youths towards making positive changes in their community.
Using the partnership and capacity building approach with KnED in delivering basic education service in the camps, JRS aims to promote education management transition upon return and focuses on the long-term sustainability of the program. At present, JRS is assisting KnED in planning and managing aspects of the program, e.g. vocational training, that has been fully handed over to the refugee education department as well as helping them develop new partnerships.
The project has the following specific end of program outcomes:
A. Improve basic education services to refugee children B. Promote opportunities for Out-Of-School Children and youth for alternative education C. Enhance refugees’ education management D. Promote peaceful and social cohesion between potential host communities and returning refugees E. Strengthen cooperation among stakeholders for basic education in Loikaw Diocese F. Synchronise coordination between JRS and stakeholders
3. EVALUATION TYPE, PURPOSE AND SPECIFIC OBJECTIVES (End program, External)
Caritas Australia is looking for an external consultant to conduct this end of program evaluation. The main purpose of the evaluation is for learning and ongoing improvement. The evaluation will focus on assessing the project as a whole, looking at formative aspects (the design and program logic), process (the implementation process), and performance (results brought about by it). It is not seeking to quantitatively validate project achievements against indicators, but rather qualitatively examine what worked well and not so well, why, and what should be done to improve future programming. It aims to assess the strengths and challenges, and provide lessons learnt to inform future similar programming.
Specifically, the evaluation objectives are:
- To assess the extent of impact and changes that the project has brought about in the lives of participants and their communities.
- To identify and assess key internal and external factors related to project development, implementation and management that contributed to, affected, or impeded the achievements of the project.
- Identify, understand and document strengths, challenges, and lessons learned and provide recommendations to improve project design, implementation and management that will assist Caritas Australia and Jesuit Refugee Service Asia Pacific to improve future programming.
4. KEY EVALUATION QUESTIONS
The evaluation will be guided by the following questions:
Relevance
- To what extent did the project respond to and meet the needs and priorities of those that the project sought to benefit?
- Were the project inputs and strategy (including its assumptions and logic) realistic, appropriate and adequate to achieve the results?
Effectiveness
- To what extent and in what ways did the project achieve its intended objectives?
- What internal and external factors influenced the achievement/non-achievement of project objectives?
Efficiency
- Was the project implemented in the most efficient way (timely, cost-effectively and to expected standards) compared to alternatives? What could be done differently to improve the cost efficiency of implementation while still maximising results?
- Did project activities overlap or duplicate other similar interventions (funded nationally and/or by other donors)?
Impact
- What significant changes, both positive and negative, intended and unintended, have occurred, and what has been the contribution of the project towards these?
- To what extent and how has inclusive development been addressed in the project delivery, in particular, gender equality and disability inclusion?
- What internal and external factors influenced (hindered or supported) the impact the project achieved?
Sustainability
- To what extent are the benefits of the project likely to be sustained after the project concludes?
- What internal and external factors influenced (hindered or supported) the sustainability of the project?
Safeguarding
- Have safeguarding (PSEAH and child safeguarding) standards been effectively integrated in to organisational systems and throughout project delivery?
Lessons learnt
- What lessons learned and best practices can be drawn from the project implementation?
- What changes could be recommended to improve the effectiveness, impact, and sustainability of future iterations of the project?
- How does the lessons learned inform the new design of the multi-year project?
5. EVALUATION SCOPE & TIME FRAME
The evaluation will cover the entire duration of the project from July 2018 to present, and look at annual activity plans, implementation, management, and results. A sample of project locations across the Mae Hong Son and Kayah State should be visited physically. If there are security risks to visit Kayah state physically, the evaluator can conduct virtual sessions to determine appropriate sample size and methodology based on information provided.
The evaluation is planned to be conducted in June – August 2022, with final report expected by the end of September 2022. The exact date shall be finalised in consultation with Caritas Australia and Jesuit Refugee Service Asia Pacific. It is estimated that a total of 28 effective days will be required to complete the evaluation.
Due to security and access issues, the evaluation will need to be conducted through a mix of face to face and remote data collection.
Timeframe for evaluation:
Activities and Effective Days
- Desk review of program documents (3 days)
- Developing evaluation plan, research instruments such as questionnaires, interview guidelines among others (3 days)
- Primary data collection (15 days)
- Preparation and presentation of preliminary findings (2 days)
- Written final report, including responding to Caritas Australia and JRS comments and revision (5 days)
The preliminary findings and recommendations will be presented in a de-briefing session which include representatives from Caritas Australia and Jesuit Refugee Service Asia Pacific. Feedback from this session will be collated and included in the final evaluation report.
6. AUDIENCE FOR THE EVALUATION
The primary audience of the evaluation will be:
- Caritas Australia
- Jesuit Refugee Service Asia Pacific, Thailand and Myanmar
Secondary audience will be:
- Downstream partner organisations
- Project participants
- Zetaman and KnED/STEDF
As appropriate evaluation reports &/or executive summaries will also be made available to:
- Australian Government Department of Foreign Affairs and Trade
- Local government
- Social Welfare Council (SWC)
7. METHODOLOGY AND EVALUATION PLAN
The external evaluator, in discussion with relevant Caritas Australia staff, will determine appropriate data sources and methods/tools of assessment. Generally evaluations will comprise the following discrete research and analysis components:
- Desk assessment – provide relevant baseline, design and implementation documentation to the external evaluator for familiarisation.
- Field data collection – the external evaluator will visit project sites and partner offices to collect data and analyse evidence. A variety of methodologies will be employed to collect quantitative and qualitative data, including interviews and focus group discussions with project participants, staff and volunteers.
- Findings analysis & testing: Review meetings regarding initial findings with primary stakeholders (communities, partner staff, and Caritas Australia staff) to check interpretation.
Caritas Australia is particularly focused on promoting accountability to our program participants, and enabling meaningful participation throughout the program cycle, including the evaluation process. Accordingly, evaluation methodologies and planning should include:
- Adequate time with a diversity of project participants and community representatives,
- Clear documentation of the perspectives of the individuals and communities in which the program is active (ideally including first person quotes or case studies or video clips-using consent forms of Caritas Australia)
- Opportunity to provide feedback regarding initial findings to check interpretation
- Key program informants and important background documents will be discussed following appointment of an evaluator.
8. EXPECTED DELIVERABLES
Develop a detailed evaluation plan (to be submitted after the document review but before the data collection – to be endorsed by Caritas Australia and JRS), outlining the proposed methodology
Meeting to present the draft findings of the evaluation, and to give feedback to Caritas Australia and JRS staff
Draft evaluation report in English to be presented to Caritas Australia and JRS for comment, and then submission of the final evaluation report.
The final report will usually include:
- An executive summary and overall recommendations
- Key findings and an analysis of these findings
- Successes/challenges and the reasons for these
- Lessons learned
- Specific recommendations on any aspect of the program, its planning, implementation, management, coordination and other related factors.
- Where the evaluation covers multiple program partners: Specific and exclusive recommendations for each partner and program (i.e. separate reports or clearly distinct sections)
- Appendices of statistical data, participant interviews, case studies, photos, Videos etc.
- Lists of key informants including partners
The length of the report should be no longer than 40 pages, excluding appendices.
Importantly the executive summary needs to be of a standard to share across the agency and potentially externally. Note that the executive summary will also be translated for dissemination to project partners and evaluation participants.
9. EVALUATOR REQUIRED KNOWLEDGE AND EXPERIENCE
- The evaluation will primarily be conducted by an external evaluator, contracted by Caritas Australia.
- The external evaluator will be selected based on the following skills, experience, and qualifications:
- Extensive knowledge of Monitoring, Evaluation and Learning (MEL) theory and practice, ideally in the resettlement context
- Demonstrated experience evaluating international development programs using quantitative and qualitative methods and participatory methodologies.
- Experience conducting evaluations using remote methodologies.
- An understanding of the historical, political, social, and cultural context of Myanmar and people fleeing Myanmar to seek refuge in Thailand; and understanding of the broader Myanmar-Thai border relations
- The ability to respectfully and effectively communicate with diverse groups of people
- Strong interpersonal and facilitation skills, and cultural and gender, Safeguarding/Social Protection/Child Protection sensitivity in working with local communities and a range of stakeholders
- Qualification in a relevant field(s) and past experiences on project evaluation and study in the sectors of development and social sector (with experience and leading-edge expertise in the sector of education is highly desired)
- Excellent oral communication skills and the ability to deliver high quality written reports in English
10. LOGISTICAL SUPPORT
The following support will be provided to the evaluator:
- Project and partner documentation (including project designs, progress reports, details of project locations and downstream local partners, monitoring trip reports) will be provided to the external evaluator by Jesuit Refugee Service Asia Pacific.
- Details on field visits i.e. transport, accommodation, meals and other logistical considerations to be coordinated by Jesuit Refugee Service Asia Pacific in consultation with the external evaluator and Caritas Australia.
- Interviews and meetings (physical and virtual) with project participants, staff, and volunteers will be coordinated by Jesuit Refugee Service Asia Pacific in consultation with the external evaluator and Caritas Australia.
11. KEY TERMS AND CONDITIONS
- Payment will be on submission of tax Invoice on delivery against milestones.
- Travel, meal, & accommodation costs and other expenses related to the assignment except where otherwise stated in the Terms of Reference above will be paid for directly by JRS, and so should not be included in the budget proposal. In circumstances where the contractor is required to purchase these, they will be reimbursed upon provision of receipt.
- All equipment and materials required for the assignment are to be provided by the contractor except where otherwise indicated in the Terms of Reference above
- Caritas Australia may request written evidence of necessary insurance (including workers’ compensation), superannuation, and taxation prior to commencement of the assignment.
- In alignment with our Child Protection Policy, prior to engagement successful applicants will be required to undergo a local police check and to sign on to Caritas Australia’s Child Protection Code of Conduct and PSEAH/Safeguarding policy.
12. APPLICATION PROCEDURE
Interested parties are requested to submit:
- A cover letter of no more than 2 pages introducing the evaluator/organisation and how the skills and competencies described above are met, with concrete examples. Please also use this cover letter to indicate the consultants’ availability for the proposed period.
- Technical Proposal: The technical proposal should include an interpretation of the objectives of the consultancy, methodology and work plan, organisation/individual capacity statement, past experience and activities related to evaluation, CVs of the consulting team and their roles in the achievement of the assignment; and details of three professional referees.
- Financial Proposal: A one-page budget of the offer, covering all major anticipated costs.
- Examples of previous evaluation reports for evaluations conducted in livelihoods or in other sectors
Please send applications or direct any queries to Anshu Adhikari, Caritas Australia Program Coordinator, at: [email protected].
Deadline of application: 20 June 2022. Early application is highly encouraged.
ANNEX: SUGGESTED FORMAT FOR THE EVALUATION REPORT
Title Page should include the title of the program, the date of report and the author/s name. The following is a list of suggested sections:
- Table of Contents
- Acknowledgements
- Maps
- List of Acronyms
- Executive Summary should include briefly describe:
- Program description and context
- Methodology and evaluation team
- Major findings
- Key recommendations
- Program Description should include: context, underlying rationale, stakeholders and participants, conceptual model, results chain or logical framework and program monitoring system.
- Purpose of the Evaluation
- Audience for and use of the evaluation
- Evaluation methodology
- Data sources
- Strengths and major limitations of the methodology
- Evaluation team
- Program description
- Project design and implementation approach
- Partner Team
- Sectoral literature review
- Evaluation findings
- Recommendations based on evidence and insights and including specific recommendations relevant to each partner and project/program:
- Lessons Learned
- Annexes (to the evaluation report):
- Terms of Reference for the evaluation
- Evaluation Plan and timetable
- List of individuals interviewed and of stakeholder groups and/or communities consulted
- List of supporting documentation reviewed
- Research instruments: questionnaire, interview guide(s), etc. as appropriate
- Project/program logical framework (if applicable)
- Specific monitoring data, as appropriate
- Summary tables of progress towards outputs, outcomes, goals (referring directly to any established indicators)
How to apply
Please send applications or direct any queries to Anshu Adhikari, Caritas Australia Program Coordinator, at: [email protected].
Deadline of application: 20 June 2022. Early application is highly encouraged.