Mid-Term and Final Evaluation of Bright Sight: NTDs prevention and Eye Health Care for persons with disabilities and vulnerable groups

Christian Blind Mission (CBM) is an international Christian development organisation whose primary mission is to improve the quality of life of people living with disabilities in the poorest countries around the world. CBM strives to fight poverty as a cause and consequence of disability.

CBM Vision Statement- An inclusive world in which all persons with disabilities enjoy their human rights and achieve their full potential.

CBM Mission Statement- CBM is an international Christian development organization, committed to improving the quality of life of persons with disabilities in the poorest countries of the world.

Based on its Christian values and over 100 years of professional expertise, CBM addresses poverty as a cause, and a consequence, of disability, and works in partnership to create a society for all.

About the project

Project objectives

The project aims at improving access to Neglected Tropical Diseases (NTDs) prevention services and Eye Care services in a 3-year time span in Central Equatoria, Eastern Equatoria and Lakes States, South Sudan, in line with targets 3.3 and 3.8 of Sustainable Development Goal no. 3.

The proposal is aimed particularly at the most vulnerable categories of the population, such as people with disabilities, women, and children, and it has been agreed upon directly with the States’ Ministries of Health, partners of the project, thanks to the long-standing presence and expertise of CBM Italy, CUAMM and CORDAID in the country. These preconditions guarantee the full endorsement of the proposal by government authorities, its alignment with the country’s thematic and geographical priorities, and its response to the needs of the area of intervention.

During the 3 years of implementation foreseen by the project proposal, the leading organization CBM Italy, in collaboration with CUAMM and CORDAID and the State Ministries of Health, partners of the project, intend to strengthen the South Sudanese health system, working on 3 strategic pillars to fight NTDs and promote eye health:

• Health education, strengthening inclusive NTDs prevention practices in local communities, training, raising awareness and promoting COVID-19 pandemic prevention and containment interventions, under the conviction that health promotion is key to monitoring and containment.
• Strengthening community health services with continued decentralization and dissemination of inclusive and accessible Eye Care and prevention services. The diffusion of proximity health services is in continuity with the strengthening of the South Sudanese health system initiated by CBM Italy in previous years and continued with CUAMM and CORDAID in the project co-funded by AICS (AID11448), improving the quality of the already existing services and extending them to new areas of the country.
• The treatment of symptoms and complex pathologies and the rehabilitation of patients with disabilities, continuing the improvement of health excellence in the field of ophthalmology carried out by the South Sudanese Ministry of Health, CBM and the Italian Cooperation.

Each partner will have a precise role related to their specific expertise, knowledge of the needs in the reference territory and the collaborations already in being with the local Ministries of Health: basic health services in Eye Care and NTDs will be strengthened in the two government hospitals of Rumbek (Lakes State) and Torit (Eastern Equatoria State), where respectively CUAMM and CORDAID support the State Ministries of Health in the delivery of health services. Eye care services will be integrated into the prevention, screening, diagnosis and treatment programmes of the two hospitals, thanks to the expertise and coordination with the programmes in each partner’s area (maternal and child services for CUAMM and NTDs for CORDAID).

CBM and the Ministry of Health will widen the range of specialist eye care services available at the Buluk Eye Centre (BEC) in Juba, the only advanced secondary level centre in the entire country, to incorporate the provision of pediatric eye care services, carried out by adequately trained specialist staff. BEC will become the first pediatric eye centre in the country equipped and capable of providing specialized pediatric care (congenital cataracts, glaucoma, specialized operations requiring anaesthesia). At the same time, BEC will be strengthened to provide rehabilitation services for persons with disabilities and support for inclusion in social and community life.

By upgrading the 3 identified health facilities in Juba, Rumbek and Torit and training a total of 326 medical, health and educational staff in the target areas, eye care services will be provided to 88,440 patients over the 3-year period of project implementation.

The overall objective of the project is to ensure health and well-being for all and for all ages in South Sudan, with particular attention to treatment and prevention in Eye Health and Neglected Tropical Diseases (NTDs), in line with targets 3.3.5 and 3.8.1 SDG 3.

The specific objectives of the project are: Improve access to tropical neglected disease prevention (NTDs) and vision care services in Central Equatoria, Eastern Equatoria and Lakes, South Sudan, with a focus on people with disabilities and the most vulnerable, over 3 years.

As the project has now reached the half way point, a mid-term evaluation is being commissioned to assess the project progress toward achieving its objectives. The study will identify lessons learnt and provide concrete recommendations in line with the given structure of the project and the donor procedures for the refinement of the project approach, if necessary, and to inform subsequent implementation phases of the project.

Objectives and Evaluation Questions

The objective of the mid-term evaluation is to understand progress of the actions, processes and methodologies used in the project so far towards achieving project objectives and outcomes and draw out lessons for how the Bright Sight project can be improved during the rest of its implementation for more positive impacts.

Specifically, the mid-term evaluation will be assessing the following:

Relevance: To what extent did the project addresses issues of accessand quality of Eye Health Services delivery in Central Equatoria, Eastern Equatoria and Lakes States with particular attention to children and vulnerable people

• Efficiency: Were the financial resources and other inputs used efficiently to achieve outputs? Are there opportunities to improve efficiency?
• Effectiveness: To what extent is the project on track to achieve its objective and outcomes? What progress has been made so far? What factors may be limiting the achievement of intended results?
• Sustainability: To what extent are the project’s positive actions likely to continue after the end of the project? In particular, what institutional changes are likely to be sustained beyond the project lifetime? What actions need to be taken to increase the likelihood of the project results being sustainable? To what extent will the target communities continue to benefit from the results achieved through project interventions? The extent of continued institutional support by the government. Could the project survive and continue providing service by its own income & government support when the CBMIT-AICS funding ends? What are the best options to overcome such an eventuality?
• Key areas to cover:

1.Quality of project design:

1. As presently designed, is the intervention logic holding true?
2. Is the current design sufficiently supported by all stakeholders?
3. Is coordination, management and financing arrangements clearly defined and do they support institutional strengthening and local ownership?
4. Is the sustainability strategy (handing over strategy to partners) fully understood by the partners?
5. Is the timescale and/or range of activities realistic with regard to the partners’ capacities?
6. If applicable: How well has the project design been adapted to make it more relevant? Was it straightforward to do contractually?

2. Efficiency to date:

1. How well is the implementation of activities managed?
2. Is the log frame or similar tool used as a management tool? If not, why not? is an activity schedule (or work plan) and resource schedule available and used by the project management and other relevant parties?
3. Is the coordination between partners and CBM field office and the CBM Headquarters supportive of implementation?
4. To what extent are activities implemented as scheduled? If there are delays how can they be rectified?
5. Are funds committed and spent in line with the implementation timescale? If not, why not?
6. How well are activities monitored by the project and are corrective measures taken if required?
7. If appropriate, how flexible is the project in adapting to changing needs?
8. If appropriate how does the project co-ordinate with other similar interventions to encourage synergy and avoid overlaps?

How well are outputs achieved?

1. Have all planned outputs been delivered to date? And in a logic sequence? What is the quality of outputs to date?
2. Are targets being reached as expected in the Logical Framework, with a focus on Cataract Surgical Rate, outcomes on surgeries, prevalence of diabetic retinopathy.
3. Are the outputs achieved likely to contribute to the intended results?
4. How well is the Partner Contribution / Involvement working?
5. Do the inter-institutional structures e.g. steering committees, monitoring systems, allow efficient project implementation?

3. Effectiveness to date:

1. How well is the project achieving its planned results?
2. What is the quality of the results/services available? Have all planned target groups access to / using project results available so far?
3. Are there any factors which prevent target groups accessing the results/services?
4. Is Buluk Eye Center performing its role as planned for supporting other partners in terms of capacity building, trainings, and service delivery?
5. To what extent has the project adapted or is able to adapt to changing external conditions (risks and assumptions) in order to ensure benefits for the target groups?

4. Impact Prospects:

1. What are the direct impact prospects of the project at Overall Objectives level?
2. What, if any impacts are already apparent?
3. What impacts appear likely?
4. Are the current OVIs/targets realistic and are they likely to be met? Are any external factors likely to jeopardize the project’s direct impact?
5. Have there been/ will there be any unplanned positive impacts on the planned target groups or other non-targeted communities arising from the project? How did this affect the impact?

5. Potential sustainability:

1. Financial / economic viability?
   1. If the services/results have to be supported institutionally, are funds likely to be made available? If so, by whom?
   2. Is there a financial/ economic phase-out strategy? If so, how likely is it to be implemented?
   3. What is the level of ownership of the project by target groups and partners will it continue after the end of external support?
   4. How far the project is embedded in local structures?
   5. What is the level of policy support provided and the degree of interaction between project and policy level?
   6. Is any public and private sector policy support likely to continue after the project has finished?
   7. How well is the project contributing to institutional and management capacity? How much in % (percentage) is the contribution of own income plus the government to cover the running cost of the project currently? Will this change by the end of the project life to enable it running?
2. How far is the project embedded in institutional structures that are likely to survive beyond the life of the project?
3. Are project partners being properly strengthened (technically, financially, and managerially) for continuing to deliver the project’s benefits/services?

6. Cross-cutting issues:

1. Have practical and strategic gender interests been adequately considered in the project strategy? Please consider the following aspects of gender mainstreaming:
   1. Has the project been planned on the basis of a gender-differentiated beneficiaries’ analysis?
   2. To what extent will / could the gender sensitive approach lead to an improved impact of the project?
   3. What is the likeliness of increased gender equality beyond project end?
2. Is the project respecting environmental needs? Please consider the following aspects of mainstreaming environmental aspects:
   1. Have environmental constraints and opportunities been considered adequately in the project design?
   2. Are good environmental practices followed in project implementation (in relation to disposal and managing of medical wastes, etc.)?
   3. What capacities exist (within project, project partners and project context) to deal with critical risks that could affect project effectiveness such as climate risks or risks of natural disasters (in the case of projects in sensitive geographical areas / natural disasters hotspots)?
   4. Has environmental damage been caused or likely to be caused by the project? What kind of environmental impact mitigation measures has been taken?
3. Has (good) governance been mainstreamed in the project/programme (P/P)? Please consider the following aspects of governance:
   1. Is the P/P designed in such a way that it takes into account potential conflict?
   2. Is regular, transparent, financial reporting built into the P/P? Are its results widely circulated and understandable?
   3. Are there effective anti-corruption monitoring tools in place?
4. As this project is an eye health project it is very important to consider the evaluation of the major strategic priorities of IEHI.
   1. How much is the project involving/supporting in the strengthening of the national/state health systems?
   2. Does it promote comprehensive eye health service like promotion, prevention and rehabilitation in addition to the curative aspect? What is it doing in this regard?
   3. Promoting inclusion in eye health-How far are the projects physically accessible to people with disabilities?
   4. Do they work with DPOs around them to receive and refer needy patients to them?
   5. Is there a plan to develop low vision service to promote inclusion at least in BEC?
5. Focus on quality of service-Cataract surgical outcome monitoring is now mandatory in CBM. Do the all the partners implement CSOM regularly with their cataract surgical service? If not, what is the challenge?
6. To what extent is the eye care services to the beneficiaries inclusive?
   1. Are the services accessible to person with disability? Are the services based on the RECU principle meaning: Reach, Enter, Circulate in and use the facilities in the built environment?
   2. Is data collected on people with disability? Is it disaggregated?
   3. Is the Washington Group set of Questions being used in gathering data?
   4. Are the eye care services affordable to people with disability?
7. Child and adult safeguarding
   1. Is there a feedback mechanism for beneficiaries to provide their concerns appropriately? Do they get timely response?
   2. Have the staff been trained in safeguarding? Do new staff receive inductions on safeguarding?
   3. Is Safeguarding responsibility allocated to a staff or is there a safeguarding focal person?
   4. Is there a code of conduct? and do staff sign the code of conduct?

Approach and methodology

The project implements a routine monitoring system based on a Log Frame developed at the beginning of the project and corresponding data collection plan to collect data against key outcome indicators. The evaluation methodology is expected to review this data and, as far as possible, allow comparability considering any issues around data collection for the first half of the project.

The consultant is to visit all the three partners both at Country Office and field (Implementation office) levels and expected to employ a variety of data collection and analysis techniques for both quantitative and qualitative data to ensure a comprehensive evaluation exercise. This will likely include, at a minimum:

• Document and systems review: Review of existing documentations, including project reports, project log frame and monitoring and evaluation data.
• Surveys: Application of structured survey questionnaires with a representative, random sample of target population to quantitatively assess outcomes. This will be in greater in scope, breadth and depth compared to standard routine project monitoring.
• Focus Group Discussions: With target groups and other stakeholders to assess implementation experiences and effectiveness, document successes, challenges and lessons learned, and develop recommendations for improvement.
• Key Informant Interviews: Consultations with key project stakeholders, including staff and partners. Guidance on appropriate stakeholders will be provided by Country Office staff.

The consultant will be expected to:

• Review relevant project documents, including but not limited to project reports, project log frame, household survey reports and data and baseline surveys.
• Collect and review relevant government policy/guideline document, relevant secondary data etc.
• Develop, test, and apply survey questionnaires. A representative random sample of the target population should be interviewed to assess outcomes and establish impact of the project interventions in line with log frame indicators. The project is putting a strong emphasis on the need to collect information’s on outcome indicators
• Design and conduct focus group discussions with relevant community members, both male and female, to assess implementation experiences and effectiveness, document successes, challenges and lessons learned, and develop recommendations for improvement.
• Carry out key informant interviews i.e. consultations with key project stakeholders, including field staff and partners. Guidance on appropriate stakeholders will be provided by project and Country Office staff.
• Collate and analyze data.
• Present findings to CBM key stakeholders and staff and invite comments at a one-day workshop in Juba town.
• Draft report and submit to CBM for comments
• Incorporate comments and produce final report

Expected Deliverables and Timeline

All written documentation is to be submitted in English using Microsoft Word in both soft and hard copy. The main body of all reports should be written in simple, non-technical language (i.e. plain English), with any technical material to be presented in annexes. All primary data collected, and analysis conducted for the purpose of the evaluation will remain the property of CBM and must be submitted electronically and in a clear and comprehensible format in Excel.

The midterm evaluation should begin no later than between the 8th of July 2024, with the evaluator(s) expected to take a total of 45 days from the day of contracting to complete the assignment.

The final evaluation, covering the requests described in the present TOR and with a focus on the implementation of the recommendations listed in the midterm evaluation and to the adaptation capacity of the implementation strategies, in order to incorporate the lessons learned, will be happening between the 30th and the 32nd month of implementation, with flexibility according to the progress of the activities.

The consultant will provide the following deliverables to CBM within the timeframe stated:

1. Inception Report: within 5 working days of evaluation launch, a detailed report on the evaluator’s proposed approach to the evaluation will be submitted for approval. This will provide preliminary findings/understandings based on document review, rationale and a detailed description of the methodology and tools, research questions, analytical methods, budget with a breakdown of costs and detailed work plan for the entire exercise. Any draft questionnaires or interview forms will also be submitted for review at this stage.
2. Data collection: testing questionnaires, refining data collection tools and administer data collection within 7 working days of the evaluation launch.
3. Data analysis and reporting: Preliminary Report and Presentation: within 25 working days of evaluation launch, the consultant will present the preliminary findings for discussion at a stakeholders’ workshop. This should include a draft set of recommendations and lessons learnt. At the end of the workshop, a report incorporating comments by stakeholders and, where necessary, responses will be submitted to the Country Director of CBM South Sudan.
4. Final Report: within 40 days of evaluation launch, a detailed report of the overall findings of the mid- term evaluation will be submitted to the Country Liaison and CBMIT for approval. This report should incorporate specific, practical and feasible recommendations for improving project delivery and impact based on learning from project design, implementation, and delivery. The main body of the report will include an Executive Summary of no more than 3 pages, outline and rationale for methodology, the main findings, lessons learnt, conclusions and recommendations. Any data collection tools used should be included as Annexes.
5. A Power point presentation outlining key findings and implications, and recommendation for future implementation to be presented at a Stakeholders’ Workshop or steering committee meeting.

Management and Implementation Responsibilities

The consultant will report directly to the Programme Development and Delivery Specialist in CBM South Sudan and the project Focal Person based in Italy. However, s/he will also be expected to work closely with the field staff. Any proposed changes to the personnel listed in the application must be explained in the inception report and approved by CBM.

CBM will provide:

• Relevant project documents
• Guidance and technical support as required throughout the evaluation;
• Logistical arrangements for all field travel;
• Copies of all key background resources identified
• Introductory meetings with key stakeholders
• Comments and feedback on, and approval of, all deliverables within agreed timeline.

The consultant will be responsible for:

• Review documents and submit inception report
• Developing the detailed evaluation methodology
• Recruitment and payment of data collectors
• Conducting all data collection
• Analysis of data and reporting in a clear and accessible format preferably paperless data collection data collection system.
• Regular progress reporting to the evaluation manager, including responding to any comments or technical inputs wherever reasonable.
• Production of deliverables within agreed timeline and in accordance with quality requirements of evaluation manager
• Seeking comments and feedback from CBM regularly, through the evaluation manager, in sufficient time to discuss and incorporate these into the final report.
• Production of two reports:
  • the midterm evaluation report containing data against all indicators in the project log frame, evidence-based responses to the key evaluation questions, summary of lessons learnt and recommendations for future implementation.;
  • The final evaluation report, containing all information and data related to the indicators present in the logical framework of the project, the responses based on concrete data to the key questions of the evaluation with a focus on the recommendations and lessons learned present in the midterm evaluation.

Qualification and Desirable Competencies:

Applications from individuals or teams are welcome and will be assessed on their ability to demonstrate the following qualifications and competencies:

Essential

• A minimum of 7 years’ experience in carrying out impact evaluations, demonstrable academic and practical experience in qualitative and quantitative research methodology, evaluation design and implementation with a focus or previous experience on eye health in South Sudan.
• Master’s degree in sectors with relevance to the evaluation (Social Sciences, Public Health, International Cooperation, and development or similar).
• Strong analytical, facilitation and communication skills.
• Experience of the health-related research and development and participatory research at the community level including the undertaking health programme evaluation with institutional donors such as IADC, EU, UN agencies.
• Excellent reporting and presentation skills.
• Good knowledge of the Disability Inclusive Development framework and rights of persons with disabilities.
• Fluency in spoken and written English and Arabic is essential.

Desirable

Previous knowledge of conducting evaluation for health programmes in South Sudan.

Interested evaluators or firms are requested to submit:

1. An Expression of Interest detailing their interpretation of the TOR, proposed methodology including sampling framework, work schedule and proposed budget for 22,000 USD
2. A capability statement demonstrating how they meet the required qualifications and competencies.
3. Copies of all relevant Curriculum Vitae (CVs). Only CVs for the specific individuals that will form the proposed evaluation team should be included.
4. A sample of an evaluation report for a similar project completed within the last 24 months (this will be treated as confidential and only used for the purposes of quality assurance).
5. Two references (including one from your last client/employer).