Impact Evaluation on O&M for Water System Sustainability in Turkana County

About Oxfam Kenya

One person in three in the world lives in poverty. Oxfam is determined to change that world by mobilizing the power of people against poverty. Around the world, Oxfam works to find practical, innovative ways for people to lift themselves out of poverty and thrive. We save lives and help rebuild livelihoods when crisis strikes. And we campaign so that the voices of the poor influence the local and global decisions that affect them. In all we do, Oxfam works with partner organizations and alongside vulnerable women and men to end the injustices that cause poverty

Read more about Oxfam from: https://kenya.oxfam.org/

1. PURPOSE AND BACKGROUND

The provision of water in Kenya is mandated to county Governments by the constitutional reform of 2010. Funding sources for both the infrastructure and ongoing maintenance of water systems can vary in different urban and rural settings. While county governments currently play a more prominent role in urban centres, donors and NGOs tend to take the lead in harder-to-reach or less commercially viable rural settings. After the initial Capex investment, the operation and maintenance of the water projects is handed over to community volunteers who do not have the technical expertise or financing management skills to manage the water projects.

Within a couple of years, the projects end up run down owing to mismanagement, socio-political and environmental injustices including drought. Currently, Kenya is experiencing one of the worst droughts experienced in 40years. The Government has reported that more than 40%of the water projects are currently dysfunctional and has been seeking aid from donors and social investors to restore functionality or conduct water trucking to provide immediate water access to communities.

Oxfam in Kenya has been piloting a project to test the viability and performance of public-private partnership to provide sustainable access to safe and adequate water to rural communities through continuous operation and maintenance (O&M) on 60 existing rural water schemes in Turkana County.

In this model,60 water projects have been bundled up and then a professional private operator was contracted to undertake professionalized O&M services for a period of one year. Key performance indicators were set for accountability to the community by ensuring minimal downtimes through professionalized operation and maintenance services.

Cost effectiveness is driven by improving tariff collection through the introduction of cashless payment mechanisms (M-Pesa) and by use of AQtaps which solar powered machines that enable smart water dispensing and collection of tariffs. Furthermore, Oxfam has been working with the Turkana County Government to set up financing mechanisms for water supply (through ringfencing resources allocated for O&M works).

Through project monitoring and evaluation, including this evaluation, Oxfam aims to show proof of concept to advocate for the business model with public and private sector investors – with the aim of achieving (and scaling up) sustainable water provision in the Arid and Semi-Arid Lands (ASALs) of Kenya.

Oxfam is recruiting a national consultant to co-lead in the impact evaluation of this pilot project on O&M for water system sustainability in Turkana County. The consultant will work alongside the Impact Evaluation Working Group (with members from Oxfam in Kenya and Oxfam GB) in carrying out the assignment. The role of the consultant is to work collaboratively to finalize data collection tools and sampling strategies, lead all aspects of data collection, and to work collaboratively with Impact evaluation team on data analysis.

2. OBJECTIVES AND LEARNING QUESTIONS

2.1: OBJECTIVES OF THE EVALUATION

The impact evaluation is intended:

• • To complement and triangulate findings and recommendations from the project’s final evaluation, which is being carried out internally. – To show proof of concept to advocate for the business model with public and private sector investors – To assess functionality, reliability and sustainability of targeted water schemes.

2.2: LEARNING QUESTIONS

The evaluation and learning questions to be addressed by the impact evaluation, in combination with the final evaluation are as follows:

1. 1. To what extent has the project delivered against its objective and expected results?
   2. What have been the project’s successes and shortcomings (both at project-level theory of change and implementation levels)?
   3. To what extent have the actions of the project led to:
      1. Reduced water insecurity?
      2. Reduced time spent gathering water, especially for women?
      3. A sense of community self-sufficiency (related to water)?
      4. Strengthened resilience against drought of community members?
      5. Changes in local conflict?
   4. What factors facilitated or impeded the achievement of the project objective?
   5. To what extent have benefits continued or are expected to continue?
   6. How could the model be replicated, scaled up and used to influence practice and policy development?
   7. What are the key lessons and recommendations that can inform design and implementation of future interventions?

Question 1 will be largely answered by the final evaluation (with some triangulation through the impact evaluation), while Questions 2 and 3 will be answered by the impact evaluation. Questions 4 through 7 will be addressed by both evaluations.

3. APPROACH

This impact evaluation is part of Oxfam GB’s second cycle of a new modality for design and conducting such evaluations that involves a decolonized approach towards increased relevance and use of evaluation findings and recommendations. The impact evaluation should prioritize the learning and evidence needs of Oxfam in Kenya and other key project stakeholders. The use of evaluation findings should be considered throughout the evaluation process, planning for stakeholder engagement from the beginning and collectively deciding which information and evaluation products will be most useful.

The Impact Evaluation Working Group has defined an initial evaluation design appropriate for answering the evaluation and learning questions. The evaluation will be ex post and theory-led, with participatory design and sensemaking processes. The approach will be mixed methods, in part following a quasi-experimental design based on existing (non-randomly assigned) treatment and control groups. The design combines the perspectives of community members and other stakeholders with information from the water schemes. Data sources will include interviews with community members (both qualitative and quantitative information), key informant interviews (qualitative information), technical and financial data (quantitative), and project documentation/literature. Analysis approaches will include qualitative analysis and synthesis, quantitative analysis including propensity score matching, configurational analysis using EvalC3 and participatory sensemaking with key stakeholders.

Data collection tools and instruments: The identified consultant will develop relevant data collection tools and identify appropriate means of collecting the data which will be discussed and agreed upon prior to the data collection exercise

Ethical Considerations: The consultant will put in place measures to ensure data collection adheres to all necessary ethics and guidelines provided by local laws on data protection and ‘Do no Harm’ standards, Oxfam safe programming and feminist principles.

4. CONSULTANT PROFILE

The lead consultant can be an individual or a team, however, for data collection a team of researchers will be requires. Key competencies of the consultant include:

• • A minimum of 5years experience in carrying out similar project and/or program evaluations – Experience managing a team of researchers to carry out high-quality, independent data collection including through individual and/or household surveys/interviews and key informant interviews – Academic qualification for the lead consultant: in a relevant field such as a WASH related discipline and/or social sciences. – Have an established team or network of known researchers who would be available to carry out the necessary data collection activities in Turkana County – Expertise in collecting water scheme technical and financial data – Experience conducting similar studies in the ASALs of Kenya – Ability to facilitate and relate to stakeholders at multiple levels and in diverse contexts – Ability to co-facilitate workshops on evaluation design and sensemaking – Familiarity with the proposed approaches for this evaluation, including participatory and decolonised evaluation processes – Strong written and verbal communication skills in Swahili and English – Sensitivity to cultural and historical contexts in the data collection and analysis process – Experience with relevant tools for digital data collection (e.g., Survey CTO) – Flexibility to accommodate any unexpected delays in evaluation activities (e.g., security issues, travel restrictions due to COVID-19) – Aware of prevailing community security and related tensions.