DESIGN, CREATION, DEPLOYMENT, AND IMPLEMENTATION OF A MICROSOFT AZURE DATA LAKE ARCHITECTURE

TERMS OF REFERENCE

PROJECT TITLE: DESIGN, CREATION, DEPLOYMENT, AND IMPLEMENTATION OF A MICROSOFT AZURE DATA LAKE ARCHITECTURE

1. INTRODUCTION

World Relief is a non-profit, Non-Governmental Organization whose mission is to empower the church to serve the most vulnerable. World Relief is hiring a consultant to support the Design, Creation, Deployment, and Implementation of a Microsoft Azure Data Lake Architecture for our International Programs division. World Relief’s International work includes a mixture of publicly and privately funded projects with a programmatic focus in the health sector, humanitarian response, and an integrated church-based model that focuses on church capacity building, economic development, agriculture, child development and protection, health, and other sectoral areas.

2. PURPOSE AND DESCRIPTION OF THE CONSULTANCY

World Relief has identified the need for a robust and scalable data lake to support data-driven decision-making and analytics in view of standardizing its systems. The objective of this project is to design, create, deploy, and implement a data lake using Microsoft Azure capable of securely storing, managing, archiving and analyzing diverse data types to meet the organization’s business needs. This data lake will serve as a centralized repository for diverse data sources and facilitate advanced data analytics for World Relief and its clients. The primary objectives of this consultancy work are as follows: Design a scalable and secure data lake architecture using Microsoft Azure services that can effectively store, manage, archive, provide data analytics capabilities, reporting and insights for the organization’s data.

1. Create, configure and deploy the data lake infrastructure in compliance with best practices and industry standards.
2. Deploy data integration, data ingestion, storage, and processing solutions within the data lake environment.
3. Implement data governance, security, and access control measures.
4. Provide documentation and knowledge transfer to World Relief staff for ongoing management and maintenance.

This consultancy will require working closely with the Impact Systems and MEAL teams within International Programs.

3. SCOPE OF WORK

3.1 Design Phase

The scope of work for this project includes, but is not limited to:

• Conduct a thorough assessment of the organization’s data sources, formats, and requirements.
• Collaborate with World Relief stakeholders mainly the MEAL team and leadership to understand data needs, sources, and objectives
• Define the data lake architecture, including storage, security, and access controls.
• Develop a data governance framework to ensure data quality and compliance.
• Create a detailed design document outlining the technical specifications and recommendations.

3.2 Creation and Deployment Phase

• Set up the data lake infrastructure using Microsoft Azure services, such as Azure Data Lake Storage, Azure Databricks, and Azure Data Factory.
• Deploy the necessary Azure resources including storage, compute, and networking components
• Configure and implement access controls, encryption, and auditing mechanisms to ensure data security and compliance
• Implement data ingestion pipelines to collect and store data from various sources and load both structured and unstructured data.
• Ensure data lake scalability and performance optimization.
• Develop data transformation and processing pipelines to clean, enrich, and prepare data for analysis

3.3 Implementation Phase

• Develop data processing workflows for data transformation, cleaning, and enrichment.
• Enable data cataloging and metadata management within the data lake.
• Integrate data analytics and reporting tools for data analysis and visualization.
• Provide training and documentation to ensure knowledge transfer to the organization’s team.
• Provide comprehensive documentation on the data lake architecture, data flow, and operational procedures
• Provide ongoing Technical Assistance to the Impact Systems and MEAL teams for a period of three months after the project handover

4. DELIVERABLES

The following deliverables are expected at the completion of the project:

1. Data Lake Architecture Design Document.
2. Azure Infrastructure Deployment and Configuration.
3. Data Ingestion and Transformation Pipelines for data sources such as Ona, SurveyCTO, SharePoint, other REST APIs and other sources.
4. Data processing workflows
5. Data catalog and metadata management setup.
6. Integration with analytics and reporting tools.
7. Regular progress reports and documentation of configurations
8. Security and Governance Documentation.
9. Training materials, documentation and knowledge transfer sessions

5. GENERAL REQUIREMENTS

The consultant/Research Firm must be able to demonstrate experience with producing similar deliverables and activities and must provide at least two references for this work, to be contacted by World Relief technical staff. Qualifications for the consultant/firm include:

• Master’s Degree or higher in a relevant field (e.g., Software Engineering, Computer Science, Machine Learning, AI) for the Lead Consultant at the firm. This person shall be the main technical point-of-contact for WR.
• Verifiable experience with delivering similar projects.
• Knowledge and experience using machine learning, AI, data pipelines analytic systems and business intelligence
• Knowledge and experience Microsoft Azure, Amazon Web Services, Google Cloud Storage.
• Good in collaboration and communication with internal team and clients to deliver similar software features and products

6. PROJECT TIMELINE

The project is expected to be completed within 3 months, starting from February 2024. The timeline may be adjusted in consultation with the consultant team to accommodate any changes or unforeseen circumstances.

7. RESPONSE

Please provide a response to this TOR to [email protected] by Friday January 19th 2024 and include the following elements:

1. Technical Proposal with a brief description indicating your understanding of the intended process to comply with the deliverables included in the TOR
2. Cost proposal outlining budget that accompanies technical proposal.
3. Schedule of assignment and time frame to produce the deliverables
4. CV of the lead consultant(s)
5. Two references of previous similar work done

*All of the above must be submitted for your application to be considered complete. If an application is missing any of the elements, it will not be reviewed.

8. REPORTING AND COMMUNICATION

Regular project updates and progress reports will be provided to the Data & Impact Analytics Manager, International Programs. The project manager or designated representative will serve as the main point of contact for communication.

9. RESPONSIBILITIES

9.1 World Relief’s Responsibilities

• Provide access to necessary data sources and systems.
• Collaborate with the consultancy team for data requirements and business rules.
• Appoint a designated contact person for project coordination.

9.2 Consultant’s Responsibilities

• Execute all project phases as outlined in this ToR.
• Provide regular progress reports and updates to the client.
• Ensure adherence to industry best practices and data security standards.

10. PAYMENT AND COMPENSATION

Payment details and compensation for the consultancy services will be discussed and agreed upon in a separate agreement. Final payment will be made only upon acceptance of final deliverables by World Relief Project Manager.

11. EVALUATION AND QUALITY ASSURANCE

The project’s success will be evaluated based on the fulfillment of the objectives, the quality of the deliverables, and client satisfaction. Periodic reviews and feedback sessions will be conducted to ensure project quality.

12. CONFIDENTIALITY AND DATA SECURITY

World Relief has sole ownership of all data, and no data or findings shall be shared or reproduced without the permission of World Relief. All project-related information, data, and intellectual property will be handled with strict confidentiality and in compliance with relevant data protection regulations.

13. LEGAL CONSIDERATIONS

This project is subject to applicable laws and regulations, including data protection, intellectual property, and any other legal requirements.

14. INQUIRIES

World Relief is seeking support from a consultant to lead the process as outlined in this document. Individuals interested in providing consultancy services associated with this project should contact [email protected] to request additional information by January 12th, 2024.