Role
DevOps Lead – Azure (Part-Time)
Job Description
We are seeking an experienced DevOps Lead/Developer to join our team and take a key role in the data lakehouse implementation for a leading water utilities company in the Philippines. This position is ideal for someone who has expertise in DevOps practices, cloud infrastructure, and is passionate about ensuring smooth deployment and automation of data engineering processes using Microsoft Fabric and Azure.
As the DevOps Lead/Developer, you will be responsible for implementing and maintaining the automation, CI/CD pipelines, and infrastructure required to support the data lakehouse architecture. Your role will be pivotal in ensuring the reliability, scalability, and performance of the lakehouse environment. You will work closely with cloud architects, data engineers, and other technical teams to ensure the effective deployment and management of the platform.
Key Responsibilities
- Lead and implement DevOps practices for the Microsoft Fabric-based data lakehouse solution, ensuring continuous integration, continuous deployment, and automated workflows for the data engineering teams.
- Design and manage CI/CD pipelines for the data lakehouse using tools like Azure DevOps, Terraform, ARM templates, or other infrastructure-as-code (IaC) tools.
- Collaborate with data engineers, cloud architects, and data scientists to optimize deployment pipelines, automate testing, and enable the smooth deployment of data pipelines, applications, and cloud resources.
- Ensure cloud infrastructure management, including provisioning and scaling of resources on Azure, and optimize the performance and costs of data lakehouse environments, including Azure Data Lake, Azure Synapse Analytics, and Microsoft Fabric.
- Develop monitoring, alerting, and logging systems to ensure the health, availability, and security of the data lakehouse environment. Leverage Azure Monitor, Log Analytics, and Application Insights to implement robust observability.
- Build and implement automated deployment processes for infrastructure, data pipelines, and analytics workloads, ensuring efficiency, security, and high availability.
- Collaborate with cloud security and data governance teams to ensure that the data lakehouse environment adheres to security policies and regulatory compliance requirements (e.g., Philippine Data Privacy Act).
- Automate routine infrastructure and maintenance tasks to improve the efficiency of cloud resource management.
- Provide mentorship and guidance to junior DevOps engineers, promoting best practices and a collaborative, agile DevOps culture within the team.
- Ensure that the deployment environment is robust and scalable, with a focus on performance optimization, cost management, and reliability for the Microsoft Fabric lakehouse.
- Troubleshoot and resolve any infrastructure-related issues or performance bottlenecks, both in the cloud infrastructure and in data pipeline deployments.
Required Qualifications
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or related fields.
- 7+ years of experience in DevOps, cloud infrastructure, and automation practices, with a strong focus on Azure environments.
- Proficiency in Azure DevOps, Git, Terraform, and CI/CD pipeline design.
- In-depth experience with Microsoft Azure services, especially those relevant to data lakehouses, such as Azure Data Lake, Azure Synapse Analytics, Azure Databricks, Power BI, and Microsoft Fabric.
- Strong knowledge of infrastructure-as-code (IaC) tools like Terraform, ARM templates, or Bicep for resource provisioning and automation.
- Proven experience managing cloud environments with a focus on scalability, reliability, and security in large-scale data processing and analytics solutions.
- Experience with monitoring, alerting, and logging tools such as Azure Monitor, Log Analytics, and Application Insights to ensure optimal system health.
- Solid understanding of data security, data privacy, and compliance regulations, particularly within the context of utilities and critical infrastructure.
- Experience in automated deployment for both data engineering pipelines and cloud infrastructure in a dynamic, multi-team environment.
Preferred Qualifications
- Microsoft Certified: Azure DevOps Engineer Expert or other Azure certifications (e.g., Azure Solutions Architect, Azure Security Engineer).
- Experience with containerization technologies such as Docker, Kubernetes, or Azure Kubernetes Service (AKS) for deploying and managing containerized data workloads.
- Familiarity with big data processing frameworks like Apache Spark or Apache Kafka used in data engineering pipelines.
- Experience working with water utilities or other regulated industries, particularly with operational data from SAP IS-U, SCADA systems, or IoT sensors.
- Knowledge of Agile methodologies and experience working in cross-functional teams to deliver large-scale projects.
- Familiarity with data governance and metadata management tools such as Azure Purview to ensure compliance in data lakehouse environments.
Soft Skills & Competencies
- Leadership skills, with the ability to guide and mentor junior DevOps engineers and collaborate across multiple technical teams.
- Strong problem-solving and troubleshooting abilities, especially in cloud-based environments.
- Excellent communication skills, both verbal and written, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
- Highly organized with the ability to manage multiple projects simultaneously and prioritize tasks effectively.
- A proactive and collaborative team player who thrives in a dynamic, fast-paced environment.
Number of Vacancies
1