Senior Data Engineer

About the Role

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Microsoft Azure, Azure Databricks, Python and cloud-native data engineering.

The successful candidate will be responsible for developing and managing scalable data processing solutions, cloud data platforms, CI/CD pipelines and API-based services within the Azure ecosystem.

Key Responsibilities

• Design, develop and maintain cloud-native data engineering solutions on Microsoft Azure.
• Develop scalable data processing solutions using Python and Apache Spark.
• Build and manage data processing workflows and orchestration pipelines using Azure Databricks.
• Work with Azure Data Lake Storage (ADLS) and integrate ADLS mount points with Databricks.
• Manage and optimise Databricks compute clusters.
• Work with Hive Metastore and support data governance practices.
• Implement and manage secrets and keys using Azure Key Vault.
• Develop and maintain CI/CD pipelines using Azure DevOps and GitHub Actions.
• Create deployment pipelines using YAML.
• Work with GitHub and Azure Repos using branching, pull request and code review practices.
• Monitor and troubleshoot applications and data platforms using Azure Log Analytics.
• Develop and consume APIs using Databricks CLI and Azure CLI.
• Develop production-ready APIs using FastAPI.
• Implement data validation and configuration management using Pydantic.
• Deploy production APIs using Gunicorn and Uvicorn.
• Build containerised applications using Docker and manage images through Azure Container Registry (ACR).
• Implement messaging and caching solutions using NATS and Redis.
• Participate in Scrum ceremonies and collaborate with cross-functional project teams.
• Contribute to technical design, troubleshooting and continuous improvement activities.

Requirements

• Minimum 7 years of relevant Data Engineering experience, Azure DevOps, Azure Databricks and Python experience.
• Strong hands-on experience with:
• Microsoft Azure
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Python
• Apache Spark
• SQL
• Azure DevOps
• Strong understanding of cloud-native development and Azure data platforms.
• Experience managing Databricks workflows and orchestration pipelines.
• Experience with Azure Key Vault and secure secret/key management.
• Good understanding of CI/CD pipelines and YAML.
• Experience with GitHub and/or Azure Repos.
• Strong understanding of data processing and distributed computing.

Pay: RM10,000.00 - RM13,500.00 per month

Work Location: In person