Senior Azure Data Engineer Databricks & DevOps

About Cloudaeon

Sr Azure Data Engineer Job Description

Cloudaeon is a global technology consulting and services company helping enterprises modernize and manage their cloud infrastructure, big data ecosystems, DevOps pipelines, and advanced analytics platforms. We deliver high-quality, scalable solutions built on deep domain expertise and leading enterprise data technologies. Our teams work actively on AI and GenAI initiatives, incorporating machine learning, intelligent automation, and modern data engineering practices to create future-ready digital solutions. With a highly skilled global team committed to excellence, we consistently exceed customer expectations and help organizations achieve their strategic goals.

Job Role – Sr Azure Data Engineer

Experience – 5+ years

Location – Pune (Hybrid)

Job Description

We are looking for a Senior Azure Data Engineer with strong expertise in designing and delivering modern data solutions on the Azure platform. The ideal candidate will lead data engineering initiatives, mentor junior engineers, and contribute to building scalable, secure, and high-quality data products. A strong foundational understanding of AI/ML concepts is essential, enabling a smooth transition into AI Engineering over time. The ideal candidate should also have a strong engineering and automation mindset, with hands-on experience improving Databricks deployment and CI/CD frameworks. The role requires someone who can combine strong Data Engineering capabilities with practical DevOps automation and modern Databricks deployment practices.

Responsibilities:

• Design, build, and optimize modern data pipelines and real-time/near real-time data streams.
• Transform and move data across bronze, silver, and gold layers using ADF, Python, and PySpark.
• Evaluate current data practices and identify opportunities for improvement and modernization.
• Leverage Azure Data Platform capabilities including ADF, ADLS, Databricks, Synapse, Azure Functions, Event Hub, and Azure Data Explorer.
• Support planning, architecture, and deployment of scalable data platform services (sizing, configuration, performance, and cost optimization).
• Implement automated monitoring, alerts, and operational dashboards.
• Ensure CI/CD and DevOps integration of data pipelines and Databricks workloads.
• Refine and implement CI/CD processes with emphasis on GitHub-based workflows and Unix/Linux scripting.
• Use Databricks Asset Bundles (DABs) for deployment and automation across development, test/UAT, and production environments.
• Use the Databricks Terraform Provider to provision and manage Databricks resources and infrastructure.
• Apply appropriate declarative and imperative approaches for Databricks deployment, configuration, and automation.
• Design, develop, optimize, and support end-to-end production-grade data pipelines, preferably in Databricks environments.
• Identify opportunities to automate and improve the existing Databricks deployment and CI/CD framework, improving reliability, consistency, scalability, and developer productivity.
• Collaborate closely with stakeholders and work as a trusted technology advisor.
• Provide technical leadership, code reviews, and mentorship to junior data engineers.
• Prepare AI-ready datasets, support feature engineering, vectorization, embeddings, and ML workflows using Azure Machine Learning and Databricks AI.
• Contribute to RAG pipelines, semantic search, and model deployment integrations.

Requirements :

• 5+ years of experience with Azure Data Engineering (ADF, Databricks, Python, PySpark, SQL).
• Strong understanding of data warehousing and data modelling.
• Proven experience building production-grade ETL/ELT pipelines.
• Hands-on experience with ADLS, Delta Lake, Synapse, and DevOps CI/CD.
• Strong hands-on experience with GitHub-based CI/CD workflows and Unix/Linux scripting; Azure DevOps experience is not mandatory.
• Hands-on experience with Databricks Asset Bundles (DABs) for deployment automation across environments.
• Practical experience using the Databricks Terraform Provider to provision and manage Databricks resources/infrastructure.
• Good understanding of declarative vs. imperative approaches, particularly in the context of Databricks automation and deployment.
• Strong hands-on PySpark development experience, including building and optimizing production-grade data pipelines.
• Proven experience designing, developing, and supporting end-to-end data pipelines, preferably in Databricks environments.
• Strong engineering and automation mindset, with the ability to evaluate and improve existing deployment and CI/CD frameworks.
• Ability to convert business requirements into scalable technical solutions.
• Excellent communication and problem-solving skills.
• Customer-centric mindset with passion for engineering quality.
• Working knowledge of Databricks AI, MLflow, model serving, feature store, vector search, and AI/ML workflows.

Nice to Have

• Experience with Azure OpenAI, Fabric AI, MLOps, and RAG pipelines.
• Experience with reusable Terraform modules and enterprise Infrastructure as Code practices.
• Experience with advanced GitHub Actions/workflow automation for data platforms.
• Experience improving deployment reliability, release governance, and developer self-service.

Ideal Candidate Profile A strong Senior Azure Data Engineer who combines deep PySpark and Databricks data engineering expertise with a hands-on engineering/automation mindset, and can improve and modernize the existing Databricks deployment and CI/CD framework.