Azure Databricks Engineer

Description :

Industry & Sector : Cloud Data & Analytics consulting focused on Microsoft Azure delivering enterprise data platforms, analytics, and AI-ready data lakes for customers across financial services, retail, and technology.

• We build production-grade, secure, and highly performant data pipelines and analytics solutions.

Primary Title : Azure Databricks Engineer.

Location : India On-site (client-delivery focused).

Role & Responsibilities :

• Design, build and maintain scalable ETL/ELT data pipelines on Azure Databricks using PySpark and Apache Spark for batch and streaming workloads.

• Implement and optimize Delta Lake architectures (schema evolution, partitioning, compaction, time travel) to ensure reliable data lakes and high query performance.

• Integrate Databricks jobs with Azure Data Factory for orchestration; implement parameterized pipelines, job scheduling, and failure/retry strategies.

• Ingest and transform data from ADLS Gen2, Event Hubs/Streaming sources, and relational systems; define data models and efficient Parquet/Delta storage strategies.

• Establish CI/CD practices for notebooks and infrastructure using Git and Azure DevOps; automate deployments, unit tests, and environment promotion.

• Monitor and tune Spark job performance, troubleshoot production issues, and build observability (logging, metrics, alerts) and runbooks to meet SLAs.

Skills & Qualifications :

Must-Have :

• 5+ years of hands-on experience delivering data engineering solutions on Azure with strong expertise in Azure Databricks and Apache Spark (PySpark).

• Proven experience with Delta Lake, data modeling, partitioning strategies, and Spark performance tuning.

• Practical knowledge integrating Databricks with Azure Data Factory and ADLS Gen2 for end-to-end ETL/ELT workflows.

• Proficiency in SQL and Python notebook development; experience with Git-based source control and Azure DevOps CI/CD pipelines.

• Experience operating production data platforms : monitoring, alerting, job orchestration, and creating operational runbooks.

• Ability to work onsite in India and collaborate directly with cross-functional engineering and client teams.

Preferred :

• Experience with Scala and building streaming data pipelines.

• Familiarity with Apache Airflow, Azure Synapse Analytics, or Terraform for orchestration and infra-as-code.

• Databricks or Azure data engineering certifications (Databricks Certified / Azure Data Engineer Associate) a plus.

Benefits & Culture Highlights :

• Opportunity to work on enterprise-scale Azure data platforms and cross-industry analytics projects with on-site delivery teams.

• Structured learning and certification support, mentorship, and clear career progression for data engineers.

• Collaborative, delivery-oriented culture with emphasis on engineering excellence and operational reliability.