Azure Databricks Engineers

Role Summary

Design, develop, and maintain scalable data engineering solutions on the Azure platform using Azure Databricks, Azure Data Factory, ADLS Gen2, and related Azure data services. Build enterprise-grade data pipelines, transformation frameworks, and Lakehouse architectures.

Key Responsibilities

• Design and implement end-to-end data pipelines using Azure Databricks and Azure Data Factory.
• Develop scalable ETL/ELT solutions using PySpark and Spark SQL.
• Build and maintain Lakehouse architectures using Bronze, Silver and Gold layers.
• Develop Delta Lake-based data models and optimize data processing performance.
• Integrate data from ERP, CRM, databases, APIs, IoT, and external sources.
• Implement data quality checks, monitoring, and error-handling frameworks.
• Create and maintain CI/CD pipelines using Azure DevOps.
• Configure and manage Unity Catalog, data governance, and access controls.
• Work with Power BI and analytics teams to deliver curated data marts and semantic models.
• Collaborate with business stakeholders, architects, and data scientists to support reporting and AI initiatives.
• Troubleshoot production issues and optimize pipeline performance.
• Prepare technical documentation, deployment guides, and operational runbooks.

Required Technical Skills

• Azure Databricks
• PySpark and Spark SQL
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS)
• Delta Lake
• SQL Server and SQL scripting
• Azure DevOps and CI/CD
• Git/Version Control
• Data Warehousing and ETL Concepts
• Performance Tuning and OptimizationRole & responsibilities

Preferred candidate profile