Azure Databricks Engineer

Job Summary

Design and develop end-to-end cloud-native data platforms using Azure services integrated with Databricks. Architect scalable and secure Lakehouse architectures leveraging Azure Databricks and Delta Lake. Establish and enforce enterprise data standards, governance, and security policies aligned with Azure best practices. Optimize data processing workflows for performance, reliability, and cost efficiency within Azure environments. Collaborate with cross-functional teams to integrate diverse data sources and Azure cloud services. Stay updated with emerging Azure cloud technologies and Databricks advancements to drive innovation and modernization.

Roles and Responsibilities

• Lead architecture, deployment, and management of Azure Databricks workspaces and multi-environment setups.

• Define and enforce best practices for cluster management, autoscaling, job orchestration, and workload optimization.

• Drive migration of legacy ETL workloads to Azure Databricks and implement Delta Lake Medallion Architecture.

• Implement robust data governance using Azure Key Vault, Managed Identity, and role-based access controls.

• Monitor, troubleshoot, and optimize PySpark and Spark SQL code for large-scale data processing.

• Collaborate with business stakeholders, data architects, and engineering teams to align data solutions with strategic goals.

• Mentor and guide junior team members on Azure Databricks and cloud architecture best practices.

• Develop and maintain CI/CD pipelines for data workflows leveraging Azure DevOps.

  Location

Locations: Bangalore, Pune, Hyderabad

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.