Azure Databricks Data Engineer

Job Summary

We are seeking a skilled Databricks Engineer with minimum of 8+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.

Key Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
• Develop data models and implement data quality, validation, and governance frameworks.
• Integrate data from multiple sources into a unified Lakehouse architecture.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Implement security controls, access management, and data governance using Unity Catalog.
• Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
• Monitor, troubleshoot, and resolve data pipeline issues.
• Support CI/CD, DevOps, and infrastructure automation practices.
• Maintain technical documentation and best practices.

Required Technical Skills

Core Technologies

• Databricks Lakehouse Platform
• Apache Spark / PySpark
• Delta Lake
• SQL
• Python

Data Engineering

• ETL / ELT Development
• Data Modeling
• Data Warehousing
• Data Quality & Validation
• Streaming & Real-Time Processing

Governance & Security

• Unity Catalog
• Data Lineage
• Row-Level Security
• Access Control & Compliance
• Data Governance Frameworks

Cloud & DevOps

• Azure / AWS / GCP
• Terraform
• GitHub Actions / Azure DevOps
• CI/CD Pipelines

Analytics & AI

• Semantic Layers
• Data Products
• BI Platforms
• Machine Learning Support
• Generative AI & RAG Architectures

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
• 8+ years of experience in Data Engineering and Data Warehousing.
• Minimum 8+ years of hands-on experience with Azure Data Engineering technologies.
• Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience in handling large-scale structured and unstructured datasets.
• Strong analytical, problem-solving, and troubleshooting skills.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203) .
• Databricks Certified Data Engineer Associate/Professional.
• Experience with Snowflake, Power BI, or Microsoft Fabric.
• Experience in real-time streaming solutions using Kafka/Event Hubs.
• Exposure to Data Governance and Master Data Management initiatives.

Soft Skills

• Strong stakeholder management and communication skills.
• Ability to lead technical initiatives and drive architecture discussions.
• Experience working in Agile/Scrum environments.
• Excellent documentation and presentation skills.
• Strong mentoring and team leadership abilities.

Nice to Have

• Microsoft Fabric
• Power BI
• Azure Event Hubs
• Kafka
• Machine Learning data pipelines
• Data Governance tools such as Purview