Associate Director - Databricks

Databricks

Role Overview

We are seeking a highly skilled Senior Data Engineer with extensive experience in Databricks and modern data platforms. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines, implementing best practices in data engineering, and leading client engagements and project teams.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-based data platforms
• Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) within Lakehouse environments
• Develop and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data
• Work with Delta Lake features including time travel, ACID transactions, and schema evolution
• Manage and govern data using Unity Catalog and ensure proper data security and compliance
• Build and maintain data models, data marts, and enterprise data warehouses
• Apply Slowly Changing Dimensions (SCD) techniques and best practices for data versioning
• Ensure performance optimization and cost efficiency of data workloads
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions
• Lead project execution including effort estimation, planning, and delivery management
• Mentor junior team members and guide development pods/teams

Required Skills & Qualifications

• Good of experience in Data Engineering, with strong focus on Databricks

• Strong programming expertise in:

• Python

• SQL

• Apache Spark (PySpark preferred)

• Hands-on experience with:

• Databricks platform and ecosystem

• Lakehouse architecture and Delta Lake

• Data pipeline orchestration and optimization

• Solid understanding of:

• Data Modeling techniques (dimensional modeling, star/snowflake schema)

• Data Warehousing concepts

• SCD (Slowly Changing Dimensions) approaches

• Experience handling large-scale data processing and distributed systems

Soft Skills

• Strong communication and stakeholder management skills
• Proven experience in leading client projects and cross-functional teams
• Ability to estimate work, manage timelines, and ensure quality delivery
• Strong analytical thinking and problem-solving capabilities

Preferred Qualifications

• Experience with cloud platforms (Azure/AWS/GCP)
• Knowledge of data governance and security best practices
• Exposure to CI/CD pipelines and DevOps practices in data engineering