Senior Developer - Databricks Data Engineering

Senior Developer Databricks Data Engineering

Experience: 5 - 9 Years
Employment Type: C2H, Full-Time
Work Mode: Hybrid
Location: Bangalore & Pune

Role Overview

We are looking for a Senior Developer Databricks Data Engineering with strong hands-on experience in Databricks, PySpark, Spark SQL, and Delta Lake to develop and maintain scalable data engineering pipelines. The ideal candidate should have experience building ETL/ELT pipelines, implementing incremental/CDC ingestion, optimizing Spark workloads, and working with Lakehouse architecture.

Key Responsibilities

• Develop and maintain PySpark and Spark SQL notebooks and jobs on Databricks.
• Build and manage scalable ETL/ELT pipelines across Bronze, Silver, and Gold Lakehouse layers.
• Design and maintain Delta Lake tables, including partitioning and schema evolution.
• Implement incremental and CDC ingestion using Delta MERGE and Auto Loader.
• Build and maintain dimensional data models for BI and analytics.
• Monitor, troubleshoot, and optimize Databricks jobs and Spark pipelines.
• Perform Spark and Delta Lake performance tuning to meet SLA and cost targets.
• Support production issue triage, root-cause analysis, and resolution.
• Develop reusable PySpark utilities and frameworks to improve engineering standards.
• Collaborate with technical leads and upstream/downstream teams on data pipeline requirements and changes.

Required Skills

• 4+ years of hands-on experience in Data Engineering.
• Strong experience with Databricks and Apache Spark.
• Proficiency in PySpark, Spark SQL, Python, and SQL.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Experience with Databricks Workflows / Jobs.
• Experience building ETL/ELT data pipelines.
• Knowledge of incremental and CDC ingestion using MERGE / Auto Loader.
• Experience with Spark performance tuning and query optimization.
• Understanding of dimensional data warehouse modeling.
• Experience with cloud storage such as ADLS or S3.
• Strong problem-solving and debugging skills.

Good to Have: Databricks Data Engineer certification, Kafka/Event Hubs experience, and Git/GitHub experience.