Senior Data Engineer Spark Scala Azure Databricks

Key Responsibilities

• Design, develop and maintain scalable data engineering solutions using Apache Spark and Scala.
• Build and manage data pipelines using Airflow and related orchestration technologies.
• Work extensively with Hive, Kubernetes, Docker and GitHub Actions.
• Develop and optimize data solutions using Apache Iceberg.
• Design and implement data engineering solutions on Azure Databricks.
• Work with ADLS, ADF, Delta Lake, Databricks Jobs & Pipelines, Lakeflow Connect and Unity Catalog.
• Work with both Classic and Serverless Databricks Compute environments.
• Implement scalable, reliable and high-performance data pipelines.
• Follow CI/CD and DevOps best practices using GitHub Actions.
• Explore and implement AI-assisted development practices across SDLC and data engineering.
• Collaborate with cross-functional teams to deliver innovative data solutions aligned with business requirements.

Must-Have Skills

1. Spark, Scala, Airflow, Hive
2. Azure Databricks, ADLS, ADF, Delta Lake
3. Kubernetes, Docker, GitHub Actions

Strongly Preferred Skills

• Apache Iceberg
• Databricks Jobs & Pipelines
• Lakeflow Connect
• Unity Catalog
• Databricks Classic & Serverless Compute
• Azure Data Engineering
• CI/CD
• AI-assisted SDLC / AI for Data Engineering

Candidate Profile

• 6+ years of overall experience in Data Engineering.
• Strong hands-on experience with Spark and Scala.
• Good experience in Azure Databricks ecosystem.
• Strong understanding of data pipelines, orchestration and distributed data processing.
• Experience with containerization and DevOps tools.
• Ability to work from Whitefield office, Bangalore, 3 days per week.
• Candidates should be available for the Saturday interview.