Senior Data Engineer - Azure Databricks, PySpark, ADF, Kafka

Senior Data Engineer - Databricks | Hyderabad | Immediate joiners to 30 days only

We are hiring an experienced Senior Data Engineer for an enterprise data engineering programme with a leading global client. You will design, build and run large-scale batch and streaming data pipelines on Azure Databricks. This is a hands-on role for someone who has built production pipelines end to end.

Roles & Responsibilities:

• Design and build scalable ETL/ELT pipelines on Azure Databricks using PySpark and Spark SQL

• Build and orchestrate workflows in Azure Data Factory (pipelines, linked services, triggers, parameterisation, monitoring)

• Develop real-time and near-real-time ingestion using Apache Kafka with Spark Structured Streaming

• Implement Delta Lake and medallion (bronze/silver/gold) architecture

• Write complex, high-performance SQL for transformation, validation and reporting

• Tune Spark jobs and clusters for performance and cost

• Turn stakeholder requirements into technical design

• Own testing, CI/CD deployment, production support and data-quality checks

• Document pipelines and mentor junior engineers

Mandatory Skills (hands-on in ALL):

• Azure Databricks (notebooks, jobs/workflows, Delta Lake, clusters)

• PySpark / Spark SQL with performance tuning

• Azure Data Factory - production ETL pipelines end to end

• Advanced SQL (complex joins, window functions, query tuning)

• Kafka - streaming ingestion (producers/consumers, topics, Spark Structured Streaming)

• ETL design, data modelling and data warehousing

Good to Have:

• Unity Catalog, Delta Live Tables, Databricks Workflows

• ADLS Gen2, Azure Synapse, Event Hubs

• CI/CD with Azure DevOps or GitHub

• Databricks or DP-203 certification

• Airflow or Snowflake

Qualification: B.E./B.Tech/MCA/M.Sc (Computer Science or equivalent) - mandatory. 7+ years total IT experience.

Notice period: Immediate to 30 days only.

How to apply: Share your updated resume with current CTC, expected CTC, notice period and current location.