Senior Data Engineer

Role & responsibilities

Hiring: Senior Data Engineer Databricks / AWS / PySpark

Location: Pan India
Experience: 6+ Years

Relevant Experience: 4+ Years

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, SQL, AWS, and ETL/ELT pipeline migration to join our team.

Key Responsibilities

• Analyze existing ETL pipelines, data sources, and transformation logic across platforms
  • Drive end-to-end migration of data pipelines to Databricks
  • Design, develop, and maintain scalable batch and near-real-time data pipelines
  • Re-engineer legacy ETL/ELT workflows using Spark, PySpark, SQL, and Databricks
  • Integrate data from multiple heterogeneous sources
  • Implement robust data transformation and processing logic
  • Perform data validation, reconciliation, and consistency checks after migration
  • Optimize pipelines for performance, scalability, and cost efficiency
  • Collaborate with business stakeholders, analysts, and data teams to understand requirements
  • Support reporting, analytics, and downstream applications with reliable datasets
  • Monitor production pipelines, troubleshoot failures, and resolve data issues
  • Implement logging, alerting, and observability using CloudWatch / Databricks monitoring
  • Follow data governance, security, and compliance standards
  • Implement CI/CD for data workflows and maintain technical documentation

Mandatory Skill

Strong proficiency in SQL and relational databases such as Oracle, PostgreSQL, etc.

Required / Desired Skills

Strong SQL and database expertise
ETL/ELT development and migration experience
Strong programming skills in Python and/or Scala
Hands-on experience with Apache Spark / PySpark
Strong experience with Databricks – Delta Lake, Notebooks, Jobs, Workflows
AWS data engineering services:

• AWS Glue
• AWS Lambda
• Amazon S3
• EventBridge
• SQS
• Amazon Redshift
• Kinesis Firehose
• CloudWatch
  Knowledge of data modeling, data warehousing, and data lake architectures
  Experience handling large-scale datasets and distributed processing
  Knowledge of pipeline orchestration and workflow management