Senior Data Engineer

We are seeking a Data Engineer with strong experience in building scalable data pipelines and processing large datasets using AWS and PySpark. The role involves designing robust data ingestion and transformation frameworks, enabling efficient data integration, and ensuring high performance and reliability of data systems.

Key Responsibilities

Develop ETL/ELT workflows to integrate data from multiple enterprise sources.

Build batch and streaming pipelines for large-scale data processing, preferable using cloud platforms such as AWS

Write and optimize complex SQL and PySpark transformations for analytics and reporting.

Ensure pipelines are performant, scalable, and reliable.

Troubleshoot and resolve issues related to data ingestion, processing, and platform integrations.

Support data model implementation and pipeline configuration migrations across environments.

Collaborate with data architects, analytics teams, and platform engineers to enable efficient data access.

Participate in agile ceremonies and technical discussions while contributing to engineering best practices.

Required Skills

Strong hands-on experience with AWS (mandatory) Key services include: AWS Glue AWS Lambda Amazon Kinesis Data Streams Amazon SQS DynamoDB or other NoSQL databases Athena / Redshift (preferred) Programming & Data Processing Python PySpark (batch and streaming) SQL Data Engineering

Experience building scalable data pipelines and ETL/ELT frameworks

Experience working with large-scale structured and semi-structured datasets

Ability to write and optimize complex SQL queries and PySpark transformations

Experience integrating applications using REST APIs. Good to Have

Exposure to Adobe Experience Platform (AEP) or other Customer Data Platforms.

Experience with streaming frameworks such as Kafka. Understanding of data modeling, data governance, and large-scale analytics systems.

Experience 5 years of professional experience in data engineering 4+ years of experience building PySpark-based data pipelines (batch and streaming) 4+ years of experience with AWS data engineering services