Python with Pyspark AWS Data Engineer

Role Data Engineer Skills – Mandatory

AWS EMR, Apache Spark, Postgre SQL, Dynamo DB, Apache Iceberg

Skills - Primary

AWS EMR, Apache Spark, Postgre SQL, Dynamo DB, Apache Iceberg

Skills - Good to have

Python and Spark programming

Qualification

Graduate

Total Experience

5+ years

Relevant Experience

3+ years

Work Location

Remote

Expected Date of Onboarding

Immediate

Work Shifts – General / US / UK

General with US overlapping hours(for this Client, the working hours typically extend until 10:30 PM – 11:00 PM IST. The start time in the morning will be flexible)

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines. This role will play a critical part in developing modern data lakehouse architecture on AWS, enabling advanced analytics and data-driven decision-making across the organization

Job Description / Duties and Responsibilities

Design, develop, and maintain analytical data models and metrics using modern, scalable data platform technologies.

Architect and implement data lakehouse solutions leveraging AWS services.

Build, optimize, and maintain robust data pipelines using PySpark, Python, and AWS Glue.

Ensure high performance, reliability, and scalability of data systems.

Collaborate closely with business stakeholders to translate requirements into data solutions.

Partner with architecture and operations teams to enforce best practices, standards, and governance.

Monitor and troubleshoot data workflows, ensuring data quality and system efficiency.

To adhere to the Information Security Management policies and procedures.

Job Specification / Skills and Competencies

• 5+ years of professional experience in software development, with a focus on analytical platform development.
• 3+ years of hands-on experience with data platforms and big data ecosystems.
• Strong expertise in big data technologies, including:

AWS EMR

Apache Spark

PostgreSQL

DynamoDB

Apache Iceberg

Proficiency in Python and Spark programming.

Experience building highly scalable, low-latency data systems (e.g., DynamoDB, Aurora, PostgreSQL, Redshift).

Proven experience designing and implementing data lakehouse architectures on AWS.

Hands-on experience with AWS services such as Step Functions, Lambda, and EventBridge.

Experience working in Agile development environments.

Preferred Qualifications

Strong knowledge of additional AWS services such as: Kinesis

Firehose

SQS

Bedrock

SageMaker

Experience with real-time

data streaming and event-driven architectures.

Familiarity with machine learning data pipelines is a plus.

Soft Skills & Competencies

Strong ability to learn and adapt to new technologies quickly.

Excellent problem-solving and analytical thinking skills.

Ability to work independently and collaboratively in a team environment.

Strong multitasking skills in a fast-paced, dynamic setting.

Excellent verbal and written communication skills.