AWS Data Engineer (AWS Glue & PySpark)

We are seeking a highly skilled AWS Data Engineer with strong expertise in AWS Glue, PySpark, Python, Redshift, Athena, and AWS Analytics Services. The ideal candidate will be responsible for designing, developing, and optimizing scalable ETL pipelines and cloud-based data integration solutions within an AWS ecosystem. This role requires hands-on development experience and is focused on Data Engineering rather than DevOps.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data integration pipelines using AWS services.
• Build data ingestion frameworks from multiple heterogeneous sources including databases, files, APIs, and Kafka streams.
• Develop data transformation and processing solutions using AWS Glue, PySpark, and Python.
• Create and optimize data pipelines leveraging AWS Lambda, Athena, Step Functions, Redshift, S3, and EC2.
• Develop and maintain data models and database objects for analytics and reporting requirements.
• Implement real-time and batch data processing workflows.
• Collaborate with architects, product owners, and business stakeholders to deliver data platform solutions.
• Perform code reviews and ensure adherence to architecture and development standards.
• Participate in Agile development processes including sprint planning, development, testing, and releases.
• Support data delivery requirements across multiple business use cases.
• Troubleshoot performance bottlenecks and optimize cloud-based data solutions.
• Demonstrate solutions to clients and internal stakeholders when required.

Required Skills

• Minimum 4 years of experience in Data Integration and Data Engineering.

• 3+ years of hands-on experience with AWS Glue.

• Strong expertise in PySpark and Apache Spark.

• Strong Python programming experience.

• Experience with AWS Lambda, Athena, Step Functions, and Redshift.

• Experience designing and developing ETL pipelines on AWS.

• Strong SQL development skills.

• Experience with PostgreSQL, Redshift, or other relational databases.

• Experience implementing data models for analytics platforms.

• Hands-on experience with AWS services including:

• AWS Glue

• Lambda

• Athena

• Redshift

• Step Functions

• S3

• EC2

• Experience with real-time and scalable workflow implementations.

• Understanding of BI and Analytics solution development lifecycle.

Preferred Skills

• Informatica Cloud.
• Talend.
• SSIS.
• Matillion.
• Kafka integration and streaming technologies.
• Enterprise Data Warehousing solutions.
• Cloud-native Analytics platforms.

Desired Candidate Profile

• Strong problem-solving and analytical capabilities.
• Excellent verbal and written communication skills.
• Experience working on multiple projects simultaneously.
• Ability to work in Agile delivery environments.
• Strong collaboration and stakeholder management skills.
• Proactive approach to issue resolution and project delivery.
• Ability to work effectively under tight timelines.