AWS Pyspark engineer

Job Description: AWS PySpark Data Engineer

Job Title: AWS PySpark Data Engineer

Experience: 5+ Years

Location: Indore

Job Summary

We are looking for an experienced AWS PySpark Data Engineer with strong hands-on expertise in building scalable data engineering solutions on AWS. The role will focus on developing and optimizing ETL/ELT pipelines, processing large-scale datasets using PySpark/Spark, and building cloud-native data solutions using AWS services.

Mandatory Skills

• 5+ years of overall experience
• PySpark / Apache Spark
• Python
• SQL
• AWS Glue
• Amazon S3
• AWS EMR
• AWS Lambda
• Amazon Redshift
• ETL/ELT development
• Data Lake/Data Warehouse concepts
• Large-scale/distributed data processing

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines using PySpark .
• Build data ingestion and transformation frameworks on AWS.
• Process large volumes of structured, semi-structured and unstructured data.
• Develop and optimize Spark/PySpark workloads.
• Build cloud-native Data Lake and Data Warehouse solutions.
• Implement data quality, validation, monitoring and governance.
• Monitor and troubleshoot production data pipelines.
• Participate in CI/CD, deployments and Agile delivery.
• Work closely with data architects, analytics teams and business stakeholders.

AWS Technology Stack

Core: S3, Glue, EMR, Lambda, Redshift

Additional: Athena, IAM, CloudWatch, Step Functions

Good-to-Have

• Databricks
• Apache Kafka
• Apache Airflow
• Snowflake
• Delta Lake
• Terraform
• Docker/Kubernetes
• Streaming data pipelines