AWS Pysark Developer

We need a Strong AWS PySpark Developer for our data engineering team. You will design and optimize scalable AWS data pipelines, leveraging expertise in distributed processing, PySpark, SQL, and Agile methodologies to maintain high data quality standards.

Key Responsibilities

• Data Pipeline Architecture & ETL: Build / Implement large-scale, high-performance data pipelines using various AWS Services like AWS EMR, PySpark, AWS Glue, and AWS DataBrew.
• Work in  Waterfall / Hybrid Agile Teams using JIRA, Actively participate in planning and managing the backlog and leveraging JIRA reporting/dashboards to track team velocity, bottlenecks, and project health.
• Reporting : Daily, Weekly and Monthly reporting to ensure Offshore and Onshore leads are fully informed and engaged.
• Data Governance & Quality: Build solutions that guarantee high Data Quality and implement comprehensive Data Lineage tracking across all enterprise data workflows.
• Performance Optimization: Apply advanced PySpark optimization techniques (handling data skewness, memory management, partitioning, and broadcasting) and rigorous SQL optimization.
• Serverless Data Processing: Architect and deploy event-driven data workflows and microservices utilizing AWS Lambda.
• Data Warehousing: Model, manage, and optimize data storage and querying within AWS Redshift for analytical reporting and BI consumption.
• Accelerated Delivery: Utilize AI coding assistants (such as GitHub Copilot and Claude Code) to streamline development, improve code quality, and accelerate project delivery timelines.

Required Qualifications & Skills

• Experience: Minimum of 4-6 years of hands-on experience in Data Engineering, Big Data, or Cloud Architecture
• Python & AWS Expertise: Proven experience operating as an AWS Data Engineer using Python (Active AWS certifications are highly preferred).
• PySpark Mastery: Exceptional proficiency in Python and Apache Spark. Must have a deep understanding of Spark's internal workings and hands-on experience optimizing heavy PySpark workloads.
• AWS ETL Ecosystem: Strong, demonstrated experience with core AWS data services including AWS EMR, AWS Glue, AWS DataBrew, AWS Lambda, and AWS Redshift.

AWS S3, IAM, and AWS Step Functions.

• SQL & Database Skills: Advanced SQL proficiency with a track record of tuning complex queries for performance across distributed databases.
• Agile & JIRA: Demonstrated experience driving Agile methodologies. Must be well-versed in JIRA, specifically in configuring workflows and generating detailed JIRA reports for stakeholders.
• DevOps & CI/CD: Strong proficiency in version control and DevOps practices using Git and GitHub Actions for automated CI/CD pipelines.
• AI-Assisted Development: Demonstrated experience successfully integrating AI tools (Copilot, Claude Code) into daily engineering workflows to boost productivity.

Nice to Have

• Experience with workflow orchestration tools like Apache Airflow.
• Knowledge of Infrastructure as Code (IaC) tools such as Terraform or AWS CloudFormation.