AWS Data Platform Support Engineer

Job Summary:
We are seeking a skilled and proactive AWS Data Platform Support Engineer with 3 to 5 years of experience to join our team. This is primarily a Production Support role responsible for monitoring, troubleshooting, incident resolution and maintaining AWS-based data platforms. The candidate must have strong hands-on experience supporting AWS Cloud, S3, EMR, CloudWatch, Lambda, Amazon Redshift, WLM configuration Apache Airflow, DAGs, Python and SQL in a production support environment. The role requires strong troubleshooting skills across Airflow DAGs, EMR jobs, Redshift queries/WLM and AWS services.
Candidates with only ETL/Data Pipeline Development experience and without hands-on production support experience will not be considered.

Key Responsibilities:
Monitor and provide production support for AWS-based data platforms and workflows.
Monitor and troubleshoot Apache Airflow DAGs, including DAG/task failures, dependency issues, scheduling errors and retries.
Investigate Airflow logs, identify root cause, resolve issues and rerun failed DAGs/tasks, followed by post-rerun validation.
Troubleshoot AWS EMR clusters and jobs, including job failures, resource issues, YARN/Spark issues and analyze EMR/CloudWatch logs.
Provide production support for Amazon Redshift, including query failures, long-running queries, performance and concurrency issues.
Write and optimize complex SQL queries for production troubleshooting, data validation, issue investigation and performance analysis.
Troubleshoot and managet WLM configuration, including queues, concurrency, query queueing and workload-related issues.
Monitor and troubleshoot AWS S3, Lambda and CloudWatch, including file/access issues, Lambda failures, alerts, logs and metrics.
Develop and maintain Python scripts for support automation, housekeeping and operational activities.
Perform incident management, root cause analysis, recovery/rerunning of jobs and participate in production support.

Required Skills & Qualifications:
3-5 years of hands-on AWS Production Support Data Platform Support L2-L3 Support experience.
Strong hands-on experience with AWS Cloud, AWS S3, AWS EMR, AWS CloudWatch and AWS Lambda in production environments.
Strong hands-on experience with Apache Airflow Production Support, including DAG monitoring, error finding, troubleshooting, rerunning and recovery.
Experience with Airflow DAG development/modification and debugging; primary experience must be in support rather than ETL pipeline development.
Strong hands-on experience with Amazon Redshift Production Support, including query troubleshooting and performance investigation.
Candidates must have strong hands-on Redshift SQL query-writing skills.
WLM Configuration and Troubleshooting is mandatory, including queues, concurrency and query queueing.
Strong experience with AWS CloudWatch Logs, Metrics and Alarms for production monitoring and troubleshooting.
Strong proficiency in SQL and Python, specifically for production troubleshooting, validation, scripting and automation.
Proven experience in incident management, root cause analysis, production recovery, job rerun and validation

Preferred Skills
Familiarity with Terraform for basic AWS infrastructure configuration and troubleshooting.
Exposure to Apache Spark, Hive, Azure Devops.

Candidates Profile
3-5 years of experience in AWS Data Platform Production Support L2-L3 Support.
Majority of experience should be in production support, monitoring, troubleshooting and incident resolution, not ETL development.
Strong hands-on experience with AWS Cloud, S3, EMR, CloudWatch, Lambda and Redshift.
Hands-on Apache Airflow production support - monitoring DAGs, identifying errors, checking logs, troubleshooting failed tasks/DAGs and rerunning/recovering jobs.
Hands-on experience with Redshift WLM configuration and troubleshooting, including queues, concurrency and query performance.