AWS Data Engineer_PAN INDIA

Mandatory : One round will be F2F

Exp 7 + years

Location : PAN INDIA

Role Summary:
Primary focus is on data pipeline monitoring, establishing CI/CD processes, and resolving pipeline failures. Leads debugging, structured logging, and the design of robust error-handling frameworks. Additionally supports data ingestion, transformations, and Redshift data pipeline development as a secondary responsibility.

• Deep understanding of data pipeline monitoring, proactive alerting, and observability (e.g., AWS CloudWatch)
• Strong experience building CI/CD processes and deployment automation
• Advanced pipeline debugging, root-cause analysis, and exception handling
• Apache Airflow (DAG writing, scheduling, task dependencies, and failure management)
• Strong experience in SQL, Python (pandas, json, logging) and PySpark for troubleshooting and automation
• Version control with Git and branching strategies
• Working knowledge of AWS Glue, Redshift, and S3
• Familiarity with JSON parsing, flattening, and automated data validation checks
• SQL proficiency for investigating data issues and validating tables
• EMR exposure
• GitLab, Airflow, and CloudWatch

Key Skills to Look For:

• Monitoring & Debugging: Proven ability to track job failures, parse driver/executor logs, and set up automated alerts for pipeline SLA breaches.
• CI/CD & DevOps: Hands-on experience configuring GitLab CI to automate the testing, validation, and deployment of data models and pipeline code.
• Error Handling: Demonstrated experience writing robust, fault-tolerant Python/PySpark code with structured logging and automated retry mechanisms.
• Data Engineering Foundation: Solid competency in maintaining, optimizing, and supporting existing AWS data workflows (S3, Glue, Redshift) alongside the core reliability tasks.

Below is the summary of the client expectation.

1. Production Support / Troubleshooting

• Ability to troubleshoot web services and data pipelines.
• Understanding of incident diagnosis and step-by-step problem resolution.
• Some candidates demonstrated strong troubleshooting skills, while others struggled.

2. Data Engineering & Pipeline Design

• Designing end-to-end data pipelines.
• Spark architecture, optimization, and troubleshooting.
• Data modelling and warehouse design.

Key Observations

• Most candidates performed well on Spark-related topics.
• Several candidates lacked depth in data pipeline design and operational best practices.

Major Gaps Identified

• Backfill design and execution: Most candidates could not explain how to safely and scalably backfill historical data, a common real-world requirement.
• Handling duplicate data and idempotency: Limited understanding of preventing duplicates when pipelines are rerun.
• Data warehouse design: Weaknesses around Iceberg/Delta tables, partitioning strategies, and query optimization.
• Production readiness: Some candidates struggled with practical troubleshooting and operational ownership scenarios.

Overall Candidates generally had solid Spark knowledge but were weaker in real-world data engineering practices such as backfills, idempotent pipelines, warehouse optimization, partitioning, and production support. These areas were the main differentiators between stronger and weaker candidates.