Senior Python Airflow Developer

Experience: 5+ Years
Role: Senior Python Airflow Developer
Employment Type: Full-Time

Job Summary

We are looking for a Senior Python Airflow Developer with strong experience in Python development, Apache Airflow, and data pipeline engineering. The candidate will be responsible for designing, developing, maintaining, and optimizing Airflow workflows and Python-based ETL/ELT pipelines.

The role involves workflow orchestration, data integration, pipeline monitoring, performance optimization, and CI/CD support while working closely with cross-functional teams in an Agile environment.

Key Responsibilities

• Design, develop, and maintain Apache Airflow DAGs for data workflows.
• Develop clean, modular, reusable, and testable Python code.
• Build and maintain scalable ETL/ELT data pipelines.
• Implement workflow scheduling, dependencies, retries, alerts, and error handling.
• Integrate data from multiple sources including databases, APIs, files, Kafka, and MQ.
• Monitor Airflow workflows and troubleshoot failed or delayed pipelines.
• Perform root-cause analysis and resolve production issues.
• Optimize DAGs and Python pipelines for better performance and reliability.
• Implement data validation and quality checks within pipelines.
• Work with Git and CI/CD pipelines for code versioning and deployment.
• Collaborate with Data Engineers, Developers, Architects, and Business teams.
• Participate in Agile ceremonies including sprint planning, daily stand-ups, and retrospectives.
• Maintain technical documentation for workflows, pipelines, and integrations.

Must-Have Skills

• 5+ years of experience in Python/Data Engineering.
• Strong hands-on experience with Apache Airflow.
• Proven experience developing and maintaining Airflow DAGs.
• Strong understanding of workflow orchestration, scheduling, task dependencies, and monitoring.
• Strong Python programming skills.
• Experience with ETL/ELT pipelines and data integration.
• Good knowledge of SQL and relational databases.
• Experience integrating APIs, databases, files, Kafka, or MQ.
• Hands-on experience with Git and CI/CD practices.
• Strong debugging, troubleshooting, and performance-optimization skills.

Good to Have

• Experience with AWS, Azure, or GCP.
• Knowledge of Docker/Kubernetes.
• Experience with cloud-based Airflow platforms such as AWS MWAA or Google Cloud Composer.
• Experience with data warehouses such as Snowflake, BigQuery, Redshift, or Databricks.
• Knowledge of PySpark.
• Experience with monitoring and logging tools.
• Experience working in large-scale enterprise data environments.