TechOps-CloudOps-Airflow - Engineer

The Opportunity

We are seeking a highly skilled Senior Apache Airflow Platform Engineer to provide L2 operational support, platform administration, automation, and continuous improvement for enterprise workflow orchestration environments. The ideal candidate will possess strong expertise in Airflow DAG development, platform operations, DevOps automation, cloud-native infrastructure, and CI/CD implementation.

Your Key Responsibilities

This role will be responsible for ensuring the reliability, scalability, security, and performance of the Airflow platform while partnering closely with Data Engineering, Cloud Infrastructure, DevOps, and Application teams.

Airflow Platform Operations

• Provide L2/L3 support for Apache Airflow production environments.
• Monitor, troubleshoot, and resolve issues related to DAG execution, task failures, scheduling delays, and workflow dependencies.
• Support Airflow platform administration, configuration management, upgrades, and patching activities.
• Manage Airflow components including Schedulers, Webservers, Workers, Executors, Metadata Database, Variables, Connections, and Plugins.
• Perform root cause analysis for recurring workflow and platform incidents.
• Develop and maintain operational runbooks, SOPs, troubleshooting guides, and knowledge articles.
• Ensure platform availability, reliability, and performance through proactive monitoring and capacity planning.

Airflow Development Orchestration

• Design, develop, and maintain Airflow DAGs for complex data and application workflows.
• Optimize DAG performance, scheduling efficiency, and resource utilization.
• Implement best practices for workflow orchestration, dependency management, and fault tolerance.
• Support onboarding of new workflows and integrations into the orchestration platform.
• Review DAG code and enforce development standards across engineering teams.

DevOps CI/CD Engineering

• Design, implement, and maintain CI/CD pipelines using GitHub Actions.
• Automate deployment, testing, validation, and release management processes for Airflow environments.
• Implement code quality controls, automated testing, security scanning, and deployment governance.
• Support GitOps and Infrastructure-as-Code practices across platform deployments.
• Collaborate with development teams to streamline release and deployment processes.

Cloud Infrastructure Automation

• Support and administer Airflow deployments on AWS EKS.
• Manage Kubernetes-based workloads and ensure application scalability and resilience.
• Develop and maintain Helm charts for deployment and configuration management.
• Build and manage Infrastructure-as-Code solutions using Terraform.
• Support containerized environments leveraging Docker and Kubernetes best practices.
• Implement automation for environment provisioning, monitoring, recovery, and operational maintenance.

Database Administration Support

• Manage and support PostgreSQL databases used by Airflow and associated platforms.
• Monitor database health, performance, storage, and availability.
• Troubleshot metadata database issues affecting Airflow operations.
• Support backup validation, recovery procedures, and database maintenance activities.
• Optimize database performance through query tuning and configuration improvements.

Skills and attributes for success

• 3-5 years of experience supporting, administering, and engineering Apache Airflow platforms in enterprise environments.
• Good experience in Airflow DAG development, workflow orchestration, scheduling, dependency management, and production platform support.
• Deep understanding of Airflow architecture, including Schedulers, Webservers, Workers, Executors, Metadata Database, Plugins, Variables, Connections, and deployment frameworks.
• Experience troubleshooting DAG failures, task execution issues, scheduler bottlenecks, performance degradation, dependency conflicts, and platform stability issues.
• Hands-on experience designing, developing, and maintaining CI/CD pipelines using GitHub Actions, including automated testing, code validation, deployment automation, release management, and governance.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.