Apache Airflow Engineer

Job Summary

• We are seeking Apache Airflow migration resources to support a client initiative focused on migrating existing Cybermation-scheduled scripts to Apache Airflow. The resources will analyze the current Cybermation schedules, understand the existing shell/API invocation patterns, convert the schedules into Airflow DAGs, validate execution behavior, and support production deployment and handover.
• The migration objective is to preserve the existing business execution behavior while improving schedule control, monitoring, logging, retry handling, operational visibility, and supportability through Apache Airflow.

Key Responsibilities

• Analyze existing Cybermation job schedules, dependency patterns, runtime parameters, execution frequency, owners, success/failure behavior, and operational support expectations.
• Review existing shell scripts and API/cURL invocation logic to determine the appropriate Airflow DAG design, task structure, error handling, and retry configuration.
• Design, develop, and maintain Apache Airflow DAGs to orchestrate shell scripts, API calls, and dependent jobs with appropriate scheduling and operational controls.
• Configure Airflow schedules, task dependencies, retries, logging, alerting hooks, variables, connections, parameters, and environment-specific configuration as per client standards.
• Perform dry runs, unit testing, schedule validation, API response validation, failure/retry validation, and production-readiness checks for migrated jobs.
• Create migration mapping, validation evidence, deployment notes, operational runbooks, and support handover documentation.
• Coordinate with client SMEs, application owners, infrastructure teams, and release teams to support deployment, issue triage, initial run monitoring, and hypercare.

Preferred Qualifications

• Prior experience migrating jobs from Cybermation or equivalent enterprise schedulers to Apache Airflow.
• Experience deploying or operating Airflow in cloud or containerized environments is preferred.
• Exposure to AWS services such as EKS, S3, CloudWatch, Lambda, Step Functions, IAM, Secrets Manager, or EventBridge is a plus.
• Experience supporting financial services, capital markets, or other regulated enterprise environments is preferred.
• Familiarity with ITSM processes, release governance, change management, and production support procedures.