Big Data DevOps Engineer

Roles and Responsibilities :

• Design, develop, and maintain scalable big data pipelines using Airflow on Kubernetes.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Implement CI/CD processes using Python scripts and Docker containers to ensure efficient deployment of applications.
• Troubleshoot issues related to big data processing, pipeline failures, and infrastructure configuration.

Job Requirements :

• 3-5 years of experience in DevOps engineering with a focus on big data technologies (Airflow, Kubernetes).
• Strong understanding of CICD methodology and ability to implement it using Python scripts and Docker containers.
• Experience working with cloud-based platforms such as Microsoft Azure or similar technologies.
• Proficiency in Terraform for infrastructure provisioning.