Lead Data Engineer

Key Responsibilities

• Lead the development of scalable ETL/ELT pipelines and data models to process high-volume data.
• Design and optimize data processing applications across platforms like Databricks, Snowflake, AWS, and Azure.
• Build and manage data pipelines using Apache Airflow for orchestration and Airbyte for data integration.
• Architect solutions for Change Data Capture, batch processing, and real-time/streaming data processing.
• Develop and manage REST APIs for data ingestion and distribution to applications and users.
• Leverage AI tools to enhance the development and deployment of data pipelines.
• Oversee CI/CD pipelines and DevOps deployments using Docker, Kubernetes, and monitoring tools.
• Mentor team members in best practices for data governance, quality, and security.

Overview

• As a Lead Data Engineer at our consulting firm, you will play a pivotal role in shaping the data architecture and engineering landscape.
• This position requires an expert with multi-platform proficiency, particularly in Databricks, Snowflake, AWS, and Azure, to lead scalable data solutions.
• You will work closely with cross-functional teams to integrate MLOps/DevOps practices and guide a team in delivering production-ready solutions.
• The role demands innovation in solving complex data challenges using a diverse toolkit and requires strong leadership skills to mentor and develop team members.
• Your expertise will help build robust, high-volume data processing pipelines and drive the adoption of AI-augmented deliverables.
• The position is based in Bangalore and Gurgaon, catering to high-profile clients in the consulting sector.
• You will be responsible for ensuring data governance, quality, and security while staying ahead of industry trends in MLOps and GenAI.
• This role offers an opportunity to collaborate with data scientists and analysts to ensure data readiness for machine learning projects and contribute to the evolution of CI/CD pipelines.

Requirements

• 5+ years of experience in Data Engineering with expertise in data lakes and warehouses.
• Proficiency in Change Data Capture (CDC), batch, and real-time data processing.
• Experience with multiple cloud data platforms such as Databricks, Snowflake, AWS, or Azure.
• Advanced SQL skills for large-scale data processing and analytics.
• Strong programming skills in Python, with experience in asynchronous programming and threading.
• Experience in creating and managing REST APIs using frameworks like FastAPI or Flask.
• Hands-on experience with Apache Airflow for workflow orchestration and Airbyte for data integration.