Lead Data Engineer - AWS/Azure Databricks

Lead Data Engineer Experience : 5+ Years

Duration : 12 Months

Work Mode : Hybrid

Role Overview We are looking for a hands-on Lead Data Engineer with strong technical leadership and expertise in designing and building high-volume, production-grade data platforms. The ideal candidate will have extensive experience with modern cloud data technologies, data lakes, data warehouses, ETL/ELT pipelines, real-time data processing, and data architecture. The role requires someone who can combine strong engineering expertise with technical leadership to architect scalable data solutions and guide a technical team.

Key Responsibilities

• Lead the design, development, and implementation of scalable data engineering solutions.
• Architect and build high-volume data lakes and data warehouses using modern cloud technologies.
• Design and optimize ETL/ELT pipelines supporting batch and real-time/streaming workloads.
• Implement Change Data Capture (CDC) pipelines and data integration solutions.
• Work with Databricks, Snowflake, AWS, and Azure to develop enterprise-grade data platforms.
• Develop robust data pipelines and integrations using Python and advanced SQL.
• Build and manage workflows using Apache Airflow and Airbyte.
• Design and implement integrations with REST APIs and third-party APIs.
• Establish and maintain Git-based CI/CD pipelines, preferably using GitLab.
• Containerize applications and data workloads using Docker and Kubernetes.
• Define and implement effective data architecture and data modeling strategies.
• Collaborate with engineering, analytics, data science, and business teams to translate requirements into scalable data solutions.
• Provide technical leadership, conduct code reviews, and mentor data engineers.
• Apply data engineering best practices for performance, scalability, security, reliability, and maintainability.
• Contribute to MLOps initiatives and support data infrastructure for machine learning workloads.
• Leverage AI coding assistants to improve engineering productivity, code quality, and development efficiency.

Must-Have Skills & Experience

• 5+ years of experience in Data Engineering with strong technical leadership capabilities.
• Strong expertise in Databricks, Snowflake, AWS, and/or Azure.
• Hands-on experience building large-scale Data Lakes and Data Warehouses.
• Strong understanding of ETL/ELT, CDC, batch processing, and real-time/streaming pipelines.
• Advanced proficiency in SQL and Python.
• Hands-on experience with Apache Airflow and Airbyte.
• Experience developing REST API and third-party API integrations.
• Strong understanding of Git-based CI/CD, preferably GitLab.
• Experience with Docker and Kubernetes.
• Strong knowledge of Data Architecture and Data Modeling.
• Understanding of MLOps concepts and practices.
• Experience using AI Coding Assistants to enhance software/data engineering productivity.
• Strong problem-solving, communication, and technical leadership skills.

Good To Have

• Experience working in retail or other large-scale data environments.
• Hands-on experience with Kafka and Spark Streaming.
• Knowledge of Customer Data Platforms (CDP).
• Experience with Agentic AI or LLM-based data pipelines.
• Exposure to Generative AI, RAG, and Vector Databases.
• Experience with observability and monitoring tools such as Splunk, Datadog, or Dynatrace.

Ideal Candidate The ideal candidate is a hands-on Lead Data Engineer who can architect and build high-volume, production-grade data platforms while providing strong technical leadership. The candidate should be comfortable working across cloud, data engineering, DevOps, and emerging AI technologies and should be able to mentor engineers while driving technical excellence.

(ref:hirist.tech)