Big Data Lead

Job Title: Big Data Lead

Experience Level: 8+ Years

Location: Pune, India

Job Summary

We are looking for a Big Data Lead to design, scale, and modernize our high-throughput data platform. You will direct technical strategy, manage data pipelines, and bridge traditional big data architectures with modern Lakehouse patterns and Generative AI applications.

Key Responsibilities

• Platform Lakehouse Architecture: Drive the architecture and migration from legacy Hadoop/Hive systems to modern Apache Iceberg formats for ACID compliance, schema evolution, and time-travel querying.
• Query Compute Optimization: Build and tune high-performance batch and streaming pipelines using Apache Spark. Deploy Trino for sub-second, federated interactive SQL querying across heterogeneous storage layers.
• Orchestration DevOps: Manage enterprise-wide DAG workflows using Apache Airflow. Deploy, auto-scale, and manage compute workloads on Kubernetes (K8s) clusters.
• GenAI Integrations: Architect ingestion and processing pipelines for unstructured data, handling vector embeddings and Retrieval-Augmented Generation (RAG) infrastructure.
• Team Leadership Governance: Lead engineering teams through code reviews, technical design sessions, platform security enforcement, cost optimization, and SLA management.

Technical Skills Qualifications

Category Requirements Big Data Core Apache Spark (PySpark/Scala), Apache Hive, Hadoop/HDFS Lakehouse Querying Apache Iceberg, Trino (Presto), Complex SQL Orchestration Infra Apache Airflow, Kubernetes (K8s), Docker GenAI Ecosystem Vector databases (Milvus/Pinecone/Qdrant), RAG ingestion pipelines, LLM data processing Programming Python, Scala, Java

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.