Lead Data Platform Engineer

Role- Lead Data Platform Engineer

Experience- 8 to 15 years

Location- Pune, Hyderabad, Bangalore

Required Skill set- Spark, Kafka, Flink, Airflow, Iceberg, Polaris/Nessie, Kubernetes

Required Skills & Experience

Must Have

• 8-15 years of data engineering or data platform experience, with substantial recent hands-on responsibility for distributed processing platforms.
• Deep hands-on experience with Apache Spark, including DataFrame or SQL workloads, performance tuning, packaging, configuration, and production troubleshooting.
• Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
• Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
• Strong understanding of Apache Iceberg, Google Cloud Storage, schema and partition evolution, snapshots, compaction, and table maintenance.
• Experience with Apache Polaris, Project Nessie, or REST-based Iceberg catalogs.
• Hands-on experience with Apache Airflow for distributed data-processing orchestration.
• Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
• Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
• Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.

Good to Have

• Experience with Trino, OpenMetadata, JupyterHub, Dataproc, data-quality frameworks, lineage, or schema registries.
• Exposure to Debezium, Kafka Connect, Datastream, database logs, or comparable CDC patterns.
• Experience with Spark Operator, Flink Kubernetes Operator, Strimzi, KubernetesExecutor, or operator-based GKE integration.
• Understanding of data security, encryption, masking, governance, data contracts, and regulated enterprise controls.

Experience with multi-tenant data platforms, regulated environments, and platform cost optimisation on GCP.