SDE IV (ML Engineering)

Job Summary

SDE III/IV - Data Platform Engineering

Mission

Architect a self-serve Data "Platform-as-a-Product" powering InMobis global-scale data ecosystem. Integrate OSS tools, proprietary services, and Cloud/SaaS into unified infrastructure. Requires deep data engineering expertise (batch/streaming pipelines, data modeling, query optimization, governance) combined with platform engineering to build production-grade solutions for data engineers and analysts.

Core Responsibilities

• Design & Development - Bridge OSS tools (Spark, Flink, Airflow, Iceberg), internal services, and cloud offerings into cohesive data platform infrastructure. Build intuitive platform integrations enabling push-button data workflows.

• Scale Engineering - Operate distributed systems processing petabytes of data daily. Own multi-region Kubernetes infrastructure with elastic scalability and fault tolerance.

• Performance Optimization - Optimize compute utilization (Spark/Flink clusters, Velox/Gluten acceleration) for large-scale batch and real-time streaming with sub-second latency.

• Observability & Data Quality - Build comprehensive telemetry (metrics, logs, traces) and data quality frameworks for 24/7 uptime. Enforce SLAs/SLOs with automated incident response and data validation.

  Required Skills & Experience

Must-Have

• 7-10 years building, optimizing, and operating production data platforms
• Deep data engineering fundamentals: data modeling, partitioning strategies, query optimization
• Distributed compute: Spark (PySpark/Scala), Flink streaming, performance tuning at petabyte scale
• Data lake architecture: Iceberg table format, Polaris catalog, schema evolution, time travel
• Orchestration: Airflow DAG development, dependency management, SLA monitoring
• Data transformation: DBT modeling, testing, documentation, incremental builds
• Data quality: Great Expectations, dqueue validation frameworks, drift detection
• Query acceleration: Velox, Gluten integration, columnar formats (Parquet, ORC)
• Data governance: OpenMetadata catalog, lineage tracking, access control
• Kubernetes platform development: operators (Spark/Flink), Yunikorn scheduler, multi-tenancy, autoscaling
• Cloud infrastructure: GKE multi-region clusters, GCS object storage, hybrid cloud/on-prem architecture
• Programming: Python, PySpark, Scala for data pipelines and platform tooling
• IaC: Terraform, Helm, GitOps for reproducible deployments

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.