AVP, Data Platform Development, LME

Location: CN-Shenzhen-HyQ

Shift: Standard - 40 Hours (China)

Scheduled Weekly Hours: 40

Worker Type: Permanent

Job Summary: Support the development, enhancement, and day‑to‑day operation of LME technology systems, including trading, market data, and supporting platforms. This entry‑level engineering role focuses on delivering high‑quality software, ensuring system stability, and collaborating with cross‑functional teams across LME Technology.

Job Duties: Role Summary

Design, build, and operate critical subsystems of the data platform. Take ownership of major components — streaming, batch, storage, or serving — drive them from design to production, and continuously improve their reliability and performance.

Responsibilities

• Streaming Infrastructure: Kafka cluster operations (broker tuning, partition rebalancing, monitoring, disaster recovery); manage schema registry and Kafka Connect connectors
• Batch & Streaming Compute: Build and optimize Spark batch jobs and Flink/Spark Structured Streaming pipelines; contribute reusable job frameworks and tuning guides
• Storage & Lakehouse: Manage Iceberg tables (compaction, snapshot expiration, orphan file cleanup, schema evolution); operate MinIO at scale (lifecycle rules, tiering, performance tuning)
• Query & Serving: Deploy and operate Trino clusters (connector config, resource groups, query monitoring); manage StarRocks/ClickHouse clusters (sharding, replication, materialized views)
• Orchestration: Build and maintain Airflow/Dagster DAGs for platform operations; extend custom operators and sensors
• Platform Observability: Implement monitoring with OpenTelemetry, Prometheus, Grafana, and Loki across all stack layers; build dashboards and alerting rules
• Kubernetes Operations: Write Helm charts, manage operator lifecycles, configure resource quotas, node affinity, pod disruption budgets
• CI/CD & GitOps: Own ArgoCD application sets, Helm-based deployments, and promotion pipelines from dev to prod
• Participate in on-call rotation; write post-mortems and runbooks
• Mentor mid-level engineers through pairing, design discussions, and code reviews

Required Skills & Experience

• 6+ years in data/platform engineering or backend infrastructure
• Strong Kubernetes: Helm chart authoring, RBAC, network policies, storage (PV/PVC), operators
• Solid Kafka: topic design, consumer group management, offset management, monitoring lag, Kafka Connect, schema registry (Apicurio or Confluent)
• Solid Spark: Dataframe/Dataset API, Spark SQL, performance tuning, troubleshooting in production
• Working knowledge of Flink or Spark Structured Streaming for real-time pipelines
• Practical Iceberg experience: table maintenance, time-travel, catalog integration
• Hands-on Trino or Presto: connector configuration, query tuning, resource group management
• Experience with an OLAP engine: StarRocks, ClickHouse, or Doris — table design, ingestion pipelines, query optimization
• Proficient in Python and either Scala or Java
• Solid CI/CD and GitOps: ArgoCD or Flux, Helm, Docker
• Airflow or Dagster for pipeline orchestration

Nice to Have

• OpenShift-specific: SCC, Routes, ImageStreams, BuildConfigs
• dbt project experience for data transformation and modeling
• Data quality frameworks: Great Expectations, Soda, or Deequ
• Kafka Streams or ksqlDB for stream processing
• Exposure to DataHub/Atlas for data discovery and lineage

Company Introduction: ITD SZ

港交所科技（深圳）有限公司，是2016年12月28日于深圳市前海自贸区成立的外商独资企业。

作为港交所的技术子公司，港交所科技（深圳）有限公司主要是为集团及其附属公司提供计算机软件、计算机硬件、信息系统、云存储、云计算、物联网和计算机网络的开发、技术服务、技术咨询、技术转让；经济信息咨询、企业管理咨询、商务信息咨询、商业信息咨询、信息系统设计、集成、运行维护；数据库管理、大数据分析；以承接服务外包方式提供系统应用管理和维护、信息技术支持管理、数据处理等信息技术和业务流程外包服务。