Staff Data Engineer (Data Platform, Lakehouse & Streaming)

• If you want to own the data infrastructure that powers real-time pricing decisions for thousands of hotels worldwide — and you’re energised by the idea of doing it in an engineering culture where AI isn’t a future ambition but how work actually gets done today — this is the role
• You’ll be the technical authority on Duetto’s data lakehouse, driving everything from pipeline architecture and data quality to the shift from batch to near-real-time streaming
• You’ll own the design, performance, and reliability of Duetto’s data lakehouse — evolving the Python/PySpark pipeline framework across a bronze → silver → gold architecture on AWS, including Glue jobs, Iceberg MERGE operations, schema evolution, and partitioning strategies
• You’ll architect the shift from batch to near-real-time streaming, building SQS-driven stream pipelines with Iceberg sinks and expanding ingestion, normalisation, and analytics layers across the full lakehouse
• You’ll drive data quality and governance at scale — extending the Great Expectations framework, leading adoption of data contracts to formalise schemas between producers and consumers, and owning the Athena SQL layer that analysts and product teams depend on
• You’ll strengthen observability and reliability through Datadog, Sentry, and Sumo Logic, while optimising Glue job performance — worker sizing, DPU allocation, Spark tuning, and cost management
• You’ll build and maintain shared internal Python libraries published to JFrog, and drive improvements to GitHub Actions, Docker-based testing, and CI/CD deployment workflows
• You’ll work AI-first every day — using Claude Code and MCP tools in your regular workflow, and contributing to AI-assisted pipeline generation, schema inference, and automated data quality alongside a custom multi-agent system with 17 specialised agents
• Team: Data Platform
• Direct reports: None

Benefits

• Work from home opportunities
• Health insurance- Working knowledge of Java for reading upstream systems
• Familiarity with Great Expectations or similar data quality frameworks
• Deep expertise in PySpark and distributed data processing — Glue, EMR, or Databricks
• Experience with dbt for data transformation and modelling
• A track record of improving data quality, governance, and pipeline reliability at scale
• Strong experience with lakehouse architectures: Iceberg, Delta Lake, or Hudi on S3
• Genuine interest in AI-assisted development and LLM-based tooling
• You don’t need to tick every box — if you’re a strong Python data engineer who’s excited by lakehouse architecture, streaming pipelines, and working in a genuinely AI-first engineering culture, we’d love to hear from you
• 7+ years building production data systems in Python
• Experience with Trino or Presto for interactive SQL analytics at scale
• Production experience with Airflow or a comparable workflow orchestrator
• Solid AWS production experience across S3, Glue, Athena, Lambda, and SQS
• Familiarity with hospitality data — reservations, rates, inventory, demand signals