Principal Data Engineer

Who you are

• Hands-on experience designing and operating large-scale data pipelines on GCP (BigQuery, Cloud Composer/Airflow, GCS), with proven knowledge of Apache Kafka and real-time streaming architectures
• Demonstrated experience leading or technically mentoring a team of data engineers, with the ability to set standards, conduct code and design reviews, and grow engineers’ capabilities
• Strong grasp of data quality practices: data contracts, pipeline observability, SLA/SLO definition, and automated alerting and remediation
• Solid understanding of SQL transformation patterns and modern tooling such as dbt, alongside experience managing ingestion estates with third-party connectors and in-house integrations
• Clear and confident communicator able to work cross-functionally with non-technical stakeholders; comfortable in a regulated financial services environment with structured delivery disciplines (sprint cadences, change control, escalation)

What the job involves

• IG’s Data function is a central capability serving both central and divisional business lines across the entire IG estate. The Data Engineering team is responsible for the platforms that underpin analytics, reporting, compliance, and client-facing data products, and that will increasingly power AI and machine-learning use cases across the firm
• The current data landscape reflects IG’s growth: a combination of GCP-native capabilities, on-premises systems, and a legacy AWS data platform. The strategic direction — endorsed by the CDO and Executive — is clear: consolidate onto GCP, raise the bar on data quality, and enable self-service access to trusted data for teams across the business
• Own the technical architecture of IG’s GCP data platform end-to-end — Core Lake (BigQuery), batch and real-time (Kafka) ingestion, transformation in the Medallion architecture, and Data-Engineering-owned integration services — setting the reference designs and patterns squads build against
• Lead delivery of the most complex, high-risk work within the GCP consolidation programme, including migration off the legacy AWS platform and on-premises systems onto GCP-native capabilities
• Define the platform contracts, interfaces, and self-service patterns that let mature divisions build and own their own models safely against trusted Core Lake data, and ensure fully-managed builds for less-mature functions follow the identical pattern so they are handover-ready by design
• Drive data quality as a first-class, firm-wide concern: establish data contracts, observability, SLA/SLO monitoring, and automated alerting and remediation across ingestion and transformation layers, and hold squads to those standards
• Act as the senior technical point of contact for stakeholders across Compliance, Marketing, Finance, Data Science, and the self-serving divisions (UK, APAC, tastytrade/US, Quants, Risk), translating requirements into well-scoped, well-architected engineering deliverables
• Set and uphold engineering standards through code and design reviews, technical mentoring, and reusable frameworks that raise the bar across squads in London, India, and Poland — growing the technical capability of the whole function
• Provide technical direction and quality assurance for third-party staff augmentation, ensuring augmented capacity engaged for time-boxed projects works to IG’s patterns and leaves behind maintainable, standards-compliant assets
• Maintain delivery rigour: shape sprint cadences, escalate technical risk early, and report on pipeline SLOs, data freshness, and DORA-style engineering metrics to the Head of Data Engineering and CDO