AI AWS Data Engineer

Join an enterprise-grade AI platform programme focused on securely and reliably bringing source-system data into a central platform for Generative AI and ML capabilities. This is an integration-focused Data Engineering role, building the connectors and ingestion pipelines that keep data flowing reliably across the CDS estate.

You’ll develop event-driven ingestion patterns and onboard new source systems, working with APIs, webhooks and enterprise platforms. The role combines Python development with AWS services and data migration, with a strong focus on reliability, reconciliation, monitoring and error handling.

Key responsibilities:

• Build source-system integrations using webhooks, REST APIs and event streams.
• Implement event-driven ingestion using API Gateway, Lambda, EventBridge and SQS.
• Develop DynamoDB-based state and watermark tracking with periodic reconciliation.
• Onboard new source systems such as Confluence and JIRA.
• Support data migration and backfilling activities using tools such as AWS DMS.
• Work with Knowledge Engineers to ensure data quality and structure for downstream use.
• Monitor integrations and implement effective error handling and alerting.

Key Skills:

• Python
• AWS
• API Gateway / Lambda
• EventBridge / SQS
• DynamoDB
• REST APIs and webhooks
• AWS DMS
• Event-driven data engineering
• Data migration

Experience with Apache Airflow, dbt, enterprise collaboration APIs and regulated environments would also be beneficial.

For more information or to apply directly, please email [email protected]. I review applications daily and will contact shortlisted candidates as quickly as possible.