Data Platform Engineer

Data Platform Engineer – Data Engineering / ML Data / Video Data / Python / GDPR / ASPICE

We are currently partnered with a global technology and engineering organisation developing advanced data platforms and machine learning technologies for large-scale visual and video datasets. As part of their continued investment in machine learning and data infrastructure, they are looking to hire a Data Platform Engineer – Video Data & ML Dataset Management to own the end-to-end platform for managing a large and growing collection of video data used for machine learning.

Key responsibilities

• Design and build automated ingestion pipelines for large video and media files
• Implement validation, metadata extraction, deduplication, registration, and quality checks during ingestion
• Design and maintain a robust relational metadata catalogue covering videos, people, recording campaigns, cameras, vehicles, and dataset usage
• Ensure strong data integrity through appropriate database relationships, constraints, validation, and lifecycle management
• Implement GDPR and consent-management capabilities for video datasets
• Link individuals to recorded video data and track consent status, expiry, retention, and relevant ownership information
• Automate notifications and workflows relating to consent and compliance requirements
• Maintain appropriate audit trails and traceability for sensitive data
• Establish dataset versioning and reproducible train/test/validation tracking linked to ML experiments

Key requirements

• Strong professional experience in data engineering and building production-quality data pipelines
• Experience with data ingestion, ETL, workflow orchestration, and automation
• Strong relational database design and data modelling skills
• Understanding of database constraints, relationships, data integrity, and lifecycle management
• Strong Python programming skills for data pipelines, automation, and API development
• Experience handling large media or binary datasets
• Experience with containerisation and secure deployment practices
• Experience with dataset versioning and reproducibility for machine-learning workflows
• Experience with data cataloguing and metadata-management platforms
• Experience integrating low-code platforms or internal tooling for data entry, workflow management, and visualisation
• Practical experience with video and media processing

Keywords

Data Platform Engineer / Data Engineer / Data Engineering / Data Platform / Data Pipelines / Data Ingestion / ETL / Workflow Orchestration / Python / Relational Databases / Database Design / Data Modelling / Data Integrity / Metadata Management / Data Cataloguing / Object Storage / File Storage / Large Media Data / Video Data / Video Processing / Machine Learning Data / ML Datasets / Dataset Management / Dataset Versioning / Dataset Reproducibility / ML Experiment Tracking / Visual Data / Data Governance / GDPR / Data Protection / PII / Consent Management / Data Retention / Audit Trails / Compliance / Data Dashboards / Reporting / APIs / API Development / Secure Deployment / Containerisation / Infrastructure Automation / ASPICE / ISO / Codebeamer / ALM / Requirements Management / Traceability / Automotive Data / Vehicle Testing / In-Car Testing / Data Collection / Data Infrastructure / ML Infrastructure / Data Lifecycle / Data Management / Data Automation / Data Platform Architecture / Data Operations / Engineering Tools

By applying to this role, you understand that we may collect your personal data and process it in line with our privacy policy.