Senior Data Engineer

Role Overview

Client is seeking a Senior Data Engineer with strong Python and SQL expertise and experience building scalable data platforms, AI/GenAI solutions, and production-grade data workflows.

The ideal candidate will combine hands-on data engineering capabilities with experience in Generative AI, Agentic AI, machine learning, data governance, and cloud-based data infrastructure. The candidate should be capable of leading technical workstreams, mentoring junior engineers, and delivering reliable data and AI solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing workflows.
• Build and optimize data solutions using Python, SQL, and relevant big data technologies.
• Develop and integrate Generative AI, Agentic AI, RAG, embeddings, and vector database solutions into production workflows.
• Implement data ingestion, transformation, orchestration, and processing using suitable frameworks.
• Build reliable AI-enabled data workflows with appropriate evaluation, validation, and guardrails.
• Implement CI/CD, automated testing, monitoring, and MLOps/LLMOps practices.
• Ensure data security, governance, and compliance, including appropriate handling of sensitive data and role-based access controls.
• Work with cloud platforms and infrastructure-as-code tools to deliver scalable data solutions.
• Lead technical workstreams, review code, mentor junior engineers, and promote engineering best practices.
• Collaborate with data scientists, software engineers, architects, and business stakeholders.

Required Skills and Experience

• 7–8 years of professional experience in data engineering or a closely related technical role.
• Expert-level proficiency in Python and SQL.
• Strong experience building and maintaining data pipelines and data processing workflows.
• Hands-on experience with Generative AI, Agentic AI, or ML-enabled data solutions.
• Knowledge of RAG, embeddings, vector databases, prompt engineering, context construction, and LLM integration.
• Experience with technologies such as Spark, Databricks, Dask, Airflow, Dagster, or Kedro.
• Strong understanding of data modeling, data quality, workflow orchestration, and performance optimization.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of MLOps/LLMOps, CI/CD for data workflows, and infrastructure as code, preferably Terraform.
• Understanding of data security, governance, PII/PHI handling, authentication, and RBAC.
• Strong technical ownership, problem-solving, communication, and collaboration skills.

Preferred Qualifications

• Experience with LangSmith, Langfuse, Opik, or similar LLM evaluation tools.
• Working knowledge of Scala or Java in enterprise data environments.
• Experience with AI coding tools such as Cursor, Claude Code, or Codex.
• Experience mentoring engineers and leading technical delivery.

Ideal Candidate Profile

A senior data engineer with 7–8 years of experience, strong Python and SQL skills, and hands-on experience delivering production-grade data and GenAI solutions. The candidate should be technically strong, comfortable owning workstreams independently, and able to collaborate across engineering and data teams.

Important: This is a full-time, permanent position based in London, UK, requiring attendance at the office five days per week.