Databricks Data Engineer (AI & Governance)

I'm working with a client who is scaling their Databricks platform to support AI and GenAI use cases, and they need people who can make sure it's governed properly from the start. This is about hands-on Databricks delivery, not just certifications. This is a remote first contract but you must be based in the UK. This is outside IR35.

What they're building

• Unity Catalog as the single governance layer across data, ML models and AI assets
• Governed GenAI and agent workloads using Mosaic AI and Agent Bricks
• Secure, auditable AI pipelines from ingestion to model serving, all within the Databricks platform

Databricks project experience we're looking for

• Unity Catalog implementations or migrations from Hive metastore, including metastore design, catalog and schema structure, and workspace and environment separation
• Fine-grained access control in Unity Catalog: row filters, column masks, and attribute-based access control (ABAC) with governed tags
• Unity Catalog lineage and system tables used for auditing, usage monitoring and compliance reporting
• Data classification and sensitive data tagging in Unity Catalog for PII and regulated datasets
• Mosaic AI Model Serving and AI Gateway, including access controls, rate limiting, guardrails and usage tracking
• Models in Unity Catalog via MLflow: registry, versioning, lineage and permissions, plus MLflow tracing and evaluation for GenAI applications
• Agent Bricks or the Mosaic AI Agent Framework in production, with governance over the data and tools agents can access
• Databricks Lakehouse Monitoring and data quality expectations in Lakeflow Spark Declarative Pipelines supporting AI reliability
• Delta Sharing or Databricks Clean Rooms for governed data collaboration
• Databricks Asset Bundles and CI/CD with governance policies built into deployment

Ideal background

• Hands-on Databricks engineering or architecture experience on Azure, AWS or GCP
• Delivery in regulated or risk-sensitive environments such as financial services, public sector or healthcare
• Able to explain what you built in Databricks, the governance decisions you made and why

Why this one

• Work on live AI projects with real governance requirements, not proofs of concept
• Exposure to the latest Databricks capabilities in production
• A client who treats governance as a core engineering discipline

If you've delivered Unity Catalog or AI governance work on Databricks, or know someone who has, apply now!