Senior AI Data Platform Engineer - Databricks

Job Summary

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

Role

AI Data Platform Engineer - Databricks Experience Guide

The opportunity

Build and operate enterprise-scale Data & AI platforms leveraging Databricks Lakehouse architecture. The role focuses on scalable data engineering, reusable platform engineering, governed self-service analytics, AI-enabled data products, APIs, enterprise service integration, Git-based delivery, Data SRE, data security, Immuta-style governance, and agentic automation using Databricks-native services including Genie, AI/BI, Mosaic AI, Unity Catalog, Vector Search, Lakeflow, and Delta Lake.

Your key responsibilities Databricks Data Engineering

• Design and implement scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Lake, Databricks Workflows, Lakeflow, and Delta Live Tables.
• Develop reusable ingestion frameworks supporting batch, streaming, event-driven, CDC, file-based, and API-based processing patterns.
• Build curated, analytics-ready, and AI-ready data products with strong quality, lineage, semantic context, and operational controls.
• Optimise workloads for performance, cost, cluster/serverless usage, storage layout, and reliability.

Lakehouse & Platform Engineering

• Create reusable platform accelerators for workspace onboarding, pipeline templates, deployment standards, logging, monitoring, and support runbooks.
• Implement Git connectivity, branching strategy, pull requests, code reviews, CI/CD, deployment bundles, and controlled environment promotion.
• Integrate Databricks with enterprise APIs, source systems, orchestration platforms, governance tools, security services, and downstream analytics consumers.
• Contribute to architecture reviews, technical design documentation, release management, and platform operations.

Genie, AI/BI & Agentic Enablement

• Configure and manage Databricks Genie Spaces, AI/BI dashboards, and governed natural-language analytics over trusted data products.
• Enable Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, agentic workflows, semantic retrieval, and AI-ready data products.
• Apply AI-assisted operations for schema drift detection, anomaly detection, pipeline failure diagnosis, automated documentation, and data quality recommendations.

Governance, Security & Data SRE

• Implement Unity Catalog governance including RBAC/ABAC, lineage, audit logging, data masking, privacy controls, policy enforcement, and AI governance.
• Integrate with Immuta, Microsoft Purview, IAM, secrets management, monitoring tools, and enterprise access workflows.
• Build Data SRE capabilities covering observability, incident management, restartability, SLA/SLO tracking, root-cause analysis, FinOps, and production readiness.

Skills and attributes for success

• Core platform - Databricks Lakehouse, Delta Lake, Lakeflow, Delta Live Tables, Databricks Workflows, Databricks SQL, Unity Catalog.
• AI and GenAI - Databricks Genie, AI/BI, Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, Agentic AI, LLMOps, APIs expertise
• Engineering - Python, PySpark, SQL, APIs, Git, CI/CD, unit testing, integration testing, data pipeline testing, GitHub Copilot.
• Cloud and DevOps - AWS, Azure or GCP, Terraform/OpenTofu, Kubernetes, Docker, GitHub Actions, Azure DevOps, Jenkins, policy-as-code, Databricks Asset Bundle
• Governance and reliability - Unity Catalog, Immuta, Purview, lineage, audit, masking, data quality, observability, Data SRE, FinOps, Responsible AI, AI security.

To qualify for the role, you must have

• 5-10 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
• Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
• Preferred certifications aligned to the relevant cloud/platform stack, data engineering, DevOps, security, governance, and AI/ML engineering.

Ideally, you'll also have

• Strong hands-on engineer with architecture awareness, delivery ownership, and a platform engineering mindset.
• Comfortable turning platform standards into reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
• Able to mentor engineers, collaborate with architects/security/SRE teams, and adopt newer AI-native and agentic engineering methods.

What we look for