Databricks AI Architect

Qualification:

The ideal candidate is a hands-on AI architect with deep Databricks and GenAI expertise, strong data engineering foundations, and a proven record of taking AI solutions to production at enterprise scale. Key requirements:

• 12–14 years of experience in data and AI/ML solutions, including designing and architecting large-scale (TB/PB) data platforms and production AI/ML systems.

• Deep, hands-on architecture expertise with Databricks — Spark/PySpark, Delta Lake, lakehouse/medallion architecture, Unity Catalog, Databricks Workflows, and Delta Live Tables — with strong performance and cost optimization.

• Proven experience architecting Generative AI / LLM solutions — RAG pipelines, chunking and embedding strategies, vector store design, prompt engineering, evaluation, and guardrails/responsible-AI patterns.

• Strong hands-on experience with native Databricks AI capabilities — Mosaic AI (Model Serving, Vector Search, Agent Framework), Foundation Model APIs, MLflow, Feature Store, and model governance via Unity Catalog.

• Solid ML/MLOps foundations — model lifecycle management, experiment tracking, deployment, monitoring, and drift/quality evaluation in production.

• Expert-level PySpark and Spark SQL, with strong Python for building reusable data and feature pipelines that power AI workloads (batch and streaming).

• Strong hands-on experience with AWS and Azure Cloud, including networking, IAM/security, cost optimization, and AI/ML services; comfortable architecting across cloud data and AI services.

• Experience with LLM orchestration frameworks (LangChain/LangGraph or equivalent), agentic patterns, tool use, and multi-agent workflows at an architecture level.

• Deep expertise in data and AI governance, data quality, lineage, security, and responsible AI — including PII handling, access control (Unity Catalog, RBAC), and audit.

• Proven ability to set architecture standards, lead design reviews, and provide technical leadership and mentorship across multiple engineering and data science teams.

• Strong communication and stakeholder-management skills; able to drive architecture decisions and translate business needs into technical blueprints for both technical and executive audiences.

Good to Have: Knowledge of LLMOps and observability tooling for AI systems; Databricks / cloud / AI certifications; experience leading cloud/data platform migration and modernization to Databricks; experience contributing to POCs, proposals, and RFPs.

Skills Required:

Databricks, AI

Role:

AI & GenAI Architecture

• Own the end-to-end architecture for AI/ML and GenAI solutions on Databricks — data and feature pipelines, model training/serving, RAG, and agentic applications.

• Design RAG and LLM architectures — retrieval and embedding pipelines, vector store design, prompt/evaluation strategy, and guardrails — using Mosaic AI and Foundation Model APIs.

• Define reference architectures, design patterns, and reusable frameworks for scalable, secure, and cost-optimized AI platforms.

Data & Platform Foundation

• Architect the lakehouse foundation (Delta Lake, medallion, Unity Catalog) and reusable PySpark pipelines that feed AI and analytics workloads.

• Guide implementation using Databricks Workflows and Delta Live Tables, with standards for data quality, incremental loads, and schema evolution.

MLOps, Governance & Reliability

• Define MLOps/LLMOps practices — model lifecycle, MLflow tracking, deployment, monitoring, and drift/quality evaluation in production.

• Establish AI and data governance, cataloging, lineage, access control, and responsible-AI standards (PII handling, guardrails, audit) ensuring security and compliance.

• Define observability, monitoring, and cost-governance practices for reliable and efficient AI systems at scale.

Technical Leadership & Collaboration

• Set and enforce architecture standards and best practices; lead design and code reviews across multiple engineering and data science teams.

• Mentor engineers, ML engineers, and leads, and provide hands-on guidance from prototype to production readiness.

• Collaborate with product owners, engineering leaders, and business stakeholders to translate requirements into AI blueprints, POCs, and roadmaps.

Experience:

10 to 14 years

Job Reference Number:

14092