Azure Databricks & Agentic AI Architect

Job Title: Azure Databricks & Agentic AI Architect

Location: Chicago, IL (Hybrid- 3Days a week)

Duration: 12+ Months

Job Description:

Role Summary:

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

• Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.

• Define autonomous workflows using AI Agents for:

• Data ingestion

• Data mapping

• Schema evolution

• Data quality remediation

• Metadata Enrichment

• Pipeline optimization

• Root cause analysis

• Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

• Lead architecture for AI-enabled Data Development Lifecycle across:

• Requirement analysis

• Data modeling

• Pipeline generation

• Automated testing

• Code review

• Documentation

• Deployment

• Monitoring

• Implement AI copilots to accelerate developer productivity.

• Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

• Design scalable Lakehouse platforms using:

• Azure Databricks

• Delta Lake

• Unity Catalog

• ADLS Gen2

• Databricks Workflows

• Delta Live Tables

Enterprise GenAI Integration

• Architect RAG-based solutions using enterprise data assets.

• Design agent orchestration frameworks using:

• Azure OpenAI

• LangGraph

• Semantic Kernel

• AutoGen

• MCP-enabled architectures

• Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

• Define guardrails for enterprise GenAI adoption.

• Implement:

• Prompt governance

• Observability

• Cost monitoring

• Auditability

• Explainability

• Security controls

• Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

• Design self-healing data pipelines.

• Implement AI-driven:

• Incident triage

• Failure prediction

• Capacity planning

• Cost optimization

• SLA monitoring

• Enable intelligent workload placement and model routing.

Technical Skills

• Data Platform

• Azure Databricks

• Delta Lake

• Unity Catalog

• Azure Data Factory

AI & Agentic Frameworks

• Azure OpenAI

• Knowledge Graph

• RAG Architecture

• LangChain

• LangGraph

• MCP Protocol

• Vector Databases

• AI Agent Orchestration

Data Engineering

• PySpark

• Spark SQL

• Python

• SQL

• ELT/ETL Modernization

DevOps & AI-DLC

• Azure DevOps

• GitHub Actions

• CI/CD

• MLOps

• LLMOps

• Evaluation Frameworks

• AI Testing Frameworks

Leadership Expectations

• Drive AI-First Data Engineering transformation.

• Define enterprise patterns, accelerators, and reusable AI agents.

• Mentor architects, data engineers, and AI engineers.

• Lead executive conversations on AI adoption, ROI, and transformation roadmaps.

Preferred Certifications

• Databricks Certified Data Engineer Professional

• Databricks Certified Solution Architect

• Microsoft Azure Solution Architect (AZ-305)

• Azure Data Engineer (DP-203)

• Microsoft Applied Skills - Azure OpenAI

• Generative AI / Agentic AI Certifications

Success Metrics

• 30-50% Data Engineering productivity improvement.

• Reduction in manual pipeline development effort.

• Improved data quality and governance compliance.

• Measurable ROI from Agentic AI adoption.

• Expansion of reusable AI agents across programs.