Databricks Data & AI Platform Architect

Job Responsibilities

• Lead the architecture, design, and implementation of enterprise-scale Databricks Data & AI platforms across AWS and Azure environments.
• Define and implement scalable Lakehouse architectures using Databricks, Delta Lake, and modern data engineering patterns for batch and real-time workloads.
• Design high-performance data pipelines and transformation frameworks using Python, SQL, PySpark, Delta Live Tables, and Databricks Workflows/Jobs.
• Architect robust data ingestion, ETL/ELT, CDC, orchestration, and data processing solutions integrating enterprise, cloud, and streaming data sources.
• Establish enterprise-wide data governance, security, access control, and data discovery using Unity Catalog, including cataloging, lineage, permissions, and data-sharing strategies.
• Design and optimize Delta Lake architectures, including schema management, ACID transactions, partitioning, optimization, data lifecycle management, and performance tuning.
• Lead the implementation of ML and AI platforms using Databricks Mosaic AI and MLflow, supporting model development, experiment tracking, evaluation, deployment, and lifecycle management.
• Architect Generative AI solutions using LLMs, RAG architectures, Vector Search, embeddings, and enterprise data, ensuring scalable and production-ready AI applications.
• Design and implement Vector Search and Retrieval-Augmented Generation (RAG) solutions for enterprise knowledge discovery, intelligent search, and AI-powered applications.
• Establish responsible AI practices covering model governance, data security, evaluation, observability, hallucination mitigation, and access controls for enterprise GenAI workloads.
• Integrate Databricks with AWS/Azure cloud services, enterprise applications, databases, APIs, and external data platforms to build end-to-end data and AI solutions.
• Define standards for data modeling, pipeline development, CI/CD, DevOps, testing, monitoring, observability, and production operations across Databricks environments.
• Drive data platform modernization and migration initiatives, including transformation of legacy data warehouses and data lakes into scalable Databricks Lakehouse architectures.
• Optimize compute, storage, Spark workloads, SQL queries, Jobs, and Delta tables to improve performance, scalability, reliability, and cloud cost efficiency.
• Collaborate with data engineers, ML engineers, data scientists, cloud architects, security teams, and business stakeholders to translate business requirements into technical solutions.
• Lead architecture reviews, technical POCs, platform evaluations, and technology strategy initiatives related to Data Engineering, AI/ML, and GenAI.
• Troubleshoot complex data engineering, Spark, Delta Lake, ML, AI, security, and platform performance issues, providing architectural solutions and root-cause analysis.
• Provide technical leadership and mentorship to data engineers, ML engineers, and developers, establishing best practices and ensuring delivery against enterprise architecture standards.

Key Technical Skills

Databricks | Python | SQL | PySpark | AWS | Azure | Delta Lake | Unity Catalog | Databricks Workflows/Jobs | Delta Live Tables | Mosaic AI | MLflow | Vector Search | RAG | LLMs | GenAI | Lakehouse Architecture | Data Engineering | AI/ML | Data Governance | Cloud Data Platforms