Databricks Architect

Role Overview

Serves as the core technical authority for designing, modernizing, and optimizing the Databricks Lakehouse architecture, code refactoring, and data execution pipelines.

Roles & Responsibilities

• Refactor legacy Synapse T-SQL stored procedures, scripts, and views into native PySpark, Databricks SQL, or Delta Live Tables (DLT)
• Transition legacy Synapse distribution schemas (HASH, ROUND_ROBIN, REPLICATE) to Delta Lake Liquid Clustering (CLUSTER BY) for optimal query performance and maintenance
• Design efficient batch and streaming ingestion patterns using Auto Loader and Delta Lake Lakehouse mechanisms
• Conduct performance tuning (OPTIMIZE, Z-ORDER, compute cluster sizing, Serverless warehouse setup)

Key Skills Required

• Designing end-to-end Target State Architectures Blueprint incorporating Databricks Lakehouse components (Unity Catalog, Delta Live Tables)
• Databricks: Databricks Runtime, Delta Lake, Delta Live Tables (DLT)
• Articulating design decisions, evaluating build vs. buy, and balancing performance with licensing costs
• Performance & Storage: Liquid Clustering, Partitioning Strategies, Auto Loader, DB SQL Serverless
• Languages: Python, PySpark, Advanced SQL, Scala (optional)

CI/CD & Tools: Databricks Asset Bundles (DABs), dbx, Git integration, MLflow