Databricks Architect

Databricks Architect

Must Have Technical/Functional Skills

Mandatory

Former Databricks Employee (Non-Negotiable Requirement)

12+ Years of IT Experience

5+ Years of Databricks Architecture Experience

Strong Lakehouse Architecture Expertise

Hands-on Spark/PySpark Experience

Enterprise Cloud Data Platform Design Experience

Data Governance & Security Expertise

Stakeholder Management and Leadership Experience

Preferred BFSI Domain Experience

GenAI / AI Platform Architecture Experience

Large-scale Cloud Migration Experience

Pre-sales / Solution Consulting Experience

Databricks Technologies

• Databricks Lakehouse Platform
• Delta Lake
• Photon Engine
• Unity Catalog
• Delta Live Tables (DLT)
• Databricks Workflows
• MLflow
• Auto Loader
• Structured Streaming
• Databricks SQL
• Databricks Asset Bundles

Data Engineering

• Apache Spark
• PySpark
• Spark SQL
• Scala (preferred)
• Python
• SQL Optimization
• Data Modeling
• Data Warehousing

Cloud Platforms

• Microsoft Azure (Preferred)
• Azure Data Factory
• Azure Synapse Analytics
• Azure Storage
• Azure Key Vault
• Azure Entra ID (Azure AD)

OR

• AWS Databricks Ecosystem
• Google Cloud Databricks Ecosystem

DevOps & Automation

• Terraform
• GitHub
• Azure DevOps
• CI/CD Pipelines
• Docker
• Kubernetes

Data Governance & Security

• Unity Catalog
• Apache Ranger
• Data Lineage Tools
• Data Quality Frameworks
• Data Classification & Governance

________________________________________

Domain Experi Ence (Preferred)

• Banking & Financial Services (BFSI)
• Capital Markets
• Wealth Management
• Insurance
• Lending & Consumer Finance
• Regulatory Reporting and Compliance

________________________________________

Certifications (Preferred)

• Databricks Certified Data Engineer Professional
• Databricks Certified Solution Architect
• Databricks Certified Machine Learning Professional
• Azure Solutions Architect Expert
• Azure Data Engineer Associate

Roles & Responsibilities

Architecture & Strategy

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks, and best practices.
• Lead modernization initiatives involving legacy data warehouse migration to Databricks.

Data Engineering & Integration

• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
• Build medallion architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
• Implement data quality, lineage, observability, and monitoring frameworks.
• Optimize data ingestion and transformation processes for performance and scalability.

Cloud & Platform Engineering

• Architect Databricks solutions on Azure, AWS, or Google Cloud Platform.
• Design secure networking, storage, identity management, and access controls.
• Implement Infrastructure as Code (Terraform, ARM, CloudFormation).
• Optimize platform usage and cloud costs through governance and monitoring.
• Lead Databricks workspace design and multi-environment deployment strategies.

AI, ML & Advanced Analytics

• Architect Databricks solutions supporting Machine Learning and Generative AI workloads.
• Design ML lifecycle management using MLflow.
• Enable enterprise AI capabilities leveraging Databricks Mosaic AI and GenAI frameworks.
• Collaborate with Data Scientists and AI teams to operationali ze ML models.
• Establish MLOps best practices and governance standards.

Security & Governance

• Implement Unity Catalog for enterprise-wide governance.
• Define data security controls including RBAC, ABAC, encryption, and access management.
• Ensure compliance with industry standards including SOX, PCI-DSS, HIPAA, GDPR, and BFSI regulatory requirements.
• Establish data governance, metadata management, and audit frameworks.

Leadership & Stakeholder Management

• Act as the primary Databricks subject matter expert.
• Engage with executive stakeholders to define data transformation roadmaps.
• Conduct architecture reviews and provide technical leadership.
• Mentor data engineers, platform engineers, and architects.
• Collaborate with business, infrastructure, security, and application teams.

Salary Range- $120,000-$150,000 a year