Databricks Solution Architect

We are looking for an experienced Databricks Solution Architect to design, architect, and deliver scalable enterprise data solutions using the Databricks platform.

The ideal candidate will have at least 5 years of hands-on experience with Databricks and strong expertise in Python/PySpark, Apache Spark, SQL, and Hive , combined with a solid understanding of modern data engineering and Lakehouse architectures.

This role requires someone who can translate business and technical requirements into robust architecture designs, provide technical leadership to engineering teams, and drive best practices across data ingestion, transformation, processing, governance, performance, and scalability.

Key Responsibilities

• Design and implement scalable Databricks-based data and Lakehouse architectures for enterprise environments.
• Define solution architecture, technical standards, design patterns, and best practices for Databricks implementations.
• Architect large-scale data processing and transformation solutions using Apache Spark, PySpark/Python, SQL, and Hive .
• Provide technical leadership and architectural guidance to Data Engineers, Developers, and other technical teams.
• Design robust ETL/ELT pipelines supporting batch and large-scale data processing workloads.
• Define strategies for data ingestion, transformation, storage, processing, and consumption across the data platform.
• Identify and address performance, scalability, reliability, and maintainability challenges within distributed data environments.
• Lead technical design discussions and translate business requirements into effective technical solutions.
• Review existing Databricks implementations and recommend architectural and performance improvements.
• Create architecture diagrams, technical documentation, solution designs, and implementation guidelines.
• Collaborate with engineering, analytics, cloud, security, governance, and business stakeholders.
• Support technical evaluations, proofs of concept, and architecture decisions for new data initiatives.
• Establish reusable frameworks and engineering standards across Databricks environments.

Required Qualifications

• 5+ years of hands-on experience with Databricks.
• Strong hands-on experience with Python and/or PySpark .
• Advanced knowledge of Apache Spark and distributed data processing.
• Strong experience with SQL and Hive .
• Strong understanding of enterprise data architecture and modern Lakehouse architecture .
• Experience designing and implementing large-scale ETL/ELT and data engineering solutions .
• Strong understanding of data modeling, data transformation, and data integration patterns.
• Experience designing scalable and high-performance data platforms.
• Strong troubleshooting and performance optimization skills.
• Experience translating technical and business requirements into end-to-end architecture designs.
• Ability to communicate complex architectural concepts to both technical and non-technical stakeholders.
• Previous experience providing technical leadership or architecture guidance to engineering teams.

Preferred Qualifications

• Experience with Delta Lake and Databricks SQL .
• Experience with Structured Streaming and real-time/streaming architectures.
• Experience working with Databricks on Azure, AWS, or GCP .
• Knowledge of data governance, security, access control, and enterprise data management.
• Experience modernizing legacy Hadoop/Hive/Spark environments into Databricks.
• Experience designing reference architectures and enterprise data-platform standards.
• Databricks certifications are a plus.

Must Have

• Databricks: 5+ years
• Python / PySpark
• Apache Spark
• SQL
• Hive
• Data Engineering
• ETL / ELT
• Data Architecture
• Lakehouse Architecture
• Distributed Data Processing

Nice to Have

• Delta Lake
• Databricks SQL
• Structured Streaming
• Unity Catalog
• Azure / AWS / GCP
• CI/CD
• Terraform / Infrastructure as Code
• Data Governance
• Data Modeling