Senior Databricks Specialist (Data Architect)

· Design and implement enterprise-scale data solutions using Azure Databricks.

· Define and maintain Databricks data architecture and platform standards.

· Design data ingestion and integration pipelines from multiple source systems.

· Develop scalable ETL/ELT pipelines using Databricks and Apache Spark.

· Design and implement Medallion Architecture (Bronze, Silver, Gold) where appropriate.

· Work with Delta Lake and optimize Delta tables and data pipelines.

· Design data models and transformation processes for analytical and operational reporting.

· Integrate Databricks with Microsoft Dynamics 365 / Dataverse and other enterprise applications.

· Develop and optimize batch and near-real-time data processing pipelines.

· Implement data quality, validation, monitoring, lineage, and governance processes.

· Optimize Spark jobs, clusters, SQL queries, Delta tables, and data pipelines for performance and cost.

· Implement appropriate data security and access controls.

· Work with Unity Catalog for data governance, access control, cataloging, and lineage.

· Support Power BI and other analytical/reporting platforms through optimized data models and data services.

· Troubleshoot complex data engineering and performance issues.

· Establish development, testing, and production standards for Databricks.

· Support CI/CD and DevOps practices for Databricks solutions.

· Conduct technical reviews and mentor data engineers.

· Ensure data platform solutions meet security, availability, scalability, and disaster recovery requirements.

Mandatory Technical Requirements

· Minimum 8+ years of experience in data engineering/data architecture.

· Minimum 4+ years of strong hands-on experience with Azure Databricks.

· Strong hands-on experience with Apache Spark and PySpark.

· Strong experience with Delta Lake.

· Strong experience designing enterprise data platforms and data pipelines.

· Strong knowledge of Databricks Medallion Architecture.

· Strong experience with Unity Catalog, data governance, access control, and data lineage.

· Strong SQL skills and experience with large-scale data processing.

· Strong experience with Azure data services.

· Experience with Azure Data Factory / Azure Data Lake Storage Gen2.

· Experience integrating Databricks with APIs, enterprise applications, CRM systems, and other data sources.

· Experience with data modeling and dimensional modeling.

· Strong understanding of data quality and data governance.

· Strong understanding of performance optimization and Databricks cost optimization.

· Experience with CI/CD, Git, Azure DevOps, and deployment automation.

· Experience working with large-volume enterprise datasets.

· Ability to troubleshoot complex production data issues.

Preferred Certifications

· Databricks Certified Data Engineer Professional

· Databricks Certified Data Engineer Associate

· Microsoft Certified: Azure Data Engineer Associate

· Other relevant Databricks/Azure certifications

Experience:

• Data Engineering/Data Architecture.: 8 years (Preferred)
• Azure Databricks: 4 years (Preferred)

Language:

• English (Preferred)

License/Certification:

• · Databricks Certified Data Engineer Professional (Preferred)
• Microsoft Certified: Azure Data Engineer Associate (Preferred)
• QID (Preferred)

Work Location: In person