Principal Azure Databricks Architect

Job Title: Principal Azure Databricks Architect

Location: Toronto, ON (hybrid)

Duration: long term contract

Rate: CAD110-CAD120/hr

Role Overview

We are seeking an experienced Principal Azure Databricks Architect to design, implement, and optimize enterprise-scale data platforms within the Azure ecosystem. The ideal candidate will have deep expertise in Azure Databricks, enterprise data lakes, Lakehouse architecture, data engineering, cloud cost optimization, and modern CI/CD practices.

This role requires strong hands-on coding skills, architectural leadership, and the ability to work independently with business and technical stakeholders to deliver scalable, secure, and cost-effective data solutions.

Key Responsibilities

• Design, architect, and implement enterprise-scale data lakes and Lakehouse solutions on Azure.
• Lead the setup and optimization of Azure Databricks and Azure Data Lake Storage environments.
• Develop scalable data pipelines using Azure Data Factory, Databricks, Python, PySpark, and SQL.
• Design robust data models and integration frameworks for structured and unstructured data.
• Write complex, production-ready code and troubleshoot application, pipeline, and performance issues.
• Establish CI/CD processes using Jenkins and GitHub.
• Implement monitoring and observability solutions using Dynatrace.
• Provide architectural guidance around Lakehouse design, Databricks environments, governance, security, and performance.
• Drive Databricks and cloud FinOps initiatives to optimize platform usage and costs.
• Work with relational and NoSQL databases to support enterprise data integration requirements.
• Collaborate with stakeholders to understand business needs and independently deliver robust technical solutions.
• Provide technical leadership, establish engineering best practices, and guide development teams.
• Evaluate emerging technologies related to AI agents, skills, and Model Context Protocol, also known as MCP.

Required Qualifications

• Extensive experience designing and implementing enterprise data lakes and Lakehouse architectures.
• Strong hands-on expertise in:
• Azure Databricks
• Azure Data Factory
• Azure Data Lake Storage
• Python and PySpark
• Advanced SQL
• Jenkins and GitHub
• Dynatrace or similar observability tools
• Strong knowledge of Databricks FinOps, cloud cost optimization, and platform performance.
• Experience working with relational and NoSQL databases.
• Strong programming experience in Python, PySpark, or Java.
• Ability to develop complex code and independently diagnose and resolve technical issues.
• Excellent analytical and problem-solving capabilities.
• Strong stakeholder management and communication skills.
• Ability to translate business requirements into scalable technical architecture.

Nice-to-Have Skills

• Experience developing dashboards and reports using Power BI.
• Exposure to one or more BI tools, such as Cognos, Power BI, Sigma, or Strategy.
• Experience building AI agents and related AI capabilities.
• Understanding of Model Context Protocol, or MCP.
• Additional hands-on experience with Dynatrace.
• Functional experience in retail banking, capital markets, or securities processing.