Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization) - Canada (Remote)

Role: Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization) Remote (Canada) Job Type: Contract Role Summary

• We are looking for a hands-on Lead Data Engineer to support a proof of concept in Azure Databricks.
• The role focuses on converting complex SQL stored procedures into optimized Python and PySpark code. The objective is to demonstrate that the platform can process 100,000 simulation years in minutes rather than hours, while maintaining accurate results.

Key Responsibilities

• Refactor complex T-SQL logic, including pricing rules, layer applications, and reinstatements, into Python and PySpark.
• Replace row-by-row processing with efficient, vectorized DataFrame operations.
• Develop Azure Data Factory pipelines to ingest loss data from Touchstone/RMS into ADLS Gen2.
• Optimize Spark jobs through effective partitioning, memory management, and skew handling.
• Validate the new solution against existing SQL results to ensure accuracy.
• Contribute to performance testing and troubleshooting.

Required Skills

• Advanced Python skills, including Pandas and PySpark.
• Strong T-SQL experience and the ability to analyze complex stored procedures.
• Hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory.
• Strong knowledge of distributed computing and Spark optimization.
• Experience with partitioning, sharding, broadcast variables, and data-skew management.
• Familiarity with CI/CD and version control for data engineering projects.

Desirable Skills

• Experience with Monte Carlo simulations or stochastic modelling.
• Knowledge of Parquet and Avro file formats.
• Experience analyzing SQL Server execution plans.
• Background in insurance, risk analytics, or simulation-based modelling.