Data Engineer

Databricks Data Engineering Consultant

Location: Sydney or Melbourne (Hybrid)

Contract position (8 weeks)

Our client is seeking an experienced Databricks Data Engineer for an 8-week contract to bring the Databricks Proof of Technology (PoT) to completion and stand up a simulated MVP environment. This role supports the Global Optimisation Platform Project, which is creating an innovative optimisation and bidding solution.

About the role:

• Be part of a core Data Engineering team, this engagement adds the specialist Databricks capacity to close that gap across the team.
• Contributing direct Databricks engineering effort within the existing team, with the goal of clearing the PoT gate on schedule and handing Quant Research a workable simulated MVP.
• Stand up the core Databricks environment for the simulated MVP, with security built in through Unity Catalog, access controls, and compute policies.
• Create simulated asset telemetry and align the market data, giving the Quant Research team a complete, end-to-end modelling pipeline
• Use real effort from reference pipelines as a baseline to refine the delivery-phase estimate for the data ingestion and integration workstream.

Required Experience:

• Over five years of data engineering experience, including a minimum of two years building and running Azure Databricks platforms in production.
• Hands-on experience designing and governing Unity Catalog, covering catalog structure, access permissions, external locations, storage credentials and system tables.
• Deep knowledge of Azure Databricks security, including VNet injection, Private Link, Entra ID with SCIM provisioning, service principals and Key Vault-backed secret scopes.
• Proven experience building real-time pipelines with Structured Streaming on Event Hubs or Kafka, along with Delta Lake and medallion architecture.
• Practical experience deploying with Databricks Asset Bundles and running CI/CD pipelines in Azure DevOps or GitHub, plus working with Git folders.
• Advanced Python skills, including building wheel packages and managing library dependencies in privately networked environments.

Should you have the experience and skills required to be a success in this role, hit APPLY today!