Databricks Engineer

Databricks Engineer – 18 Month Contract
Wellington | 18-month contract

An exciting opportunity is available for an experienced Databricks Engineer to join a large-scale transformation programme in the public sector. This role will sit within a major data transformation initiative, supporting the replacement of a legacy CRM platform and the introduction of new data sources into a modern Azure data environment.

The role is highly hands-on and focused on data engineering, ETL, integration and ingestion. You’ll be responsible for building, developing and deploying data pipelines and ensuring new data sources are successfully integrated into the organisation’s Azure data warehouse.

A key part of the initial engagement will be supporting a pilot to demonstrate the new approach and prove how the newly developed data architecture can be implemented in practice.

This is not an architecture role. However, you will need to review and understand the new architecture, data patterns and technical standards, then ensure your engineering work aligns with them.

Key responsibilities:

• Design, develop and deploy robust data pipelines using Databricks
• Build ETL and data ingestion processes for new data sources
• Develop data integration solutions across the Azure data environment
• Work extensively with PySpark and SQL
• Support the initial data engineering pilot and subsequent implementation
• Review and work within the organisation’s newly developed data architecture
• Ensure pipelines are reliable, scalable and maintainable
• Troubleshoot and optimise data processing and pipeline performance
• Work closely with technical and programme stakeholders throughout the
  transformation

About you:
You’ll be an experienced Data Engineer with strong hands-on experience across Databricks and Azure data engineering environments.
You’ll ideally have:

• Strong Databricks experience
• Strong SQL skills
• Strong PySpark experience
• Experience building ETL and data ingestion pipelines
• Strong Azure Data Engineering experience
• Experience integrating multiple data sources into data warehouse environments
• Ability to work within and align engineering solutions to an established architecture
• Strong understanding of data pipeline performance, reliability and optimisation

Power BI experience would be advantageous, particularly where you have used it to demonstrate reporting, data availability or performance, but it is not essential.

This is a great opportunity to work on a significant data transformation programme, with an initial pilot providing the opportunity to help establish the engineering approach before moving into the wider implementation.
18-month contract | Wellington | $130–$150 per hour
 
Please apply to the link provided.