Lead Data Engineer

Job Title: Lead Data Engineer
Location: Australia (Remote and Hybrid)

Role Description & Requirements:

• Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
• Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
• Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team
• Perform pair programming with customer’s data engineers
• Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
• Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

• Expertise Required
• In-depth knowledge and experience of GCP data and analytics technologies
• Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
• Define solution architecture and detailed design, and perform hands-on implementation for – data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL· BQ query performance optimisation
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)
• Experience applying Generative AI technologies and integrations in enterprise environments
• Must have: GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform
• Must have: programming knowledge and willingness to be hands-on - Python, Java

Good to Have

• Experience with dbt or Dataform, Terraform in context of data pipelines
• Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
• Understanding and experience with NoSQL databases such as HBASE, MongoDB
• Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations
• Certification
• Good to have GCP Data engineeringIMPO : Key Skills:
• GCP Dataflow, BigQuery, Cloud Spanner, Pub/Sub
• Cloud Composer, Cloud Workflows
• Confluent Kafka
• Terraform, Cloud Build
• Python and Java
• Real-time and batch data ingestion platforms
• Large-scale transaction data platform architecture
• Solution design, technical leadership, and customer-facing consulting

About Us:

We are Delivery Centric, a dynamic technology company that is transforming the delivery of cloud implementations and cybersecurity. We are headquartered in Australia with a global presence. We are tenacious, future-thinking, and highly driven to achieve an ambitious vision to be the leading global provider of innovative technologies for companies that keep our world clean, safe, and secure.

Our fast-paced and supportive environment will offer you lots of benefits, including a competitive salary, a friendly team, a healthy work environment, and opportunities for training and development.

Please contact us at [email protected] for additional details

Job Type: Full-time

Work Authorisation:

• Australia (Preferred)

Work Location: Hybrid remote in Australia