Data Engineer

Data Engineer – Databricks Specialist

Job Type: Contract

Location: Dublin, Ireland

Working Mode: Hybrid – 2 to 3 days per week onsite at the client office

Role Purpose

We are looking for an experienced Data Engineer – Databricks Specialist to build scalable data pipelines and ensure curated, high-quality cyber data is readily available for analytics and reporting.

The ideal candidate will have strong hands-on experience with Databricks, PySpark, Spark SQL, Delta Lake, and cloud-based data platforms , with exposure to streaming technologies and cybersecurity data.

Key Responsibilities

• Build scalable data ingestion pipelines for logs, network data, and security tools .
• Develop robust ETL/ELT pipelines using Databricks and Apache Spark.
• Implement and maintain Delta Lake architecture using Bronze, Silver, and Gold layers.
• Optimise data pipelines for performance, scalability, and cost efficiency through partitioning, caching, and other optimisation techniques.
• Ensure high standards of data quality, lineage, reliability, and governance .
• Work with streaming data sources and technologies such as Kafka or Azure Event Hub .
• Collaborate with data, cybersecurity, and analytics teams to deliver reliable curated datasets.

Required Skills & Experience

• Strong hands-on experience with PySpark and Spark SQL .
• Proven experience with the Databricks ecosystem , including:
• Delta Lake
• Databricks Jobs
• Databricks Workflows
• Experience building batch and streaming data pipelines .
• Strong understanding of Kafka and/or Azure Event Hub .
• Cloud experience, preferably AWS , with hands-on knowledge of:
• Amazon S3
• IAM
• AWS Glue
• Strong understanding of Lakehouse architecture and data engineering best practices .
• Experience working with large-scale datasets and performance optimisation.

Key Deliverables

• Scalable and reliable data ingestion pipelines .
• High-quality, curated cybersecurity datasets .
• Optimised Databricks/Lakehouse architecture .
• Reliable Bronze–Silver–Gold data layers.
• Improved data quality, pipeline performance, and operational reliability.

Interested candidates can share their updated CV with: 📧 [email protected]