Apache Data Engineer (SC Cleared)

Job Title: Apache Data Engineer

Location: Hybrid (up to 3 days per week on-site, across offices in London & South England)

Contract Length: 6 Months (with scope to extend)

Start Date: ASAP

IR35: Outside

Interview Process: 1 Stage, MS Teams

Clearance Required Before Applying: SC (with eligibility for higher levels of clearance)

Our client in the public section is hiring for an "Apache Data Engineer". The successful candidate will join a defence programme delivering a data Lakehouse architecture across AWS and ‘on premises’ infrastructure. The role sits within a collaborative engineering team working in agile sprints. It focuses on building secure, performant, well-labelled data pipelines from source systems into the Lakehouse.

Key Responsibilities

• Design and build data pipelines and ETL processes into a Lakehouse architecture across cloud (AWS) and on premises platforms
• Implement change data capture and event-driven patterns to ingest data from source systems
• Develop data models for the pipeline layer
• Handle non-functional requirements including security, performance and metadata labelling
• Write unit and integration tests for pipeline components
• Work within a defined SDLC using Git-based CI/CD tooling
• Contribute to sprint delivery as part of an integrated development team

Required Skills

• Experience building data pipelines within Lakehouse architectures
• Comfortable with Apache tooling, for data engineering work
• Experience with event sourcing, change data capture and event-driven architectures
• Java and/or Groovy, with a solid grasp of object-oriented design principles
• Experience working in an SDLC with an integrated team on CI/CD tooling (Git)

Nice-to-have:

• Apache NiFi
• AWS, particularly EMR, S3 and SQS
• Awareness of domain-driven design
• Linux and scripting
• Postgres or PL/SQL