Observability Engineer

Purpose

To operate and extend the platform observability layer — onboarding subsystems to the instrumentation standard, building and maintaining dashboards and alerting, and retaining the capability for the life of the programme after the architect engagement concludes.

Key responsibilities

• Onboard platform subsystems to the instrumentation standard, working with subsystem teams to ensure metrics, structured logs and correlation identifiers are implemented consistently.
• Build and maintain dashboards across the three defined layers — programme delivery, platform health and business key performance indicators.
• Develop and tune alert rules, and manage alert quality so that the on-call signal remains meaningful.
• Maintain the Prometheus, Grafana and Loki estate: retention, cardinality management, storage growth and query performance.
• Package observability configuration and alert rules as versioned artefacts for deployment by the integration team.
• Support service level objective measurement, including instrumentation of cross-party dependencies.
• Extend and maintain coverage at the disaster recovery site.
• Produce and maintain operational documentation and runbooks.

Essential experience

• Three or more years operating Prometheus, Grafana and Loki, or an equivalent open-source observability stack, in production.
• PromQL to a working standard — able to write and debug non-trivial queries and recording rules.
• Instrumentation of JVM and Spring Boot applications, including metrics libraries and correlation identifier propagation.
• Alert rule development and tuning, with a demonstrable approach to reducing alert noise.
• Linux, containers, Kubernetes and Helm to a working standard.
• Scripting in Python or equivalent for automation.

Desirable experience

• LogQL and log pipeline configuration at volume.
• OpenTelemetry instrumentation and distributed tracing.
• OpenSearch or Elasticsearch, sufficient to work across the boundary with the security logging estate.
• On-premise infrastructure monitoring, including hardware and database layers.
• Java or Spring Boot development background.

Expected hours: 40.0 – 45.0 per week

Application Question(s):

• Have you delivered software into a multi-party programme, where the organisation deploying it was not the one building it?
• Have you delivered production work in an on-premise or hybrid environment (not cloud-only)?

Experience:

• On-premises: 4 years (Required)

Work Location: In person