Observability Architect

Purpose

To own the event streaming and integration architecture across the platform, establish the integration contract with the wider consortium, and deliver the change data capture pipeline that feeds the metadata, billing, management information and audit functions throughout the life of the platform.

Key responsibilities

• Document the observability architecture across the infrastructure, application and database layers, including the log routing model, retention schedule and access control matrix.
• Package Grafana, Prometheus and Loki as parameterised Helm charts with externalised configuration, deployable to development, staging and both production sites.
• Define the instrumentation standard for Spring Boot subsystems — including metric naming, required metrics, structured log format, correlation identifier propagation, and distributed tracing — delivered as a shared library or starter for subsystem teams to consume rather than reimplement.
• Build three distinct dashboard layers for separate audiences: programme delivery, platform health, and business key performance indicators.
• Define service-level objectives with named measurement points, including instrumentation of cross-party dependencies on a receiver-confirms basis.
• Design alerting and escalation, with alert rules packaged alongside the stack rather than configured by hand.
• Extend observability coverage to the disaster recovery site, including monitoring behaviour during a site failover.
• Derive delivery telemetry automatically from systems of record — GitLab CI, ArgoCD, the issue tracker and the requirements traceability matrix — rather than from manual reporting.
• Develop the nominated counterparts to independent capability.

Essential experience

• Prometheus, Grafana and Loki designed and operated at scale in production, including retention and cardinality management.
• Instrumentation of JVM and Spring Boot services, including propagation of correlation identifiers across service boundaries.
• Service-level objectives and error budgets defined and used operationally.
• On-premise or hybrid deployment. Managed cloud observability services do not substitute.
• Has packaged an observability stack for deployment by another team.

Desirable experience

• OpenSearch or Elasticsearch operations, sufficient to reason about the boundary with the security logging estate.
• Distributed tracing at scale, with a considered sampling strategy.
• Environments where telemetry was used contractually — service credits, penalty measurement or regulatory reporting.

Expected hours: 40.0 – 45.0 per week

Application Question(s):

• Have you delivered software into a multi-party programme, where the organisation deploying it was not the one building it?
• Have you delivered production work in an on-premise or hybrid environment (not cloud-only)?
• Are you comfortable working within a settled technology

decision?

Experience:

• On-premises: 6 years (Required)

Work Location: In person