Observability Engineer

Role Overview

The Observability Expert is the subject-matter expert for all monitoring and telemetry platforms used in this engagement. This is a critical-path role that must be onboarded in Week 1. Without deep, hands-on knowledge of Datadog, Nagios, Azure Monitor, and Solarwinds, the alert normalisation schema for Track 2 cannot be designed and the metric taxonomy for Track 1's Usage Agent will be built on wrong assumptions. This role advises all three tracks and owns the dependency graph design.

Key Responsibilities

• Design the unified alert normalisation schema covering Nagios, Datadog, Azure Monitor, and Solarwinds alert payloads
• Define the metric taxonomy for the Usage Agent in Track 1: which metrics to pull, from which APIs, at what granularity
• Own the service dependency graph design: CI relationship model, node and edge types, refresh strategy
• Advise Track 2 team on alert correlation rules, suppression logic, and time-window configuration
• Configure and validate Datadog API integration for both metric pull (Track 1) and alert ingestion (Track 2)
• Review Nagios and Solarwinds webhook configurations and advise on polling fallback approaches
• Validate Azure Monitor alert schema and advise on event vs metric alert handling differences
• Define false positive thresholds and suppression tuning parameters for Track 2
• Support the observability dashboard design in Track 1 with expertise on what usage data is meaningful
• Provide on-call advisory support across the engagement for any monitoring or telemetry issues

Must-Have Requirements

• 5+ years hands-on experience in an SRE, platform engineering, or observability engineering role
• Deep expertise in Datadog: API usage, metric queries, service maps, alert webhook configuration, and monitor management
• Working knowledge of Nagios: alert payload structure, check configuration, notification webhooks
• Experience with Azure Monitor: alert rules, action groups, log analytics queries, and metric alert schemas
• Understanding of Solarwinds alert structures and webhook/API integration patterns
• Experience designing or working with CMDB CI relationship models in ServiceNow
• Strong understanding of alert correlation concepts: deduplication, parent-child suppression, noise reduction

Nice to Have

• Experience with OpenTelemetry instrumentation and collector configuration
• Familiarity with Backstage.io software catalog and its service dependency model
• Knowledge of Prometheus and Grafana in addition to the platforms listed above
• Experience with graph-based data models for dependency mapping