AI Ops Engineer

AI Ops Engineer

ETS AI | GenAI Observability & Operations

Role Purpose

• Own post-deployment observability, reliability, and operational health for ETS AI agents across Copilot Studio and pro-code Azure applications.
• Replace manual, Power BI-centered monitoring with automated dashboards, proactive alerts, and actionable production telemetry for enterprise AI workloads.

Key Responsibilities

• Instrument agents, workflows, and supporting services using OpenTelemetry and distributed tracing across user requests, model calls, retrieval steps, tool execution, and agent workflow stages.
• Build operational dashboards using Azure Monitor, Application Insights, Log Analytics, and KQL; integrate Copilot Studio and Microsoft Foundry telemetry where available.
• Define and operationalize alerts, thresholds, SLIs, and SLOs covering availability, latency, error rates, task completion, retrieval quality, token consumption, and inference cost.
• Correlate agent runs, distributed traces, evaluation results, logs, and production incidents to identify failures and accelerate root-cause analysis and remediation.
• Automate operational reporting and anomaly detection for failed tool calls, latency spikes, quality degradation, abnormal error patterns, and unexpected AI spend.
• Establish standards for structured logging, telemetry consistency, retention policies, sensitive-data redaction, access controls, and production monitoring across ETS AI solutions.

Core Skills & Experience

• Strong hands-on experience with OpenTelemetry, distributed tracing, Azure Monitor, Application Insights, Log Analytics, and KQL.
• Practical observability experience for LLM, RAG, and multi-agent applications, including model traces, retrieval metrics, tool-call traces, quality metrics, latency, token usage, and inference-cost monitoring.
• Strong experience with dashboarding, alert engineering, incident response, production support, and operational troubleshooting; familiarity with Power BI-based reporting is valuable.
• Understanding of LangGraph and/or LangChain execution models, along with Microsoft Foundry and Copilot Studio telemetry, tracing, and evaluation data.
• Ability to collaborate closely with GenAI Engineers and Azure AI Platform Engineers to ensure telemetry is secure, standardized, actionable, and aligned with production reliability goals.