AI Observability Engineer

Role & responsibilities

AI Model Monitoring & Observability

• Design and implement observability frameworks for AI/ML and GenAI applications.
• Monitor model performance, prediction quality, latency, throughput, and availability.
• Track model drift, data drift, concept drift, and performance degradation.
• Establish AI-centric KPIs, SLAs, and SLOs.

Generative AI Observability

• Monitor LLM applications for:

• Hallucinations

• Response quality

• Prompt effectiveness

• Toxicity detection

• Cost and token consumption

• Retrieval quality in RAG systems

MLOps & Governance

• Collaborate with data scientists and ML engineers to deploy monitoring solutions.
• Ensure compliance with Responsible AI and governance policies.

Incident Management

• Investigate AI service outages and model failures.
• Perform root cause analysis (RCA).

Preferred candidate profile

• Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or related field.
• 5+ years in Data Engineering, AI/ML, MLOps, or Platform Engineering.
• 2+ years working with GenAI/LLM solutions.
• Experience supporting production AI applications.