LLMOps & Observability Engineer

Our Cutomer is establishing a dedicated AI team to bring structured AI engineering capabilities to its software delivery organization. The team is divided into two functions: a Platform team that builds shared infrastructure, and an Enablement team that collaborates with engineering and product teams to ship AI use cases and transfer the capability to operate independently. The goal is to achieve measurable AI capability across the SDLC within six months.

This role is part of the Platform team and is responsible for the infrastructure that makes AI workloads measurable, including tracing, evaluation pipelines, quality alerting, test data management, and prompt versioning.

Missions

• Own observability infrastructure: distributed tracing of LLM calls, latency, cost, and quality signals. Assess current tooling such as Langfuse and decide whether to continue using it or switch to alternatives like Langsmith.

• Develop and maintain evaluation pipeline infrastructure that enables teams to define, run, and version their evaluation suites continuously.

• Manage test data: generation, curation, versioning, and access control for evaluation datasets.

• Design and build prompt and model versioning systems, as none currently exist.

• Implement alerting mechanisms for production quality degradation.

• Develop model registry patterns as the portfolio of use cases expands.

Tools & Environment

• Observability tooling including Langfuse, Langsmith, Braintrust, and others.

• TypeScript-first technology stack (Python is not in scope).

• AWS infrastructure for deployment and operation of the observability stack.

Working Conditions

• Role within a newly formed AI Platform team.

• Focus on building scalable, production-grade AI observability and evaluation infrastructure.

• Goal-oriented with measurable success criteria within six months.

1. Observability engineering production experience with tracing, metrics, and alerting for complex systems
2. Direct LLM observability experience preferred; strong distributed systems observability with understanding of LLM-specific challenges acceptable
3. Evaluation pipelines experience designing or running eval pipelines for AI systems including unit-level prompt tests, integration-level flow tests, and production quality monitoring
4. LLMOps tooling working knowledge of current landscape such as Langfuse, Langsmith, Braintrust, with ability to evaluate and select between them
5. TypeScript proficiency as the stack is TypeScript-first (Python not in scope)
6. AWS experience comfortable owning deployment and operation of observability stack on AWS infrastructure