Technical Program Manager - AI Delivery

Program Delivery

• Own end-to-end delivery of AI/LLM programs across multiple engineering teams, from discovery and proof of concept through production rollout and post-launch optimization.
• Define program scope, milestones, dependencies, and success criteria, balancing experimentation with predictable delivery.
• Run program cadences (weekly reviews, risk reviews, steering updates) and keep Jira roadmaps, epics, and dependency maps accurate and actionable.
• Identify and resolve cross-team blockers across ML, data, platform, backend, frontend, and QA teams.

Technical Leadership

• Partner with ML engineers and architects on design decisions such as hosted APIs vs. open-weight models, RAG vs. fine-tuning, model routing, and agent/tool-use architectures.
• Drive delivery of retrieval-augmented generation (RAG) pipelines, including document ingestion, chunking strategies, embeddings, vector stores, and re-ranking.
• Oversee prompt engineering workflows, prompt versioning, and structured output design.
• Coordinate fine-tuning and adaptation efforts (e.g., LoRA/PEFT, instruction tuning) where applicable, including dataset preparation and labeling programs.
• Track and manage non-functional requirements: latency, throughput, token cost, context-window limits, scalability, and availability.

Evaluation & Quality

• Establish evaluation frameworks with ML and QA teams: offline benchmarks, golden datasets, LLM-as-judge pipelines, human review loops, and regression gates in CI/CD.
• Define and track quality metrics such as accuracy, groundedness/faithfulness, hallucination rate, retrieval precision/recall, task success rate, and user satisfaction.
• Ensure A/B testing and staged rollouts (shadow mode, canary, feature flags) are built into delivery plans.

LLMOps & Infrastructure

• Coordinate delivery of LLMOps capabilities: model and prompt registries, observability and tracing, cost dashboards, drift monitoring, and incident response.
• Work with platform/cloud teams on GPU/compute capacity planning, inference serving (e.g., vLLM, TGI, managed endpoints), caching, and cost optimization.
• Manage vendor and model-provider relationships, including API quotas, SLAs, pricing changes, and model deprecation timelines.

Responsible AI, Security & Compliance

• Embed guardrails into delivery: prompt-injection defenses, PII redaction, content filtering, access controls, and audit logging.
• Coordinate with security, legal, and compliance teams on data residency, privacy, model risk, and applicable AI regulations and standards (e.g., EU AI Act, ISO/IEC 42001, NIST AI RMF).
• Ensure customer and sensitive data is handled according to policy throughout training, evaluation, and inference.

Stakeholder Management

• Translate technical trade-offs (cost vs. quality vs. latency) into clear options for leadership and clients.
• Produce concise executive reporting on program health, risk, spend, and outcomes.
• Build shared understanding of AI capabilities and limitations across non-technical stakeholders.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field (Master’s preferred).
• 8+ years in software engineering, technical program management, or engineering leadership, with a hands-on engineering background.
• 2+ years delivering ML, AI, or LLM-based systems into production.
• Solid understanding of transformer-based LLMs, embeddings, tokenization, context windows, and their practical limitations.
• Practical familiarity with LLM application frameworks and tooling (e.g., LangChain, LlamaIndex, Semantic Kernel, or similar) and vector databases (e.g., pgvector, Pinecone, Weaviate, Milvus, OpenSearch).
• Experience with major model ecosystems (e.g., Anthropic, OpenAI, Google, Azure OpenAI, AWS Bedrock, or open-weight models such as Llama/Mistral).
• Working knowledge of cloud platforms (AWS, Azure, or GCP), containers and Kubernetes, and CI/CD pipelines.
• Ability to read Python code, API specs, and system architecture documents and engage meaningfully in design reviews.
• Strong experience with Agile/Scrum and scaled delivery, and advanced use of Jira and Confluence (including JQL, dashboards, and cross-project dependency tracking).
• Excellent written and verbal communication, with experience reporting to senior leadership and international clients.