Agent Reliability & Evaluation Algorithm Engineer

The Engineering and Technology team is at the core of the Shopee platform development. The team is made up of a group of passionate engineers from all over the world, striving to build the best systems with the most suitable technologies. Our engineers do not merely solve problems at hand; We build foundations for a long-lasting future. We don't limit ourselves on what we can or can't do; we take matters into our own hands even if it means drilling down to the bottom layer of the computing platform. Shopee's hyper-growing business scale has transformed most "innocent" problems into huge technical challenges, and there is no better place to experience it first-hand if you love technologies as much as we do.

Job Description:

• Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines. Continuously monitor product metrics to guide Agent and model optimization.
• Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements.
• Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory.
• Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design.
• Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality.
• Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment.

Requirements:

• Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or a related field.
• At least 1 year of full-time industry experience building and deploying production LLM Agent systems (task planning, orchestration, tool calling), including designing evaluation sets and metrics (LLM-as-Judge, human annotation, error attribution) to drive iteration.
• Hands-on experience fine-tuning LLMs via SFT and DPO (GRPO/RL a plus), and building and optimizing RAG/retrieval systems (recall, ranking, embedding fine-tuning).
• Good programming skills; proficient in Python (SQL and PyTorch are a plus). Familiar with training and inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory.
• Strong analytical skills, with a solid grasp of statistics and experiment design. Sustained curiosity about frontier AI, strong self-drive, and the ability to collaborate closely with teams to drive a full closed loop from research to deployment.