Agentic Evaluation Specialist (Malay)

About the role

You will evaluate the quality of an AI agent's performance on real-world, multi-step tasks. This is evaluation work on agentic behaviour, not simple chatbot replies. You will read how an AI agent reasons, calls tools, and interacts with a data environment, and judge whether its final answer is truly correct. You'll be trusted to make careful, well-reasoned judgments on complex cases and to explain your decisions clearly.

Key responsibilities

• Review agentic traces from AgentX, including tool calls, tool results, and interactions with its data environment, to understand the full sequence of steps the agent took, not just its final answer
• Read two candidate responses for a given task and select the one that better completes the task, follows instructions, and behaves appropriately along the way
• Verify each response against the trace: check that numbers, facts, and conclusions match what the tools actually returned, and catch hallucinations, misreadings, or skipped steps
• Write short, specific justifications (typically 2–3 sentences) explaining why one response is better
• Apply consistent judgment criteria (accuracy, helpfulness, safety, and adherence to task instructions) across many tasks
• Evaluate language quality and cultural appropriateness for the Singapore language market
• Flag unclear, incomplete, or ambiguous cases according to guidelines

About you

• Native proficiency in Malay language with strong English comprehension for task instructions
• Familiarity with agentic AI workflows: comfortable with tool calling, multi-step task execution, and how an AI agent interacts with external systems and data
• Ability to read structured technical output such as JSON, tables, basic SQL queries, and API responses, enough to tell whether a response matches the data
• Strong attention to detail and the ability to apply evaluation criteria consistently across many tasks
• Clear written English for rating justifications
• Based in Singapore, with cultural and market familiarity relevant to the language
• Master's degree or Bachelor's degree with relevant experience