Agentic AI Evaluation Specialist

Hiring: Agentic AI Evaluation Specialist | Remote | India

Work Mode: Remote

Engagement Type: Freelance

Location: India

Qualification : Master's degree in Computer Science, IT, Engineering, Data Science, or a related field.

About the Role

We are looking for detail-oriented professionals to evaluate the performance of AI agents on complex, real-world tasks. This role goes beyond reviewing chatbot responsesyou will assess how AI agents execute multi-step tasks, use tools, interpret data, and arrive at their final answers.

You will compare AI-generated responses, verify their accuracy against actual tool outputs, identify errors or unsupported claims, and provide clear, evidence-based feedback to improve AI systems.

Key Responsibilities

• Evaluate AI agent workflows, including tool calls, tool outputs, and multi-step execution traces.
• Compare two AI-generated responses and select the one that best satisfies the task requirements.
• Verify facts, calculations, and conclusions against source data and tool results.
• Identify hallucinations, reasoning errors, missed steps, and instruction-following issues.
• Write concise justifications explaining evaluation decisions.
• Apply consistent quality, accuracy, safety, and instruction-adherence criteria.

Requirements

• Master's degree in Computer Science, IT, Engineering, Data Science, or a related field. Candidates from other disciplines with relevant experience are also welcome.
• Strong English comprehension and written communication skills.
• Experience in AI/LLM evaluation, data annotation, software QA, API testing, data analysis, or similar analytical work.
• Ability to interpret structured data such as JSON, tables, basic SQL queries, and API responses.
• Strong attention to detail, analytical thinking, and evidence-based decision-making.

Preferred Qualifications

• Experience with RLHF, AI model evaluation, response rating, or side-by-side comparison tasks.
• Familiarity with AI agents, tool calling, multi-step workflows, or AI automation.
• Experience using Claude Code, Cursor, Codex, or similar AI-powered coding tools.
• Background in software testing, data validation, technical research, or quality assurance.

Note: Prior experience with agentic AI is an advantage, but candidates with strong analytical and technical validation skills are encouraged to apply.

Selection Process

Shortlisted candidates will complete a brief screening assessment involving an AI task, its execution trace, and two candidate responses to evaluate.