AI Specialist/Data annotation - Non voice

Role Summary

The GenAI Specialist Rating evaluates AI-generated responses against defined quality rubrics

to determine whether responses meet gold-standard expectations. Specialists assess factual

accuracy, grounding, completeness, helpfulness, safety, and user satisfaction while providing

clear rationale for every decision.

Their work directly contributes to high-quality evaluation datasets used for model training,

benchmarking, and calibration.

Key Responsibilities

• Evaluate AI-generated responses using standardized evaluation rubrics
• Assess factual accuracy of geographic and local information
• Verify that responses are properly grounded in available map and place data
• Evaluate:

Factuality

Grounding

Helpfulness

Completeness

Relevance

Clarity

Safety

Instruction following

• Identify hallucinations and unsupported claims
• Distinguish between critical and minor defects
• Provide detailed justification for ratings
• Escalate ambiguous or policy-sensitive cases
• Maintain consistency across evaluation tasks
• Participate in calibration exercises
• Contribute feedback to improve evaluation guidelines

Core Skills

Generative AI Evaluation

• Understanding of LLM behavior
• Prompt-response evaluation
• Hallucination detection
• Response quality assessment

Analytical Thinking

• Critical reasoning
• Evidence-based decision making
• Attention to detail
• Pattern recognition

Maps Knowledge

• Geographic reasoning
• Navigation concepts
• POIs (Points of Interest)
• Local search behavior
• Route interpretation

Language Skills

• Excellent written English
• Reading comprehension
• Ability to interpret nuanced prompts

Secondary Skills

• Search verification techniques
• Knowledge of local businesses
• Travel and navigation terminology
• Familiarity with Maps products
• Basic data annotation experience
• Spreadsheet proficiency

Ancillary Skills [Good to know]

• SQL (basic)
• Annotation platforms
• Jira
• Documentation tools
• AI evaluation platforms

Qualifications

• Work Experience: 03 years in data annotation, content evaluation, technical writing,

or a related field. Experience with AI/LLM evaluation or editing is preferred.

• Educational Qualifications: Bachelor’s degree in Linguistics, English, Computer

Science, Humanities, or a related field, or equivalent practical experience.