Health AI Conversation Evaluator

About The Job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Generalist Annotator — Health AI Conversation Quality Evaluation

Type: Contract Compensation: $20–$160/hour Location: Remote Role Responsibilities

• Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.
• Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
• Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications Must-Have

• Strong written English skills.
• Ability to explain judgments in two to four specific sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

Preferred

• Experience in annotation, human-feedback, evaluation, or content review.
• Background in explaining technical concepts to non-experts.
• Experience with rubric-based grading or quality assurance.

Important

• Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Application Process (Takes 20–30 mins to complete)

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
• For any help or support, reach out to: [email protected]

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.