Content Specialist III (AI Content Evaluation & Quality)

Job Title: Content Specialist III (AI Content Evaluation & Quality)

Location: Remote (U.S. East Coast or West Coast Preferred)

Duration: 6+ Month Contract (Extension Possible)

Contract: W2

We are seeking an experienced Content Specialist III to help evaluate, improve, and shape next-generation AI products. This role combines content expertise, analytical thinking, and AI evaluation to ensure high-quality, accurate, and helpful model responses across a wide range of topics.

The ideal candidate is an exceptional writer and editor who can apply structured guidelines consistently, identify quality issues, provide detailed feedback, and help improve AI behavior through prompt refinement and rigorous evaluation techniques.

Responsibilities

• Evaluate AI-generated responses across diverse subjects and conversation types.
• Test new model capabilities and document success cases, failures, and edge cases.
• Assess responses using detailed quality standards, scoring rubrics, and evaluation frameworks.
• Provide actionable feedback on model performance, reasoning quality, tone, factual accuracy, and instruction adherence.
• Refine prompts and evaluation criteria to improve output quality and consistency.
• Perform quality reviews of both human and AI-generated evaluations.
• Conduct fact-checking, content validation, and root-cause analysis on model failures.
• Collaborate with cross-functional teams to improve evaluation processes and model performance.
• Document findings clearly and communicate recommendations effectively.
• Adapt quickly to changing priorities and evolving project requirements.

Required Qualifications

Bachelor's degree or equivalent practical experience.

5+ years of experience in one or more of the following:

Writing

Editing

Journalism

Content Production

Linguistics

Technical Writing

Policy Analysis

Research

STEM-related disciplines

• Exceptional writing, editing, proofreading, and communication skills.
• Deep expertise in at least one subject area and the ability to apply that expertise when evaluating content.
• Strong analytical thinking and judgment.
• Ability to interpret and consistently apply detailed guidelines and evaluation criteria.
• Strong attention to detail and commitment to quality.

Preferred Qualifications

• Experience working with AI systems through evaluation, annotation, prompt development, quality assessment, red teaming, or model testing.
• Experience performing fact-checking, source verification, and failure analysis.
• Familiarity with structured quality review processes and rubric-based evaluations.
• Ability to work independently in a fast-paced environment with shifting priorities.
• Strong problem-solving and documentation skills.

Top Skills:

AI Content Evaluation

Writing & Editing

Prompt Engineering

Quality Assurance

Content Strategy

Fact Checking

Research & Analysis

Linguistics

Technical Writing

Content Review

Annotation

Generative AI