AI Quality Analyst (Personalization) – Japanese

Location: Remote

Job Type: Contract

Duration: 3 months

Rate: $15/hour

Hours: 4–20 hours per week

About The Role We are looking for a Japanese-speaking AI Quality Analyst to evaluate personalized AI interactions.

You will create realistic, multi-turn prompts based on your personal experiences and assess how effectively the AI uses relevant information to provide accurate, natural, and helpful responses.

This role combines AI evaluation, prompt design, analytical reasoning, Japanese language expertise, and attention to detail .

What You’ll Do

• Create and test multi-turn prompts, typically 1–5 turns, using personal context.
• Evaluate whether AI responses appropriately address the user's intent.
• Assess the accuracy and relevance of personalization.
• Identify incorrect assumptions, unsupported inferences, hallucinations, and forced connections.
• Evaluate how naturally personal information is integrated into responses.
• Compare two AI responses side-by-side and determine which is more helpful, relevant, natural, and enjoyable.
• Write clear rationales explaining your evaluations, referencing specific conversation turns.
• Review relevant model debug information and verify that available data sources were appropriately used.
• Provide detailed annotations and constructive feedback.
• Follow required data hygiene procedures, including deleting evaluation conversations when required.

Requirements

• Strong Japanese reading and writing skills.
• Excellent analytical and critical-thinking abilities.
• Strong attention to detail.
• Ability to evaluate nuanced or ambiguous AI responses.
• Ability to identify subtle differences in naturalness, relevance, and quality.
• Strong written communication skills.
• Ability to create creative prompts and realistic conversational scenarios.
• Ability to work independently in a remote environment.
• Desktop or laptop with a reliable internet connection.

Preferred Experience Experience in any of the following is a plus:

• AI quality evaluation
• Data annotation
• Content moderation
• Linguistic evaluation
• Quality assurance
• Research or analytical work
• Computer Science, Linguistics, Law, Policy, Ethics, Journalism, or related fields

Personal Account Requirement

For authentic personalization testing, you must be willing to use your primary personal Google account rather than a testing account.

The project may require access to relevant personal data sources such as previous AI conversations, Gmail, Google Search, and YouTube activity.

Schedule

• Minimum 4 hours per day
• Up to 20 hours per week
• 4 hours of overlap with PST required
• Global 24-hour operations environment

Selection Process

• Shortlisted candidates receive a Job Interest Form.
• Profiles are reviewed for relevant qualifications and availability.
• Selected candidates receive an assessment, which must be completed within 24 hours.
• Successful candidates proceed to pre-onboarding.

Ideal Candidate

You are a strong fit if you are a fluent Japanese writer with excellent analytical judgment, enjoy testing AI systems, can identify subtle quality issues, and can clearly explain why one AI response is better than another.