Business Domain Expert — AI Model Evaluation

Business Domain Expert — AI Model Evaluation Pilot

Job Type: Independent Contractor

Location: Remote

Schedule: Flexible, short-term project

Compensation: $60–$80/hour (approximately $124,800–$166,400 annualized , based on 40 hours/week)

About The Opportunity We are seeking experienced professionals in Business Operations, Marketing, Human Resources, and Accounting to participate in a structured AI model evaluation pilot.

You will complete one realistic professional task within your area of expertise and evaluate five AI-generated attempts at the same task. Your professional judgment will help assess how effectively advanced AI models perform real-world professional work.

This is a one-time evaluation study , rather than an ongoing production role.

What You'll Do

• Sign a confidentiality agreement before receiving study materials.
• Complete one realistic, domain-specific task using the provided source materials.
• Review five AI-generated outputs for the same task.
• Rank the outputs from 1–5 with no ties.
• Score each output from 0–100 based on its quality and effectiveness.
• Write clear, evidence-based rationales referencing specific parts of each output.
• Provide structured feedback on the task and study design.

Areas of Expertise

We are looking for professionals with experience in one of these domains:

• Administration / Business Operations
• Marketing
• Human Resources
• Accounting

Time Commitment

The Study Requires Up To 13 Hours Total

• 3–10 hours to complete the assigned task.
• Approximately 2 hours to review, rank, and score the five outputs.
• Approximately 1 hour to provide feedback.

Study Requirements

• No rubric or predefined answer will be provided. Your professional judgment is the basis of the evaluation.
• The use of LLMs, AI assistants, or other AI tools is prohibited throughout the study, including while completing the task, evaluating outputs, assigning scores, or writing rationales.
• Evaluations must be based on your own professional knowledge and experience.

Requirements

• 5+ years of hands-on professional experience in at least one of the four target domains.
• Current or recent professional experience in your area of expertise.
• Strong professional judgment and ability to assess the quality of real-world business work.
• Excellent written communication skills.
• Ability to provide specific, clear, and evidence-based reasoning.
• Reliable availability to complete the study within a short turnaround window.

Eligibility Restriction

You are not eligible for this pilot if you have previously worked on Project Alchemy in any capacity, including as a task author, reviewer, domain expert, or anyone who has had access to its source-world data. Prior exposure to the material would compromise the integrity of the study.

Contract Details

• Independent contractor engagement.
• Fully remote with flexible scheduling.
• One-time, short-term evaluation project.
• Projects may be extended, shortened, or concluded early depending on requirements and performance.
• Payments are made weekly based on services rendered.
• H-1B and STEM OPT candidates are not currently supported.
• Work will not require access to confidential or proprietary information belonging to another employer, client, or institution.

Why Participate

• Apply your professional expertise to the evaluation of frontier AI models.
• Help measure how effectively AI performs realistic professional tasks.
• Contribute expert human judgment to the development of next-generation AI systems.
• Participate in emerging AI evaluation work within your area of expertise.

Ideal Candidate: An experienced professional who can independently evaluate business work, identify meaningful strengths and weaknesses, and clearly explain their judgment using specific evidence.