Software Engineers

Software Engineers: Coding Tasks for AI Evaluation [$40 per hour] A paid study on the effectiveness of coding evaluation harnesses used to test AI agents

How It Works

• The candidate will walk through a series of proposed coding tasks and environments during an AI-moderated session and review the structure, difficulty, and realism of these programming challenges.
• The candidate will provide technical feedback on the evaluation harnesses and suggest improvements to the test suites. The conversation will focus on ensuring these tasks accurately reflect real-world engineering requirements.

Who This Is For

• Experienced software engineers based in South Asia who have a strong background in building and reviewing complex codebases
• Backend developers, test automation engineers, and full-stack engineers who understand evaluation harnesses
• Ideal candidates have hands-on experience verifying realistic programming tasks in professional environments

Role Responsibilities

• Review proposed coding tasks and programming environments for realism
• Provide technical feedback on the structure and difficulty of evaluation harnesses
• Walk us through your approach to verifying complex code challenges
• Suggest improvements to make the test suites more robust

Good Candidature

• Professional software engineer based in South Asia
• Experience building or reviewing realistic programming tasks
• Familiarity with coding evaluation harnesses or test automation
• Comfortable discussing technical workflows in an interview setting