Software Engineer

Engagement: Project-Based
Location: Remote — U.S.
Commitment: 20 hours/week
Compensation: $80–$100/hour

About the Role

We are looking for an experienced Software Engineer with expertise in reinforcement learning, AI evaluation, or coding agents to help build next-generation software engineering environments and benchmarks.

You will design realistic, challenging coding tasks that require AI models to work through complex software repositories, debug issues, implement features, refactor code, and reason across multiple steps. Unlike simple coding exercises, these environments are designed to evaluate sustained software engineering ability in realistic development workflows.

This is a hands-on, project-based opportunity for an experienced engineer who enjoys working at the intersection of software engineering, reinforcement learning, AI agents, and evaluation.

What You’ll Do

• Design and develop long-horizon coding tasks that require multi-step reasoning and implementation across real codebases
• Build reinforcement learning environments and software engineering task suites for AI models
• Create robust tests, graders, and evaluation frameworks to reliably measure task completion and code quality
• Develop challenging scenarios involving debugging, feature development, refactoring, testing, and repository-level reasoning
• Create realistic tasks that require AI agents to understand and modify unfamiliar codebases
• Analyze model performance and evaluation results to identify weaknesses and opportunities for improvement
• Iteratively improve task difficulty, quality, diversity, reliability, and resistance to shortcut solutions
• Apply strong software engineering judgment to ensure tasks reflect realistic development practices

Required Qualifications

• 8 years of professional software engineering experience
• Hands-on experience designing or developing reinforcement learning tasks, coding evaluations, AI benchmarks, or agentic coding environments
• Experience working extensively in complex production codebases
• Strong skills in software testing, debugging, architecture, and code quality
• Familiarity with modern reinforcement learning concepts and iterative model improvement workflows
• Ability to design realistic, measurable engineering tasks that are resistant to shortcut solutions
• Strong ability to understand unfamiliar repositories and identify meaningful engineering challenges
• Excellent written and technical communication skills
• Availability to contribute 20 hours per week

Preferred Qualifications

• Experience building long-horizon coding evaluations, benchmarks, or AI agent tasks
• Experience with DeepSWE-style coding evaluations or similar repository-level software engineering benchmarks
• Background developing evaluation infrastructure for AI coding agents or software engineering models
• Experience with reinforcement learning training or evaluation platforms such as Mercor, Handshake AI, Turing, or Micro1
• Experience analyzing AI model performance and using evaluation results to improve training or task design

Why Join

This is an opportunity to work on challenging problems at the intersection of software engineering, reinforcement learning, and artificial intelligence.

Your work will directly contribute to the environments, evaluations, and benchmarks used to measure and improve AI systems#39; ability to perform complex, real-world software engineering tasks.