Senior Software Engineer | $300/task

Role: Senior Software Engineer / Coding Domain Expert — Terminal-Bench 3.0

Experience: 6+ years

Domain: Software Engineering and Programming

Project Type: Technical Benchmark Development and AI Agent Evaluation

About the Role

We are seeking experienced software engineering professionals with prior hands-on experience working on Terminal-Bench 3.0 projects. The selected candidate will design, implement, test, and validate realistic terminal-based coding tasks used to evaluate the capabilities of AI coding agents.

Candidates must possess strong programming expertise, practical software development experience, and the ability to build reproducible computational environments.

Key Responsibilities

• Design multi-step terminal tasks based on realistic software development workflows.
• Develop self-contained computational environments with pinned dependencies and reproducible configurations.
• Implement reference solutions using relevant programming languages, frameworks, and command-line tools.
• Create automated tests, graders, and validation scripts to assess solution correctness.
• Develop tasks involving debugging, algorithm implementation, software integration, performance optimization, and code maintenance.
• Define expected behavior, edge cases, execution constraints, and acceptance criteria.
• Ensure tasks execute successfully without runtime downloads.
• Debug dependency conflicts, runtime failures, performance bottlenecks, and environment configuration issues.
• Validate task reliability, output structure, and reproducibility.
• Document assumptions, technical decisions, limitations, and evaluation criteria for reviewers.

Required Qualifications

• Bachelor's degree from a university ranked among the top 100 institutions in the country.
• Minimum 6 years of relevant professional software engineering experience.
• Mandatory: Prior hands-on experience working on Terminal-Bench 3.0 projects.
• Strong practical expertise in software development, programming, and command-line environments.
• Relevant educational background in Computer Science, Software Engineering, Computer Engineering, or Informatics.
• Ability to develop, debug, test, and validate complex coding tasks independently.
• Experience with version control, dependency management, automated testing, and software debugging.

Preferred Qualifications

• Experience evaluating AI coding agents or terminal-based AI agents.
• Experience creating coding benchmarks, automated graders, or evaluation pipelines.
• Research Software Engineering experience.
• Open-source contributions or experience developing reproducible software projects.
• Experience with CI/CD, containerization, and software testing infrastructure.

Eligible Backgrounds

Senior/Staff/Principal Software Engineer, Backend Engineer, Frontend Engineer, Full-Stack Engineer, Compiler Engineer, Software Architect, Tech Lead, or CTO with relevant hands-on technical experience.

Application Requirements

Applicants must demonstrate prior Terminal-Bench 3.0 project experience, meet the education criteria, and possess at least 6 years of relevant experience. Candidates who do not meet these mandatory requirements will not be considered.