Braintrust

Help a top AI lab evaluate and improve large language models through security-focused coding tasks. Bring your software-engineering judgment and hands-on security experience to work involving vulnerabilities, exploit verification and security patches.
This is a contracting engagement, with potential for a longer-term engagement. Remote candidates in the selected countries are elegible.
What you will do
<ul>
<li>Evaluate coding tasks involving software vulnerabilities, exploit verification and security patches.</li>
<li>Create high-quality coding prompts and reference answers for benchmark-style problems.</li>
<li>Evaluate model outputs for code generation, refactoring, debugging and implementation.</li>
<li>Identify and document model failures, edge cases and reasoning gaps.</li>
<li>Compare private language models with leading external models.</li>
<li>Build or configure coding environments for evaluation and reinforcement learning.</li>
<li>Follow detailed annotation and evaluation guidelines consistently.</li>
</ul>
What you bring
<ul>
<li>At least five years of professional software-development experience and strong Python skills.</li>
<li>Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing, backporting or validating security patches.</li>
<li>The ability to apply structured evaluation criteria and write clear technical feedback.</li>
<li>Fluency in written and spoken English.</li>
</ul>
Helpful, not required
<ul>
<li>Professional code review, coding annotation, LLM/code evaluation or benchmark design.</li>
<li>Knowledge of another programming language.</li>
<li>Team leadership or mentoring experience.</li>
</ul>

Senior Application Security Engineer - AI Code Evaluation

Senior Application Security Engineer - AI Code Evaluation

This job is no longer available

Talents

Recruiters

About