Thinking Machines Lab

About the Role
We’re looking for a researcher to help develop reliable model evaluations for research signals. This role spans evaluation creation, usability, auditing, and efficiency.
You’ll work closely with researchers and engineers across post-training and the broader research organization. Depending on your interests and experience, you may focus on one area or work across several of these problems.
What You’ll Do
<ul>
<li>Create
internal evaluations and research signals
for capabilities and behaviors important to model research and post-training.</li>
<li>Develop
usability evaluations
that measure whether models are genuinely useful in real research and product workflows, and partner with the data flywheel to turn evaluation insights into better data and training signals.</li>
<li>Improve
evaluation robustness
, including grader reliability, ambiguous ground truth, evaluator disagreement, false positives and negatives, and gaps between measured and intended behavior.</li>
<li>Build
benchmark auditing
methodologies that help researchers understand, trust, and appropriately use evaluation signals.</li>
<li>Develop specialized
agentic evaluation environments and user simulators
, including supporting harness development and studying cross-user, cross-harness and cross-environment generalization.</li>
<li>Develop evaluations for
personalized preferences, biases, values, and other nuanced dimensions of model behavior
in collaboration with post-training crafting.</li>
</ul>
Skills and Qualifications
Minimum qualifications:
<ul>
<li>Bachelor’s degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding.</li>
<li>Experience designing, building, or analyzing
evaluations, benchmarks, datasets, graders, or other measurement systems
.</li>
<li>Strong written and verbal communication skills, with the ability to collaborate effectively across research and engineering teams.</li>
</ul>
Preferred qualifications — we encourage you to apply if you meet some but not all of these:
<ul>
<li>Experience with
LLMs, post-training, reinforcement learning, or agentic systems
.</li>
<li>Experience with
evaluation auditing, human evaluations, LLM-judges, or open-ended task evaluation
.</li>
<li>Experience with
agentic evaluation, harnesses, long-horizon tasks, or RL environments
.</li>
<li>Experience evaluating
preferences, personalization, biases, values, or other nuanced model behaviors
.</li>
<li>Track record of developing
new evaluation methodologies
or research signals that meaningfully influenced model development.</li>
<li>Proficiency in Python and familiarity with at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX). Comfortable with debugging distributed training and writing code that scales.</li>
<li>Strong research judgment: clean ablations, honest baselines, and clear technical writing.</li>
<li>PhD in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding; or, equivalent industry research experience.</li>
</ul>
Logistics
<ul>
<li>Location: This role is based in San Francisco, California.</li>
<li>Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.</li>
<li>Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.</li>
<li>Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.</li>
</ul>

Research, Post-Training Evals

Research, Post-Training Evals

This job is no longer available

Talents

Recruiters

About