Senior Research Scientist (Model Evaluation)

• Evaluation is critical to making progress in scaling intelligence
• As models continue to become superhuman in many real-world use cases, we must continue to develop new evaluation techniques that accurately reflect what models are already capable of, as well as set the agenda for what future models should be capable of
• In this role, you are responsible for creating these next-generation evaluation methods and infrastructure to measure LLM progress
• Create ambitious new evaluation benchmarks that push the limits of what our models can accomplish
• Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations
• Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges; refining LLM-based data synthesis pipelines; and improving evaluation efficiency
• Build scalable and reusable tools for digging into model performance

Benefits

• Six weeks’ paid vacation
• Equity / stock options
• RRSP, 401(k), and Pension Scheme contributions
• Coverage for 100% of your insurance premiums across health, dental, vision, and travel
• Additional coverage for accessing mental health providers/services
• Six months of fully paid parental leave, including adoption and surrogacy
• Financial support for egg freezing and IVF in Canada and the UK
• A monthly fitness and wellness allowance
• Globally dispersed company that supports a remote work culture
• A $2,000 annual education benefit for professional development
• A weekly stipend for meals when working remotely and catered lunch when working from one of our global offices
• A monthly arts and culture allowance
• A monthly quality time allowance- You enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of, and you have developed resources to measure those capabilities
• You have strong software engineering skills
• You are obsessive about rigorously measuring AI capabilities, and also about making sure your measurements actually align with the capabilities you care about
• You have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality
• If some of the above doesn’t line up perfectly with your experience, we still encourage you to apply!