Member of Technical Staff (Data Analysis and Evaluation)

• As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability, and performance of our large language models (LLMs)
• Your primary focus will be on designing and conducting data collection tasks, assessing and evaluating dataset quality, and analysing the robustness and generalisability of our models
• You will work closely with cross-functional teams, including researchers, engineers, and data annotators, to conduct data-driven decision-making and improve the overall effectiveness of our AI systems
• This role combines expertise in statistics, experimental design incl
• Human annotators, and machine learning to ensure that our models are trained on high-quality data and perform reliably across diverse scenarios
• You will contribute to Cohere’s mission of advancing AI by ensuring our systems are robust, scalable, and impactful
• Design and oversee data collection tasks, including supporting human annotators and ensuring data quality
• Develop and apply statistical methods to evaluate the quality and reliability of datasets
• Analyse and assess the generalisability and robustness of ML systems across diverse use cases
• Collaborate with teams to improve dataset quality and model performance
• Train and fine-tune large language models (LLMs) on distributed training infrastructures
• Conduct experiments to evaluate model performance and identify areas for improvement

Benefits

• Six weeks’ paid vacation
• Equity / stock options
• RRSP, 401(k), and Pension Scheme contributions
• Coverage for 100% of your insurance premiums across health, dental, vision, and travel
• Additional coverage for accessing mental health providers/services
• Six months of fully paid parental leave, including adoption and surrogacy
• Financial support for egg freezing and IVF in Canada and the UK
• A monthly fitness and wellness allowance
• Globally dispersed company that supports a remote work culture
• A $2,000 annual education benefit for professional development
• A weekly stipend for meals when working remotely and catered lunch when working from one of our global offices
• A monthly arts and culture allowance
• A monthly quality time allowance- Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
• One or more papers at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)
• Hands-on experience training large language models (LLMs) on distributed training infrastructures
• Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance
• Excellent communication skills to collaborate effectively with cross-functional teams and present findings
• Extremely strong software engineering skills
• Strong expertise in designing and conducting data collection tasks, including working with human annotators
• Experience analysing datasets with respect to their quality, biases, and suitability for training ML models
• Familiarity with evaluating and improving the generalisability and robustness of ML systems