Applied Research Scientist

About the role

Applied Research Scientist role focused on post-evaluation and LLM training. The position involves research experience in vision-language models, computer vision, and multimodal AI.

Key responsibilities

• Design benchmark datasets, test suites, or evaluation frameworks for language or multimodal models
• Conduct evaluation science and benchmarking work with deep understanding of metric design, scoring reliability, and measurement validity
• Design and execute evaluation studies and protocols that are reproducible across datasets, model versions, and evaluation runs
• Perform quantitative analysis including statistical analysis, sampling, uncertainty, significance testing, and error analysis
• Evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability
• Collaborate directly with technical stakeholders including research scientists, ML engineers, data scientists, and customer technical counterparts
• Present nuanced technical conclusions, assumptions, and limitations clearly to stakeholders
• Reason about model behavior, failure modes, and tradeoffs across tasks and domains
• Synthesize complex experimental findings into actionable recommendations

About you

• MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field (PhD strongly preferred)
• 5+ years of relevant experience in applied research or research science in ML/AI, with substantial work in LLMs or foundation models
• Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research
• Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems
• Strong coding skills in Python for research experimentation and analysis (e.g., data processing, evaluation pipelines, statistical analysis, visualization)
• Experience working with modern ML tooling/frameworks (e.g., PyTorch, Hugging Face, JAX/TensorFlow as applicable)
• Experience with human evaluation methods and quality assurance considerations (e.g., rubric design, inter-rater reliability, adjudication frameworks)
• Understanding of post-training methods and how training objectives interact with evaluation outcomes
• Ability to collaborate directly with technical stakeholders and communicate complex technical information clearly