Senior Data Scientist

Who you are

• Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation)
• Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth
• Proficiency in Python and SQL, with experience evaluating models in production environments
• Strong judgment in defining quality metrics and ground truth for ambiguous outputs
• Excellent communication and cross-functional collaboration skills to align technical teams and leadership
• Strong problem-solving ability in fast-paced, greenfield environments
• 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome
• Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates
• Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data
• Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences
• Fluency with AI tools in research and engineering workflows

What the job involves

• Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents
• Build repeatable eval loops: Track quality over time and catch regressions before release
• Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses
• Drive iteration: Translate eval results into actionable recommendations for system improvements
• Establish quality standards: Set evaluation guidelines, documentation, and review practices
• Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team

Benefits

• Stock Options
• Health benefits from day 1 (internationally, this includes a stipend value to offset medical costs)
• New Hire Home-Office Setup: One-time USD $500
• Monthly Stipend: USD $150 per month via a Brex Card
• Work with awesome people, clients and partners from around the world
• Medical, Dental, Vision
• Supplemental health care