Senior Software Development Engineer in Test (AI/ML)

We are hiring a Sr Software Engineer in Test specializing in ML/AI quality, including automation for model evaluation, LLM-assisted test generation, and validation of AI-powered workflows

The role includes guiding quality strategy and developing automation frameworks

You will also manage implementation for complex, cross-functional, machine learning-powered products and services

This role involves more than test development

You will lead quality initiatives from start to finish, ensuring teams stay synchronized, dependencies are managed, and releases are delivered confidently

You will work closely with ML, engineering, product, and infrastructure teams

You will shape how quality is built, monitored, and expanded while leading embedded QE efforts within projects

ML-powered services and platform components

Use LLMs and other AI-assisted techniques to generate, expand, and maintain high-value test cases for ML-powered workflows

Design scenario-based test suites for AI features, including adversarial prompts, edge cases, ambiguous inputs, and underrepresented user scenarios

Lead QE efforts for multi-functional projects, driving risk assessment, dependency management, and release readiness

Design, develop, and maintain scalable automation frameworks for backend services, APIs, and ML inference systems using Python and/or Java

Build automated validation for ML and LLM outputs, including ranking behavior, score distributions, prompt/response quality, hallucination indicators, and probabilistic model evaluation

Debug test failures, service anomalies, model inconsistencies, and AI behavior regressions to identify root causes and drive resolution

Perform functional, integration, regression, API, end-to-end, performance, and reliability testing for distributed systems

Improve automation reliability, reduce flakiness, and optimize execution efficiency

Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines and release workflows

Collaborate across teams to establish scalable quality standards, tooling, and guidelines

Experience with CI/CD systems and test pipelines (Jenkins, GitHub Actions, etc.)

Bachelor’s degree in Computer Science or equivalent

Passion for scalable, reliable, and maintainable automation systems

Strong understanding of SDLC, Agile methodologies, and release processes

Preferred qualifications

Experience using LLMs to generate, transform, and prioritize test cases for AI-powered experiences

Experience with AI evaluation tooling, prompt evaluation frameworks, model monitoring, or human-in-the-loop review workflows

5+ years of experience as an SDET or QE engineer focused on backend and distributed systems

Proficiency in Python, Java, JS or similar languages for automation development

Ability to influence engineering teams and promote quality guidelines

Strong experience testing RESTful APIs, microservices, and distributed architectures

Practical understanding of how to test non-deterministic AI systems and separate model variance from quality regressions

Excellent problem-solving, debugging, and communication skills

Familiarity with databases, monitoring, and observability tools

Hands-on experience with automation frameworks such as pytest, JUnit, Selenium, Playwright, Cypress, or Appium

Proven risk management and dependency coordination skills

Experience with cloud and container technologies (AWS, GCP, Kubernetes, Docker)

Ability to drive quality across complex cross-functional initiatives

Strong combination of automation engineering and delivery ownership

What sets you apart

Familiarity with ML infrastructure, data pipelines, or model-serving platforms (Seldon, KServe, Ray Serve, etc.)

Experience validating ML outputs using statistical analysis or scenario-based testing approaches

Prior work in content moderation ML, security, fraud detection, or adversarial ML

Experience testing high-scale, low-latency online services

Experience with Databricks or similar ML platform tooling

Familiarity with Node.js, React, or modern frontend technologies

Experience testing mobile, console, or other non-PC platforms