AI Test Engineer / Machine Learning QA Analyst / AI Validation Engineer

Role Description

Tests and validates AI and machine learning systems to help ensure they work reliably, meet product requirements, and behave as expected across different situations. Works with software engineers, data scientists, product managers, and subject matter experts to understand system requirements and translate them into test plans, validation criteria, and measurable quality goals.

Designs and runs tests for AI features, including model outputs, data pipelines, application integrations, and end-to-end user workflows. Evaluates performance using appropriate metrics and test cases, and checks how results vary across inputs, user groups, operating conditions, and model versions. For generative AI systems, testing may include response accuracy, relevance, consistency, grounding in provided information, handling of ambiguous requests, and resistance to inappropriate or unsafe outputs.

Investigates defects and unexpected behavior by reviewing inputs, outputs, logs, datasets, and system configurations. Documents findings clearly, works with development teams to reproduce and resolve issues, and verifies fixes before release. Helps create and maintain test datasets, evaluation suites, regression tests, and automated quality checks so that changes to models, prompts, data, or software do not introduce new problems.

Monitors AI systems after deployment and helps identify changes in performance, data quality, or user experience over time. Contributes to validation reports, release decisions, and documentation of known limitations. May also support reviews of privacy, security, fairness, explainability, and other responsible AI considerations, based on the system’s purpose and applicable requirements.

Qualifications

• Bachelor’s degree in computer science, software engineering, data science, statistics, or a related field; equivalent practical experience may be considered
• Experience in software quality assurance, machine learning validation, model evaluation, or data testing
• Understanding of software testing methods, including functional, regression, integration, and end-to-end testing
• Familiarity with machine learning concepts, model performance metrics, and the effects of data quality on system results
• Ability to design structured test cases and evaluation criteria for both expected and unexpected inputs
• Proficiency in Python or another programming language; experience with test automation frameworks is an asset
• Experience working with SQL, APIs, logs, datasets, and version control tools
• Familiarity with tools and platforms for model evaluation, experiment tracking, or machine learning operations is beneficial
• For generative AI roles, knowledge of prompt testing, retrieval-augmented generation, response evaluation, and safety testing is an asset
• Strong analytical and investigative skills, including the ability to reproduce issues and identify likely causes
• Clear writing skills for documenting test plans, defects, results, and system limitations
• Careful, detail-oriented approach to test data, evaluation results, and release documentation
• Ability to collaborate with technical and non-technical teams and explain quality findings clearly
• Knowledge of privacy, security, fairness, or responsible AI practices is an asset for relevant systems and industries