QA Engineer – AI Evaluation & Software Testing

We are looking for experienced QA Engineers / Software Test Engineers to support a high-volume software quality evaluation project focused on AI training and Human-in-the-Loop (HITL) evaluation.

The ideal candidate must have strong hands-on experience in manual and automated software testing along with prior paid professional experience in AI evaluation, human data annotation, data labeling, RLHF, LLM response evaluation, or other human-in-the-loop AI training activities.

Important: Traditional QA/software testing experience alone is not sufficient. Prior paid experience in human-data or AI evaluation is mandatory.

No AI development experience is required.

Key Responsibilities

• Evaluate and rate AI-generated technical outputs against defined quality criteria, guidelines, and rubrics.
• Identify technical inaccuracies, omissions, logical issues, inconsistencies, and incomplete responses.
• Design and execute functional, regression, negative, boundary, and edge-case testing.
• Review test cases and testing documentation for coverage, accuracy, and completeness.
• Analyze bug reports for reproducibility, severity, completeness, and business/technical impact.
• Identify, isolate, and document software defects with clear steps to reproduce.
• Maintain accurate testing records using defect-tracking and test-management tools.
• Provide clear, structured, and actionable written feedback and annotations.
• Evaluate the correctness, consistency, quality, and completeness of technical responses.
• Support the refinement of evaluation rubrics, testing guidelines, and quality standards.
• Contribute to continuous improvement of QA and AI evaluation processes.

Required Qualifications

• 7+ years of professional experience in QA Engineering, Software Testing, SDET, Test Engineering, or a related field.
• Strong knowledge of:
• Quality Assurance
• Manual Testing
• Automated Testing
• Functional Testing
• Regression Testing
• Negative Testing
• Boundary & Edge-Case Testing
• Test Case Design
• Defect/Bug Reporting
• Quality Evaluation
• Hands-on experience with automation testing frameworks, preferably Selenium or equivalent.
• Experience with test management and defect-tracking tools.
• Strong analytical and problem-solving abilities.
• Excellent attention to detail.
• Strong written English communication skills (B2 level or above).

Mandatory AI / Human Data Experience

Candidates must have prior paid professional experience in at least one of the following:

• Human Data Annotation
• Data Labeling
• AI Response Evaluation
• AI/Model Output Evaluation
• RLHF – Reinforcement Learning from Human Feedback
• LLM Response Evaluation
• AI Training Data Quality Evaluation
• Rubric-Based Grading
• Human-in-the-Loop (HITL) AI Evaluation

Candidates with only traditional software QA/testing experience and no prior paid AI evaluation or human-data experience will not be considered.

Preferred Tools & Technologies

Experience with one or more of the following is preferred:

• Selenium
• Playwright
• Cypress
• Appium
• Postman
• Jira
• TestRail
• Zephyr
• BrowserStack
• Other equivalent automation, API testing, defect-tracking, or test-management tools

Candidate Requirements

• 7+ years of practical QA/testing experience.
• Strong experience in both manual and automation testing.
• Prior paid professional AI evaluation / human-data experience is mandatory.
• Ability to critically evaluate technical outputs using predefined rubrics.
• Strong written communication and documentation skills.
• Ability to work independently in a remote environment.
• LinkedIn profile is mandatory.
• Only candidates currently on bench will be considered.
• BGV is not required.

Application Question(s):

• Do you have prior paid professional experience in AI response evaluation, LLM/model output evaluation, human data annotation, data labeling, RLHF, rubric-based grading, or Human-in-the-Loop (HITL) AI evaluation? If yes, mention the project/company, duration, and type of work performed.
• Do you have 7+ years of professional QA/Software Testing experience with hands-on manual and automation testing, and are you currently available as a bench resource for immediate remote contract work ?

Work Location: Remote