Automation Test Engineer with AI

We are looking for a QA Engineer II to ensure the quality, reliability, and performance of Conversational AI applications. The ideal candidate should have strong software testing fundamentals, hands-on experience with Python and Bash scripting, and familiarity with test automation frameworks such as Playwright or Selenium.

The role involves testing AI agents whose responses may vary, evaluating their behavior, validating functionality, and maintaining reliable benchmarks for continuous quality improvement.

Key Responsibilities

• Design, create, manage, and maintain test benchmarks for AI agents and Large Language Model (LLM)-based applications.
• Perform functional, regression, and manual testing to validate application behavior and identify defects.
• Evaluate AI-generated responses based on semantic accuracy, expected behavior, and task completion rather than exact-match assertions alone.
• Support test automation and benchmarking using Python scripts.
• Test WebSocket communications and validate message payloads.
• Identify, document, track, and report defects while collaborating with developers and QA teams.
• Maintain test cases, test plans, requirement traceability, and test coverage documentation.
• Prepare quality reports for releases and track improvements in test coverage.
• Support performance and load testing using tools such as JMeter.
• Participate in Agile development cycles and contribute to continuous quality improvements.

Mandatory Skills and Qualifications

• Minimum 3+ years of experience in software or product quality assurance.
• Strong working knowledge of Python and Bash scripting.
• Hands-on experience in manual testing, test case design, and test execution.
• Basic familiarity with automation frameworks such as Playwright or Selenium.
• Experience testing WebSocket communications and message payloads.
• Basic understanding of performance and load testing tools such as JMeter.
• Familiarity with AI testing concepts, including prompt evaluation, function calling, and tool-use testing.
• Strong analytical, problem-solving, and debugging skills.
• Ability to document test results and collaborate effectively with cross-functional teams.

Preferred Skills

• Experience writing Python-based test automation scripts.
• Exposure to CI/CD pipelines and automated test integration.
• Familiarity with Conversational AI, NLP-based applications, and LLM-powered systems.
• Experience with test management tools, benchmark tracking, and test result reporting.
• Exposure to evaluating non-deterministic AI outputs and conversational agent behavior.