Senior Quality Assurance AI Engineer II CloudAngles

.

About the Role

We are looking for QA Engineers with 5+ years of experience to test AI-powered applications, LLM applications, RAG systems, and agentic AI workflows. The candidates will work under the Senior QA/Test Strategy Lead and will be responsible for executing functional and AI-specific testing while building automated test coverage.

Key Responsibilities

• Develop and execute functional and non-functional test cases.
• Test AI/GenAI-powered applications across different user scenarios.
• Validate LLM responses for accuracy, relevance, consistency, grounded ness, and completeness.
• Test RAG pipelines including retrieval quality and response grounding.
• Test AI agents for tool selection, tool execution, workflow completion, and failure handling.
• Create test datasets and evaluation scenarios for AI applications.
• Execute AI regression tests following model, prompt, or application changes.
• Perform API testing and integration testing.
• Automate repetitive test scenarios.
• Identify, document, and track defects.
• Perform root-cause analysis in collaboration with developers.
• Validate fixes and conduct regression testing.
• Test edge cases, adversarial inputs, and unexpected user behavior.
• Support performance and reliability testing.
• Integrate automated testing into CI/CD pipelines.
• Maintain test documentation and quality reports.

Required Skills

• 5+ years of QA/software testing experience.
• Strong understanding of functional, regression, integration, and API testing.
• Experience with automation testing.
• Strong knowledge of REST APIs and tools such as Postman.
• Experience with Selenium, Playwright, Cypress, or equivalent.
• Experience testing AI/ML/GenAI applications strongly preferred.
• Understanding of LLM behavior and AI application testing.
• Understanding of prompt testing and AI evaluation.
• Strong analytical and debugging skills.
• Working knowledge of Python or another scripting language.
• Experience working in Agile development environments.

Preferred Skills

• Experience with RAG and vector databases.
• Experience testing AI agents.
• Experience with Ragas, DeepEval, Promptfoo, LangSmith, or equivalent.
• Experience with LLM evaluation datasets.
• Experience with cloud applications, preferably AWS.
• Experience with performance testing tools.

Interested candidate kindly share their profile at [email protected]