Senior QA Automation Engineer

Job Requirements

Senior/Lead QA Automation Engineer AI/ML & GenAI

Power Systems AI HUB

Power Systems builds highly available, massively scalable AI digital platforms for the enterprise. As part of our AI Hub organization, we are looking for a highly skilled Automation Test Engineer to support our AI/ML and GenAI product ecosystem.

Role Overview

We are seeking a Senior QA Engineer – Automation Testing with 5–6 years of experience in designing, developing, and executing automation frameworks for AI/ML-driven applications. The ideal candidate brings hands-on experience with GenAI Testing, LangSmith, PyTest, Agentic frameworks, and strong AI/ML evaluation expertise.

Key Responsibilities

AI / GenAI Testing

• Design and execute V&V model strategies for classical ML and GenAI/ agentic systems.

• Define test plans, tests cases, and execute them

• Evaluate AI models for accuracy, performance, safety, robustness, hallucinations, bias, and correctness.

• For GenAI, rely on LangSmith for LLM observability, tracing, monitoring, and evaluation, for offline and online evaluation

• For classical ML UC, ensure V&V tests are done during the whole life cycle of the model (ML monitoring)

Test Automation

• Build scalable automation frameworks using Python & PyTest integrated with CI/CD pipelines.

• Develop automated test suites for APIs, microservices, and AI pipelines.

• Perform sanity, integration, functional, regression, and UAT testing across QA, UAT, and Prod environments.

• Implement automation strategies: Keyword-driven, Data-driven, BDD, Hybrid frameworks.

Quality Assurance & Reporting

• Analyze test results and provide insights to engineering teams.

• Log and track defects using standard bug-reporting tools.

• Maintain high-quality documentation for all test processes and reports.

Required Skills & Experience

• 5–6 years of experience in software testing & automation.

• Strong programming expertise in Python.

• Good understanding of Data Science related metrics (for both classical ML and GenAI UC) Experience with AI/Chatbot products and testing AI applications.

• Experience with Git, GitHub Copilot, and CI/CD pipelines.

• Knowledge of Azure/AWS cloud environments.

• Familiarity with MySQL, PostgreSQL, Oracle, SQL Server, NoSQL databases.

• Strong verbal & written communication skills.

• Experience with multi-stakeholder delivery models including onshore–offshore.

Preferred

• Experience in AI/ML model evaluation.

• Exposure to AI observability tools and automated prompt validation.

• Hands-on expertise with LangSmith, Agentic frameworks, PyTest, BDD/V&V frameworks.

 Soft Skills

• Analytical & Critical Thinking

• Curiosity & Continuous Learning

• Strategic Communication (Active Listening, Technical Translations to business impacts)

Work Experience

Educational Qualification

• Bachelor’s or Master’s degree in Computer Science, Electronics, or related field

Experience

• Minimum 5 to 6 years of experience in AI/ML & GenAI Testing