Functional Leader - Quality Assurance

Job Description: Quality Assurance (QA) Engineer – Conversational AI & Agentic AI

Job Overview

We are looking for a Quality Assurance (QA) Engineer with 3-7 years of experience in testing, QA, and reporting , specifically within the field of Conversational AI and Agentic AI systems . The ideal candidate will have a strong understanding of standard testing practices and be passionate about ensuring the highest quality standards in AI-driven applications. This role involves validating conversational experiences, AI-powered workflows, autonomous agent behaviors, integrations with external tools and APIs, and ensuring the reliability, safety, and performance of AI systems in production environments.

Key Responsibilities

• Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications.

• Collaborate with product managers, developers, data scientists, prompt engineers, and other stakeholders to ensure quality standards are met throughout the development lifecycle.

• Perform functional, regression, integration, performance, and end-to-end testing across conversational AI platforms and agent-based AI systems.

• Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources.

• Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality.

• Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization.

• Identify, document, and track defects using JIRA and provide detailed reports on testing results.

• Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios.

• Conduct prompt, response, and workflow validation to ensure optimal performance and user experience.

• Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards.

• Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights.

• Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities.

Required Qualifications

• 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies.

• Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques.

• Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications.

• Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures.

• Experience with JIRA for defect tracking and project management.

• Strong proficiency in Excel for data analysis, reporting, and test metrics.

• Familiarity with API testing tools such as Postman, Swagger, or equivalent.

• Experience in functional, integration, performance, security, and user acceptance testing.

• Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks.

• Ability to work effectively in cross-functional teams and adapt to fast-paced environments.

• Strong analytical, troubleshooting, and problem-solving skills.

Desired Qualifications

• Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, LangChain, or similar platforms.

• Experience with automated testing tools and scripting languages (Python, JavaScript, etc.).

• Exposure to NLP, NLU, LLM evaluation, prompt testing, and AI model validation.

• Knowledge of cloud AI platforms such as Google CCAI/Dialogflow, Azure AI, AWS Bedrock, Amazon Lex, or OpenAI-based solutions.

• Familiarity with version control systems (Git, GitHub, Azure DevOps).

• Experience developing AI test automation frameworks and evaluation pipelines.

• Understanding of Responsible AI principles, AI governance, model monitoring, and risk assessment.

Personal Attributes

• Attention to detail and a commitment to delivering high-quality work.

• Strong communication and stakeholder management skills, both written and verbal.

• Self-motivated with a passion for AI innovation, quality engineering, and continuous improvement.

• Ability to manage multiple priorities in a dynamic, fast-paced environment.

• Curiosity to explore emerging AI technologies and evaluate complex AI agent behaviors.

• Strong critical thinking skills with the ability to identify risks, edge cases, and quality gaps in AI systems.

Preferred Experience Areas

• Conversational AI Testing (Chatbots, Voicebots, Virtual Assistants)
• Generative AI & LLM Testing
• Agentic AI Testing
• RAG Validation
• Prompt Testing and Optimization
• AI Safety & Responsible AI Testing
• API and Integration Testing
• Test Automation
• Cloud-based AI Solutions

Responsibilities: Quality Assurance (QA) Engineer – Conversational AI & Agentic AI

Job Overview

We are looking for a Quality Assurance (QA) Engineer with 3-7 years of experience in testing, QA, and reporting , specifically within the field of Conversational AI and Agentic AI systems . The ideal candidate will have a strong understanding of standard testing practices and be passionate about ensuring the highest quality standards in AI-driven applications. This role involves validating conversational experiences, AI-powered workflows, autonomous agent behaviors, integrations with external tools and APIs, and ensuring the reliability, safety, and performance of AI systems in production environments.

Key Responsibilities

• Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications.

• Collaborate with product managers, developers, data scientists, prompt engineers, and other stakeholders to ensure quality standards are met throughout the development lifecycle.

• Perform functional, regression, integration, performance, and end-to-end testing across conversational AI platforms and agent-based AI systems.

• Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources.

• Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality.

• Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization.

• Identify, document, and track defects using JIRA and provide detailed reports on testing results.

• Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios.

• Conduct prompt, response, and workflow validation to ensure optimal performance and user experience.

• Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards.

• Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights.

• Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities.

Required Qualifications

• 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies.

• Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques.

• Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications.

• Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures.

• Experience with JIRA for defect tracking and project management.

• Strong proficiency in Excel for data analysis, reporting, and test metrics.

• Familiarity with API testing tools such as Postman, Swagger, or equivalent.

• Experience in functional, integration, performance, security, and user acceptance testing.

• Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks.

• Ability to work effectively in cross-functional teams and adapt to fast-paced environments.

• Strong analytical, troubleshooting, and problem-solving skills.

Desired Qualifications

• Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, LangChain, or similar platforms.

• Experience with automated testing tools and scripting languages (Python, JavaScript, etc.).

• Exposure to NLP, NLU, LLM evaluation, prompt testing, and AI model validation.

• Knowledge of cloud AI platforms such as Google CCAI/Dialogflow, Azure AI, AWS Bedrock, Amazon Lex, or OpenAI-based solutions.

• Familiarity with version control systems (Git, GitHub, Azure DevOps).

• Experience developing AI test automation frameworks and evaluation pipelines.

• Understanding of Responsible AI principles, AI governance, model monitoring, and risk assessment.

Personal Attributes

• Attention to detail and a commitment to delivering high-quality work.

• Strong communication and stakeholder management skills, both written and verbal.

• Self-motivated with a passion for AI innovation, quality engineering, and continuous improvement.

• Ability to manage multiple priorities in a dynamic, fast-paced environment.

• Curiosity to explore emerging AI technologies and evaluate complex AI agent behaviors.

• Strong critical thinking skills with the ability to identify risks, edge cases, and quality gaps in AI systems.

Preferred Experience Areas

• Conversational AI Testing (Chatbots, Voicebots, Virtual Assistants)
• Generative AI & LLM Testing
• Agentic AI Testing
• RAG Validation
• Prompt Testing and Optimization
• AI Safety & Responsible AI Testing
• API and Integration Testing
• Test Automation
• Cloud-based AI Solutions

Qualifications: BS/BE 6-8 years of relevant experience