QA Engineer

About Chargebee

Chargebee is a leading provider of billing and monetization solutions, empowering businesses with recurring revenue models to streamline revenue and finance operations, capture actionable insights, and drive growth.

Chargebee is trusted by businesses of all sizes, including Zapier, LegalZoom, Lambda, Freshworks, DeepL, Condé Nast, and Pret a Manger, and is proud to have been consistently recognized by customers as a Leader in Subscription Management on G2.

With headquarters in North Bethesda, Maryland, our team members are based primarily in India, the U.S., and Europe.

About the Team

Our AI Products team builds Chargebee Copilot and other product experiences that bring Chargebee’s AI Agents platform to real users. We turn the platform’s agents, tools, and APIs into products people use every day, from the backend services and APIs that power them to the interfaces on top. We care about good conversational experiences and about giving people the tools to configure and improve how an AI agent behaves.

About the role (Job Summary)

We’re looking for an AI Quality Engineer to own the quality of our AI-powered products and agents throughout their lifecycle. Unlike traditional QA, AI systems aren’t deterministic, so they need continuous evaluation before release and ongoing improvement based on real-world usage. You’ll make sure our AI delivers accurate, reliable, high-quality experiences by designing effective evaluations, validating new capabilities, analyzing production behavior, and driving improvements across the AI stack. This role is ideal for someone who enjoys understanding why AI systems behave the way they do and is passionate about improving AI products through evidence, experimentation, and engineering collaboration.

What you’ll do

• Design, review, and improve AI evaluation datasets, benchmarks, and regression suites, ensuring coverage reflects real customer scenarios, edge cases, and expected user behavior
• Validate new AI capabilities before they are released to customers, and continuously improve evaluations and release criteria
• Monitor AI quality in production using customer feedback, sampled conversations, telemetry, and quality metrics
• Perform root cause analysis to understand why AI interactions succeed or fail, classifying issues across prompts, retrieval, tool usage, model behavior, workflows, product knowledge, and application logic
• Identify recurring quality patterns and recommend engineering improvements
• Convert production learnings into evaluation cases to prevent future regressions
• Partner with Engineering, Product, and Support to improve prompts, agent workflows, retrieval systems, tools, and evaluation frameworks
• Define and track AI quality KPIs, and measure the impact of improvements using benchmarks and production metrics

What you’ll bring (required skills and experience)

• Strong analytical and investigative mindset with excellent problem-solving skills
• 2+ years of experience
• Ability to identify root causes rather than symptoms
• Excellent written communication and documentation skills
• Comfort working with APIs, logs, structured data, and technical systems
• Ability to collaborate effectively with engineering teams
• Passion for building high-quality AI products

Nice to have (preferred)

• Experience in QA, SDET, Technical Support Engineering, Solutions Engineering, Product Analysis, or AI/ML applications
• Familiarity with LLMs, prompt engineering, RAG, MCP, AI agents, or AI evaluation frameworks
• Experience designing evaluation datasets, benchmarks, or testing strategies
• Basic programming or scripting skills (Python, SQL, JavaScript, etc.)

Benefits we offer

At Chargebee, we believe that feeling supported is the foundation for doing great work. We offer a comprehensive benefits package designed to support your personal, financial, and professional growth and ensure you thrive in every aspect of your life. Our commitment to supporting our employees is universal, while our benefits are thoughtfully designed to meet the unique needs of each region. We offer comprehensive packages that generally include:

• Health & Wellness: Robust coverage and access to programs to support your physical and mental well-being.
• Financial Security: Initiatives to help you build long-term financial stability.
• Work-Life Balance: Flexible options and resources to help you thrive both professionally and personally.

Chargebee is an equal opportunity employer. At Chargebee, we believe that diversity of backgrounds, perspectives, and experiences is not a “good-to-have” but a “must-have” as it strengthens us as a company and as individuals. We are committed to fostering an inclusive workplace where everyone feels valued, respected, and empowered to thrive.

We do not discriminate on the basis of race, color, religion, sex (including pregnancy, childbirth, and related medical conditions), sexual orientation, gender identity/expression, age, national origin, ancestry, caste, citizenship, physical or mental disability, genetic information, marital status, military or veteran status, or any other characteristic protected by applicable law.