Senior Quality Assurance Automation Engineer

About NoteSpark

NoteSpark is an AI-powered call analytics product built as a connector on the LeadSquared

marketplace. It converts recorded sales calls into structured intelligence: transcripts,

diarization, sentiment, BANT signals, and action items, and enables conversational querying

over call history using RAG. Because the product’s core output is AI-generated, quality here

means more than “does the button work”; it means validating that the intelligence itself is

accurate, consistent, and trustworthy.

What you’ll do

• Own test strategy and execution for NoteSpark across its modules: configuration playground, ingestion pipeline, lead-level insights, reports, and dashboards
• Build and maintain automation suites for API and UI testing, integrated into our CI/CD pipeline
• Write BDD-style test scenarios (Gherkin/Cucumber or similar) in collaboration with product and engineering, so requirements and tests stay in sync
• Design test approaches for AI-generated outputs: evaluating transcription accuracy, diarization quality, summary correctness, extraction precision (action items, BANT), and regression across model/prompt changes
• Test event-driven flows: webhook triggers, queue processing, and end-to-end call analysis pipelines
• Log clear, reproducible defects and work closely with engineers through fix and verification
• Use AI tools day to day to accelerate your own testing work: generating test data, drafting scenarios, and analyzing results

Must have

• 3–4 years of QA experience with a strong foundation in test automation (Selenium, Playwright, Cypress, or similar; API automation with Postman/RestAssured or similar)
• Hands-on experience with BDD frameworks: writing and maintaining Gherkin scenarios with Cucumber, SpecFlow, or equivalent
• Good working knowledge of AI tools and comfort using them as part of the testing workflow
• Solid understanding of QA fundamentals: test design, regression planning, defect lifecycle, and CI integration
• Strong analytical skills and attention to detail, especially for outputs where “correct” isn’t binary

Good to have

• Experience testing AI/LLM-based applications: evaluating non-deterministic outputs, building golden datasets, prompt regression testing, or model output scoring
• Exposure to audio/speech products or data-heavy SaaS platforms
• Basic scripting ability (Python, JavaScript, or C#) beyond test frameworks
• Familiarity with AWS or cloud-based test environments

Why join

You’ll be the quality owner on a small, senior team shipping an AI product to enterprise

customers. This is a chance to define how AI-based applications should be tested (a skill set the industry is only beginning to figure out) rather than following an established playbook.