Qa Automation Testing Engineer

Role Summary

We're looking for a junior QA/Automation engineer who can validate both classical software and AI-specific behavior: agentic workflows, ML scoring outputs, and the governance/compliance guardrails that sit around them. You'll work across the team's foundational platform (frameworks, tooling, prompt infrastructure) and its live agentic products, building both traditional automation and new testing approaches for probabilistic, LLM-driven systems.

What You'll Work On

Governance & platform foundation testing

• Validate the Agentic Engineering Framework (AEF) and Prompt Manager (currently in Beta) testing prompt versioning, configuration behavior, and framework-level guardrails that other engineering teams will depend on.
• Test the internal agentic code review tool (built on Codex CLI + Azure DevOps APIs) for correctness of its evidence-based reporting output.
• Build and maintain automated regression suites for platform tooling, including API-level and data-layer checks against legacy database systems undergoing AI-assisted optimization.

Agentic & ML product validation

• Validate agent behavior across the team's production AI agents — FSA Nudges, Fraud Sentinel, and Automated Claims — each built on a four-layer hybrid architecture (deterministic compliance guardrails, ML scoring, LLM message generation, deterministic attribution).
• Test each layer appropriately: deterministic guardrails with traditional pass/fail automation, ML scoring outputs with statistical/threshold-based validation, and LLM-generated content with structured evaluation techniques (golden sets, rubric-based scoring, hallucination and drift checks).
• Verify that governance gates actually hold under edge cases — adversarial inputs, boundary conditions, and scenarios designed to probe where compliance guardrails might be bypassed.
• Partner with engineering to define acceptance criteria for agentic features before they ship, not just after.

What We're Looking For

• 2-4 years of experience in QA, test automation, or software engineering with a testing focus; strong fundamentals in test design, not just tool usage.
• Working knowledge of at least one scripting/programming language (Python preferred) for writing automated tests and scripts.
• Familiarity with API testing (REST) and comfort querying relational databases (SQL) to validate data-layer outcomes.
• Genuine curiosity about testing AI/ML and LLM-based systems — you don't need production experience, but you should be able to speak to how deterministic testing differs from validating probabilistic outputs (e.g., through coursework, self-study, or side projects).
• Comfort working with ambiguity: agentic systems don't always have a single "correct" output, and part of the job is defining what "good enough" and "safe" look like.
• Strong attention to detail and documentation habits — governance validation in a regulated domain requires auditable, reproducible test evidence, not just a pass/fail checkbox.
• Good written communication for translating test findings into terms both engineers and non-technical stakeholders can act on.

Why This Role

This is a rare seat for a junior tester: hands-on validation of live, production LLM agents in a regulated industry, alongside foundational platform work that shapes how every other team at Alegeus tests and governs AI features. You'll help define what "quality" even means for agentic systems as the practice is still being built — not just execute against an existing playbook.