Software Test Engineer

Role & responsibilities

• Test and validate LLMbased and RAGbased agents, including reasoning, memory, and decisionmaking behavior

• Design and execute prompt engineering and adversarial testing to uncover hallucinations, bias, and edge cases

• Perform hallucination testing, relevance checks, and factual accuracy validation

• Implement unit tests for LLMs using LLMasaJudge techniques

• Validate agent reasoning chains, memory states, and conversation persistence

• Embed LLMs within testing pipelines to:

• Detect hallucinations

Preferred candidate profile

• LangChain + LangSmith
• DeepEval
• Python (LLM testing harnesses)
• OpenAI / Azure OpenAI APIs
• Git + CI integration
• AWS AgentCore
• AWS CloudWatch / Grafana / Prometheus
• Cursor AI / GitHub Copilot / Claude CLI

Perks and benefits