Applied AI Engineer

LocationSeattle or San Francisco

Employment typeFull-time

Location typeIn-person

DepartmentEngineering

Why this role exists

Commenda is building an autonomous agent for global compliance and tax work. Teams use it to register entities, manage obligations, and stay compliant globally. Today that work requires additional headcount, local vendors, and disconnected systems.

This role owns the quality of our AI products. You'll decide how we measure whether our agents are right, build the evaluations that tell us, and use the results to make them better, including fixing problems that sit outside the model, in our data and operations.

What you'll be accountable for

AI evaluation. Define what "correct" means for each AI feature, then measure it. When someone says a fix worked, you're the person who can show whether it did.

Fixing the inputs, not just the model. Many agent failures trace back to data the agent can't see or can't trust: a status that lives in someone's notes, a field the operations team fills in inconsistently, an API that doesn't expose what the agent needs. When your evals point to a cause like that, you take it upstream, to an operations process, a backend change, or the data model, and use the evidence to get it prioritized. When it's faster to fix it yourself, with a SQL backfill, a script, or a redesigned intake form, you will.

AI product quality over time. Find where our agents fail, rank failures by how often they happen and what they cost customers, and drive them down.

AI architecture. Decide how our agents, workflows, and content pipelines are built: the harness they run in, the context they get, and how they connect to the rest of the product.

The entity graph. Our agents reason over a data model of companies: their entities, people, governance, operations, and compliance obligations. You'll iterate on that model so agents can understand customer scenarios clearly and act on them correctly.

How the team uses AI. Advise engineering, product, and content on evaluation and AI tooling, and help them find the fastest reliable way to get work done.

What you'll actually do day-to-day

• Read user queries and agent responses, and check what the agent did against our production data.
• Build eval datasets from production traces and expert annotation, choose the scores, and write and calibrate LLM judges.
• Run evals in Braintrust and report what changed and whether it's real.
• Change prompts, context, tools, or models, and prove the change helped.
• Work with operations and engineering when a failure traces back to how data is captured or exposed.
• Write design docs for agent architecture and review them with the engineers who will build them.

What we're looking for

Required:

• Statistical rigor. You know the difference between an anecdote and evidence. You can design a measurement that answers the question being asked and tell when a result is noise.
• Hands-on experience evaluating model output. You've scored or ranked LLM output against a standard, written judges or rubrics, and checked them against human judgment.
• You build your own tools. You've built harnesses, pipelines, or analysis tools to answer questions nobody assigned you.
• Judgment about AI systems. You can tell when to change a prompt, add context, add a tool, switch models, or add a human review step.
• Self-direction. You find the important problem, decide how to attack it, and come back with results.
• Clear written and spoken English. You can explain a result, its limits, and what to do next to engineers and non-engineers.
• Curiosity about complex domains. You'll learn enough about legal, tax, and regulatory requirements to know what a right answer looks like.

Preferred:

• Experience with Braintrust or another evaluation platform.
• A background in data science or machine learning. You've applied classical ML and LLMs to real business problems, and you know when each one fits.
• A background in tax, accounting, or compliance.

You don't need a computer science degree or a background as a software engineer. Many strong people in this field taught themselves, and we care about what you've built and measured.

Where the role sits on our team

You'll report to the Head of Product and work day to day with the engineers who build our AI systems. You'll sit either in our Seattle office, with the team that talks to customers every day, or in our San Francisco office, with our CTO and Head of Product.

We're an ambitious, intense team spread across Bangalore, San Francisco, and Seattle. We work like a sports team: we practice, challenge each other, and take shared responsibility for winning. AI is part of how we build every day. You'll have real ownership early, direct access to the people making decisions, and room to grow as you prove what you can take on.