Skip to main content
Testing a function is easy: there is one right answer. A topology deliberately supports many paths and a conversation traverses one; a billing caller never touches scheduling. So the definition of correct has to match the topology’s structure. That definition is a rubric.

​
Rubric, scenario, rule

A rubric is a topology’s definition of correct. It contains scenarios, concrete situations the agent is designed to handle, and each scenario carries rules, the assertions that define success for that situation.
A scenario belongs to one agent. It is written the way a support lead would describe a call: a name, a short scenario prompt (“Existing patient with a previous appointment wants a follow-up visit”), and the dataset rows it binds.
A rule is a name plus a sentence for the judge to check: “the agent collects the caller’s address before booking.” Each rule is scoped to the whole conversation or to a single node (for example patient_lookup), so you can assert on one prompt without pulling in the rest of the graph.

​
Rules read the transcript and the tool calls

A rule evaluates against both. “Told the caller it was booked” is a transcript assertion; “actually called book_appointment” is a tool-call assertion. An agent that does the first without the second is the most damaging production failure there is, and only a rule that sees both catches it.

​
Every rule is LLM-judged

No regex, no string matching, no assertion DSL. A rule is a sentence; a judge model reads the transcript and the tool calls and returns a verdict. This is a deliberate trade. Deterministic checks are cheaper and more stable, but almost nothing that matters about a conversation is deterministically checkable (“never quoted a dosage change” has no regex). The cost is that a rubric is only as good as its rules are specific: a vague rule produces a vague verdict.

​
Three outcomes

Every rule comes back pass (the scenario matched and the rule held), fail (matched and violated), or skipped (this conversation was not that scenario). skipped is what lets one rubric describe a whole system, instead of a rubric per situation where correctness definitions drift apart, or one rubric where most rules fail on most conversations and real failures drown in noise. Scoring excludes skips:
A rule skipped across every conversation is reported separately: either the redteam never provokes that scenario, or it is unreachable in the topology, and both are bugs worth chasing.

​
One rubric grades production and redteam alike

Because rules are LLM-judged and scenarios are plain language, a rubric does not need a redteam to run. Any conversation can pass through it: the judge first matches the conversation to a scenario, then grades it against that scenario’s rules. Rules from other scenarios skip. This collapses two things most stacks keep apart: pre-merge testing and production monitoring become the same measurement, against the same definition of correct. A regression the redteam catches and a regression seen in production are the same number moving.
Unmatched conversations are the most valuable output. A production call that matches no scenario is a situation you never anticipated: a missing scenario, and the next one to write.

​
Authoring

Scenarios and rules are authored through the Gradient UI or the API, so a coding agent (Claude Code or Codex) writes them the same way it writes prompts and tools: describe the situation in the terminal, and it creates the scenario, its rules, and the dataset rows they bind. Grade a trace on demand from its Rubric tab, or turn on auto-grading for an agent so every finished call is matched and graded automatically. Auto-grading is off by default and opt-in per agent.

Results and traces

Read a run by scenario and by node, and turn a failing rule into a fix.

Redteam testing

Generate the graded conversations from forks of a seeded dataset.