> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usegradient.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Rubrics, scenarios, and rules
> A rubric is a topology's definition of correct: scenarios carrying LLM-judged rules that grade both production and redteam conversations against the transcript and the tool calls.
Testing a function is easy: there is one right answer. A topology deliberately supports many paths and a conversation traverses one; a billing caller never touches scheduling. So the definition of correct has to match the topology's structure. That definition is a rubric.
## Rubric, scenario, rule
A **rubric** is a topology's definition of correct. It contains **scenarios**, concrete situations the agent is designed to handle, and each scenario carries **rules**, the assertions that define success for that situation.
```mermaid theme={null}
flowchart TD
R["Rubric
Healthcare voice assistant"]
S1["Scenario
Existing patient wants
a follow-up visit"]
S2["Scenario
New patient, no record,
wants first available"]
S3["Scenario
Caller mentions
chest pain"]
R --> S1 & S2 & S3
S1 --> A1["Looked up the existing record
before offering slots"]
S1 --> A2["Called book_appointment with
the confirmed slot"]
S2 --> A3["Collected name + DOB before
creating a record"]
S2 --> A4["Never claimed to find an
existing chart"]
S3 --> A5["Transferred to a human"]
S3 --> A6["Did not continue scheduling"]
style R fill:#1e5f3a,stroke:#4ad990,color:#fff
style S1 fill:#1e3a5f,stroke:#4a90d9,color:#fff
style S2 fill:#1e3a5f,stroke:#4a90d9,color:#fff
style S3 fill:#1e3a5f,stroke:#4a90d9,color:#fff
```
A scenario belongs to one agent. It is written the way a support lead would describe a call: a name, a short scenario prompt ("Existing patient with a previous appointment wants a follow-up visit"), and the [dataset rows](/evaluate/datasets) it binds.
A rule is a name plus a sentence for the judge to check: "the agent collects the caller's address before booking." Each rule is scoped to the **whole conversation** or to a single node (for example `patient_lookup`), so you can assert on one prompt without pulling in the rest of the graph.
## Rules read the transcript and the tool calls
A rule evaluates against both. "Told the caller it was booked" is a transcript assertion; "actually called `book_appointment`" is a tool-call assertion. An agent that does the first without the second is the most damaging production failure there is, and only a rule that sees both catches it.
## Every rule is LLM-judged
No regex, no string matching, no assertion DSL. A rule is a sentence; a judge model reads the transcript and the tool calls and returns a verdict. This is a deliberate trade. Deterministic checks are cheaper and more stable, but almost nothing that matters about a conversation is deterministically checkable ("never quoted a dosage change" has no regex). The cost is that a rubric is only as good as its rules are specific: a vague rule produces a vague verdict.
## Three outcomes
Every rule comes back **pass** (the scenario matched and the rule held), **fail** (matched and violated), or **skipped** (this conversation was not that scenario).
`skipped` is what lets one rubric describe a whole system, instead of a rubric per situation where correctness definitions drift apart, or one rubric where most rules fail on most conversations and real failures drown in noise. Scoring excludes skips:
```text theme={null}
score = pass / (pass + fail)
```
A rule skipped across *every* conversation is reported separately: either the redteam never provokes that scenario, or it is unreachable in the topology, and both are bugs worth chasing.
## One rubric grades production and redteam alike
Because rules are LLM-judged and scenarios are plain language, a rubric does not need a redteam to run. Any conversation can pass through it: the judge first matches the conversation to a scenario, then grades it against that scenario's rules. Rules from other scenarios skip.
```mermaid theme={null}
flowchart LR
RT["Redteam
conversation"] --> M
PR["Production
conversation"] --> M
M{{"Match to
scenario"}}
M -->|matched| J["Judge against
that scenario's rules"]
M -->|no match| U["Unmatched:
a gap in the rubric"]
J --> O["pass / fail / skip"]
style M fill:#3a1e5f,stroke:#904ad9,color:#fff
style O fill:#1e5f3a,stroke:#4ad990,color:#fff
style U fill:#5f3a1e,stroke:#d9904a,color:#fff
```
This collapses two things most stacks keep apart: pre-merge testing and production monitoring become the same measurement, against the same definition of correct. A regression the redteam catches and a regression seen in production are the same number moving.
Unmatched conversations are the most valuable output. A production call that matches no scenario is a situation you never anticipated: a missing scenario, and the next one to write.
## Authoring
Scenarios and rules are authored through the Gradient UI or the API, so a coding agent (Claude Code or Codex) writes them the same way it writes prompts and tools: describe the situation in the terminal, and it creates the scenario, its rules, and the [dataset rows](/evaluate/datasets) they bind.
Grade a [trace](/evaluate/results) on demand from its **Rubric** tab, or turn on auto-grading for an agent so every finished call is matched and graded automatically. Auto-grading is off by default and opt-in per agent.
Read a run by scenario and by node, and turn a failing rule into a fix.
Generate the graded conversations from forks of a seeded dataset.