🛠️ How-to guide
<aside> 💡
WHAT IT DOES
Evals are the checks that prove your agent still does what it should. Every time the Designer verifies a change it saves the checks it used, so your suite builds itself as you work — and you can re-run the whole set at any point to catch anything that has slipped.
</aside>
Before you start: your agent needs to exist already, and you need permission to edit it — Editor, Publisher, or Admin. Viewers cannot open Evals.
Open your agent and choose Evals. It sits with the other things you use to watch how the agent is doing, alongside the conversation Log.
The Checks tab lists every check on the agent. Each row gives you its name, what kind of check it is, how big it is, and how its last run went.
The Checks tab for a Riverton 311 agent: four saved checks, each labelled scripted or journey, with the result of its last run.
Use the filters to jump to what matters — All, Passing, Failing, Scripted, or Journey — or search by name or by the wording of the prompt. Run all checks re-runs the suite, and New check adds one of your own.
| Kind | What it does | Use it for |
|---|---|---|
| Scripted | Sends a fixed message and checks the reply — that it contains something, does not contain something, or meets a criterion you write out. | Facts you cannot get wrong, and lines the agent must never cross |
| Journey | Goal-driven rather than word-for-word: the agent is judged on whether it got the caller where they needed to go. | Tone, patience, and multi-step conversations |
The Run history tab is where a regression becomes obvious. The pass rate across the whole suite is charted newest-first, and underneath, each check gets a row of squares — one per verification pass, green for a pass and red for a fail. A single red square in an otherwise green row tells you exactly which change broke something, and each column is tied to the agent version it graded.
Run history: the suite pass rate over time, and a row of squares per check — the red ones mark the passes where that check failed.
On High effort the Designer runs these checks for you after it applies a change, and fixes what fails before handing the work back. On Low effort it skips them for speed. So the suite you see here is the record of testing that has already happened — not homework waiting for you.
Do I have to write these myself? No. The Designer saves the checks it used to verify each change, so most agents arrive with a suite already. Add your own with New check when there is something specific you want guarded.