> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# Testing / Evals

Sero uses repository tests and promptfoo evals as separate quality signals.
Not every suite runs for each pull request. Run real LLM evals only when their
provider cost and credentials are intentional.

## Current root command surface

```bash
pnpm typecheck
pnpm build
pnpm test
pnpm test:ci
pnpm eval:snapshot
pnpm eval:search
pnpm eval:file-tools
pnpm eval:memory-save
pnpm eval:memory-search
pnpm eval
pnpm eval:view
```

`pnpm test` runs `turbo run test`. Use the root commands above as the supported
entry points.

## CI workflows

The **Test** workflow runs for pushes to `main`, pull requests to `main`, and
manual dispatch. It classifies changed paths. The affected jobs run
`pnpm typecheck`, `pnpm build`, and `pnpm test`.

The root `pnpm test:ci` command also runs `pnpm e2e:contract`, but the **Test**
workflow does not call that root command. The separate **E2E Contract** and
**E2E Agent** workflows run each day and by manual dispatch. **E2E Workflow** is
manual-only. No workflow runs `pnpm eval` or `pnpm eval:snapshot`.

The **Test** workflow also runs the tool coverage check on every pull request,
outside turbo, so a change to a plugin's tools alone still runs it. **Tool
Check** runs the full tool check and is manual-only.

## Agent tool check

The tool check proves that every action of every agent tool works, at every
place an agent runs. It has two parts.

### Coverage check

```bash
pnpm --filter @sero/desktop exec vitest run electron/__tests__/tool-check/coverage.test.ts
```

It loads every agent tool and Sero CLI command from the code, then fails when an
action has neither a scenario nor an exemption, or when a scenario names an
action that no longer exists. It makes no model calls and takes seconds.

When you add a tool or an action, the coverage check fails until you add it to
the tool check:

1. Add a step for the action to the tool's probe module in
   `apps/desktop/e2e/helpers/tool-check/probes/`. A new tool gets its own module,
   listed in `apps/desktop/e2e/helpers/tool-check/table.ts`. Run mutating actions
   in order: create, show, change, delete.
2. Make the step's `expect` check the action's effect, such as a file, a record
   or a state change, not only that the call returned.
3. Build what the step needs in its `setup`. An exemption is allowed only for a
   paid outside service or a real outside account, with a sentence saying which.

### Full run

```bash
pnpm --filter @sero/desktop e2e:tools
```

It starts Sero on a fresh profile and workspace. A local stub model drives
every action, so the run needs no API key and costs nothing. The run does these
things:

- It runs each action at the first place an agent is given its tool: chat,
  subagent, Workflow worker, Architect owner or Room member.
- It runs each tool once at every one of those places, and in Code Mode scripts
  from a chat and from a subagent.
- In each Code Mode site, it checks that a failing call rejects and stops the
  script.
- It cancels a long command at every place and checks that the session takes
  its next prompt.

It writes `apps/desktop/test-results/tool-check/ledger-<runtime>.json`. The
last test compares that ledger with the actions the run should have proved and
lists every failure. A host run takes about four minutes.

| Variable                     | Effect                                                                                                                                   |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `SERO_E2E_RUNTIME`           | Workspace runtime: unset for host, `docker` on Linux, `apple-container` on a Mac.                                                        |
| `SERO_TOOL_CHECK_ONLY`       | A comma list of tools, such as `bash,sero git`, to check one fix. The ledger comparison then fails, because the other tools did not run. |
| `SERO_TOOL_CHECK_KEEP_HOME`  | `1` keeps the profile, the workspace and `test-results/tool-check/main-process.log` after the run.                                       |
| `SERO_TOOL_CHECK_REAL_MODEL` | `1` adds the real-model pass below. It needs `DEEPSEEK_API_KEY`, and is skipped without it.                                              |

### Real-model pass

With `SERO_TOOL_CHECK_REAL_MODEL=1`, the run ends with one short task on
`deepseek/deepseek-flash` at high thinking, at chat, Code Mode chat, subagent,
Code Mode subagent and a Workflow step. The model writes a file, reads it back
and runs `git status --short`. Each site passes on the file it wrote and on the
app's own record of the session's model and tool calls, not on what the model
says. The stub still drives every launch and planner, so a run costs under one
cent. Each site prints its model, its tool calls and its cost.

Room members and the Architect owner are not covered yet (#665). Their model is
fixed when the app starts.

To run only the real-model pass after building the desktop app:

```bash
SERO_TOOL_CHECK_REAL_MODEL=1 pnpm --filter @sero/desktop exec playwright test --project=tools --grep 'real-model pass'
```

This filtered run checks the five real-model tasks. It does not run the action
pass or compare the coverage ledger. The tools project does not retry failed
tests automatically.

### In CI

**Tool Check** (`.github/workflows/tool-check.yml`) never runs on a pull
request or a schedule. Start it from the Actions tab, or:

```bash
gh workflow run tool-check.yml --ref <branch> -f target=linux-host
```

The `target` is `all`, `linux-host`, `linux-docker`, `macos-host` or
`windows-host`. Add `-f real_model=true` for the real-model pass, which reads
the `DEEPSEEK_API_KEY` repository secret. Each target uploads its ledger and
Playwright report as an artifact.

To run only the five real-model tasks on macOS:

```bash
gh workflow run tool-check.yml --ref <branch> -f target=macos-host -f real_model=true -f grep='real-model pass'
```

The optional `grep` input filters test titles. Leave it empty for a full run.

## Evals command reference

| Command                   | Source script                                                                                                              | When to use                                                                   | Cost/auth                                                           |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `pnpm eval:snapshot`      | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-snapshot.yaml --no-cache`      | Fast prompt assembly/cache drift check                                        | No live LLM calls; low/no provider cost.                            |
| `pnpm eval:search`        | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-search.yaml --no-cache`        | Bash, FFF, Graphify, and combined search behavior, five runs per task and arm | Requires credentials and may cost money.                            |
| `pnpm eval:file-tools`    | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-file-tools.yaml --no-cache`    | Runtime-backed file-edit behavior, batching metrics, and result feedback      | Requires credentials and may cost money.                            |
| `pnpm eval:memory-save`   | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-memory-save.yaml --no-cache`   | Memory save-recall report: saved, missed and noise saves                      | Requires credentials and may cost money.                            |
| `pnpm eval:memory-search` | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-memory-search.yaml --no-cache` | Memory recall hit rate and false-hit rate per threshold                       | No LLM calls. Hybrid mode downloads the local embedding model once. |
| `pnpm eval`               | `node eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval`                                                       | Real agent behavior checks                                                    | Requires credentials and may cost money.                            |
| `pnpm eval:view`          | `node scripts/run-promptfoo.mjs view`                                                                                      | Inspect saved promptfoo results                                               | No new model calls.                                                 |

## Snapshot evals

Snapshot evals use `eval/promptfoo-snapshot.yaml` and `eval/snapshotProvider.ts`. They assemble an approximation of the full Sero session prompt from real prompt-building functions and check:

- SDK/base prompt block presence
- CLI prompt block presence
- container/subagent prompt guidance where applicable
- prompt block ordering for cache stability
- full prompt size against baseline
- metadata completeness

Run snapshot evals before committing changes to prompt assembly, CLI instructions, container prompt blocks, subagent guidance, or session setup.

## Real LLM evals

Real evals use `promptfooconfig.yaml` and `eval/seroProvider.ts`. They run through promptfoo with actual model calls. The default config uses the Sero provider with a 120s timeout and an Anthropic grading provider for rubric assertions.

The search eval uses `eval/promptfoo-search.yaml`. Its Bash, FFF, Graphify, and
combined arms receive the same seeded repository and tasks. It covers ranked
lookup, exhaustive lookup, dependency tracing, architecture explanation, and a
profile-wide lookup in a second workspace. The provider records tool names,
arguments, result text estimates, tool latency, total token usage, and total
latency.

The Graphify arms load the real plugin hooks and query engine. The fixture
seeds known graph files and profile state, so this eval does not run the Python
indexer. Graph commands use the bridged `sero-cli` model tool. The Bash and FFF
controls must report that profile-wide search is unavailable rather than search
outside the current workspace. Promptfoo runs cases serially so temporary
profiles and native indexes do not overlap across arms.

The file-tool eval sets `toolMode: runtime` in
`eval/promptfoo-file-tools.yaml`. The provider builds its session from Sero's
host file-tool factory through `eval/runtimeFileTools.ts`, so recorded calls
exercise the runtime `edit` and `write` tools rather than Pi's built-ins.
`eval/assertions/editBatching.ts` reports replacements per call, same-file edit
runs, calls by tool name, result tokens, latency, and failures.
`eval/seroProvider.test.ts` covers extension-loader isolation in the default and
runtime modes.

### Memory evals and metrics report

The memory checks measure memory; none of them gates a build.

- **Save-recall** (`eval/promptfoo-memory-save.yaml`, `eval/memorySaveProvider.ts`,
  `eval/assertions/memorySave.ts`). The provider plays each test's
  `conversation` in a real Pi session with the memory plugin and a
  memory-only `sero-cli` tool. Turns are separated by a line that holds only
  `---`. Each test lists moments, and each moment has marker words. A saved
  entry that holds all the markers of a moment counts for that moment, one
  entry per moment. Other entries, including a second save of the same fact,
  count as noise. The assertion reason holds the report line:
  `saved: … | missed: … | noise saves: …`. The model comes from the provider
  `model` config. Use `deepseek/deepseek-v4-flash` with `DEEPSEEK_API_KEY`,
  or `openai-codex/gpt-5.6-luna` on the OpenAI subscription.
- **Offline search** (`eval/promptfoo-memory-search.yaml`,
  `eval/memorySearchProvider.ts`, `eval/memory-search/`). The provider writes
  the fixed entries in `eval/memory-search/fixture.json` to a temporary
  profile and scores each fixture query with the plugin's own search. It
  reports the hit rate and false-hit rate for a range of thresholds, in
  keyword mode and hybrid mode. The test passes when the default threshold
  keeps the hit rate at 0.8 or more and the false-hit rate at 0.1 or less.
  The defaults, 0.6 for hybrid and 0.5 for keyword search, come from this
  report.
- **Live metrics report** (`scripts/memory-metrics-report.mjs`). The memory
  plugin writes one `metrics-YYYY-MM-DD.jsonl` file per day to
  `<SERO_HOME>/debug/memory/`. The script summarises them:

  ```bash
  node scripts/memory-metrics-report.mjs [--dir <path>] [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--json]
  ```

  A missing folder gives an empty report. Any other read error fails the
  script.

The memory plugin also has a test of the real search index under Electron's
Node: run `pnpm --filter @sero-ai/plugin-memory test:qmd`. Plain `pnpm test`
skips it.

Set `SERO_EVAL_MODEL` to use a specific model in all arms. Use a canonical
`provider/model` value, for example:

```bash
SERO_EVAL_MODEL=anthropic/claude-haiku-4-5 pnpm eval:search
```

If you do not set it, the eval uses the default model in the Sero agent settings.

Auth/cost notes:

- `pnpm eval` can consume paid provider tokens.
- It expects provider credentials such as `ANTHROPIC_API_KEY` from the shell or eval environment handling.
- The eval provider can apply env credentials as runtime API-key overrides before falling back to `~/.sero-ui/agent/auth.json`.
- Do not run live evals in CI or on PRs unless budget and credentials are explicitly intended.

## Scenario matrix

| Scenario file                          |                  Tests | Mode                          | Coverage                                                                                                                                          |
| -------------------------------------- | ---------------------: | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eval/scenarios/prompt-stability.yaml` |                      7 | Snapshot                      | Prompt block presence, ordering, size, and metadata.                                                                                              |
| `eval/scenarios/file-ops.yaml`         |                      3 | Real LLM                      | Create/read/edit file behavior and latency.                                                                                                       |
| `eval/scenarios/coding-tasks.yaml`     |                      3 | Real LLM                      | TypeScript/React generation, null-safety fixes, utility generation.                                                                               |
| `eval/scenarios/cli-ops.yaml`          |                      4 | Real LLM                      | `sero-cli` use for todos, workspace info, batch commands, and VCS status.                                                                         |
| `eval/scenarios/search-tools.yaml`     | 6 × 4 arms × 5 repeats | Real LLM                      | Task completion, tool choice, cross-workspace coverage, follow-up count, result size, and latency for Bash, FFF, Graphify, and their combination. |
| `eval/scenarios/file-edits.yaml`       |                      7 | Real LLM (runtime file tools) | Independent changes, a block move, duplicate text, a stale match, failure recovery, whole-file replacement, and identifier renaming.              |
| `eval/promptfoo-memory-save.yaml`      |                      5 | Real LLM                      | Memory saves for a correction, a preference, a decision with a reason, a surprise, and a conversation with nothing to save.                       |
| `eval/promptfoo-memory-search.yaml`    |            1 × 2 modes | Offline                       | Memory recall hit rate and false-hit rate per threshold, in keyword and hybrid mode.                                                              |

To add scenarios, create/edit a YAML file under `eval/scenarios/` and add it to the relevant promptfoo config.

## Failure interpretation

| Failure                                           | Likely next step                                                                                                  |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Snapshot says a block is missing                  | Inspect prompt assembly source and confirm the block is still intentionally included.                             |
| Snapshot ordering fails                           | Treat as cache-sensitive; confirm the prompt order change was intentional.                                        |
| Prompt size growth fails                          | Remove accidental verbosity or update the baseline with an intentional prompt change.                             |
| `pnpm eval` auth fails                            | Check env credentials and stale profile auth under `~/.sero-ui/agent/auth.json`.                                  |
| Real eval times out                               | Inspect provider latency and scenario complexity; adjust timeout only when justified.                             |
| Tool-sequence assertion fails                     | Inspect `context.providerResponse.metadata.toolCalls` in the result viewer.                                       |
| Search eval uses FFF for the exhaustive case      | Treat it as a contract failure; the model must use `bash` with `rg` when completeness matters.                    |
| Graphify command appears in a Bash call           | Treat it as a prompt or bridge failure; Graphify commands must use the `sero-cli` model tool.                     |
| Memory save eval reports a miss or noise          | Read the `saved`, `missed` and `noise saves` report line; the save rules are in the memory plugin's instructions. |
| Memory search eval fails at the default threshold | Read the per-threshold rows; a scoring change moved hits below the threshold or false hits above it.              |
| LLM rubric fails                                  | Read the output; rubrics are useful but can be noisy.                                                             |

## Relationship to other tests

| Risk area                          | Best current signal                                                | Notes                                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| Prompt assembly / cache stability  | `pnpm eval:snapshot`                                               | Low-cost check for prompt block drift, ordering drift, and size regressions.                           |
| Agent file-editing behavior        | `pnpm eval`                                                        | Exercises real tool use in isolated temp workspaces.                                                   |
| Runtime file-tool behavior         | `pnpm eval:file-tools`                                             | Runs the runtime `edit` and `write` tools with batching metrics and diff feedback.                     |
| Agent CLI usage patterns           | `pnpm eval`                                                        | Checks that the agent prefers `sero-cli` in supported scenarios.                                       |
| Memory saves and recall            | `pnpm eval:memory-save`, `pnpm eval:memory-search`, metrics report | Reports only; the metrics report measures real use.                                                    |
| Search and graph behavior          | `pnpm eval:search`                                                 | Compares task completion and tool choice; use the FFF plugin benchmark for model-free latency samples. |
| Desktop startup/session wiring     | desktop Vitest + Playwright CI                                     | Not primarily an eval concern.                                                                         |
| Plugin/runtime bridge regressions  | package tests + focused e2e                                        | Better covered by targeted source tests.                                                               |
| Container lifecycle/full-render UX | local/manual Playwright runs                                       | Environment-sensitive and not a generic promptfoo check.                                               |

## Related docs

- [Running Evals](/guide/running-evals.md)
- [Development Setup](/guide/development-setup.md)
- [Troubleshooting](/reference/troubleshooting.md)
