For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/testing-evals.md.

Testing / Evals

Sero uses repository tests and promptfoo evals as separate quality signals. Not every suite runs for each pull request. Run real LLM evals only when their provider cost and credentials are intentional.

Current root command surface

pnpm typecheck
pnpm build
pnpm test
pnpm test:ci
pnpm eval:snapshot
pnpm eval:search
pnpm eval:file-tools
pnpm eval:memory-save
pnpm eval:memory-search
pnpm eval
pnpm eval:view

pnpm test runs turbo run test. Use the root commands above as the supported entry points.

CI workflows

The Test workflow runs for pushes to main, pull requests to main, and manual dispatch. It classifies changed paths. The affected jobs run pnpm typecheck, pnpm build, and pnpm test.

The root pnpm test:ci command also runs pnpm e2e:contract, but the Test workflow does not call that root command. The separate E2E Contract and E2E Agent workflows run each day and by manual dispatch. E2E Workflow is manual-only. No workflow runs pnpm eval or pnpm eval:snapshot.

The Test workflow also runs the tool coverage check on every pull request, outside turbo, so a change to a plugin's tools alone still runs it. Tool Check runs the full tool check and is manual-only.

Agent tool check

The tool check proves that every action of every agent tool works, at every place an agent runs. It has two parts.

Coverage check

pnpm --filter @sero/desktop exec vitest run electron/__tests__/tool-check/coverage.test.ts

It loads every agent tool and Sero CLI command from the code, then fails when an action has neither a scenario nor an exemption, or when a scenario names an action that no longer exists. It makes no model calls and takes seconds.

When you add a tool or an action, the coverage check fails until you add it to the tool check:

  1. Add a step for the action to the tool's probe module in apps/desktop/e2e/helpers/tool-check/probes/. A new tool gets its own module, listed in apps/desktop/e2e/helpers/tool-check/table.ts. Run mutating actions in order: create, show, change, delete.
  2. Make the step's expect check the action's effect, such as a file, a record or a state change, not only that the call returned.
  3. Build what the step needs in its setup. An exemption is allowed only for a paid outside service or a real outside account, with a sentence saying which.

Full run

pnpm --filter @sero/desktop e2e:tools

It starts Sero on a fresh profile and workspace. A local stub model drives every action, so the run needs no API key and costs nothing. The run does these things:

  • It runs each action at the first place an agent is given its tool: chat, subagent, Workflow worker, Architect owner or Room member.
  • It runs each tool once at every one of those places, and in Code Mode scripts from a chat and from a subagent.
  • In each Code Mode site, it checks that a failing call rejects and stops the script.
  • It cancels a long command at every place and checks that the session takes its next prompt.

It writes apps/desktop/test-results/tool-check/ledger-<runtime>.json. The last test compares that ledger with the actions the run should have proved and lists every failure. A host run takes about four minutes.

VariableEffect
SERO_E2E_RUNTIMEWorkspace runtime: unset for host, docker on Linux, apple-container on a Mac.
SERO_TOOL_CHECK_ONLYA comma list of tools, such as bash,sero git, to check one fix. The ledger comparison then fails, because the other tools did not run.
SERO_TOOL_CHECK_KEEP_HOME1 keeps the profile, the workspace and test-results/tool-check/main-process.log after the run.
SERO_TOOL_CHECK_REAL_MODEL1 adds the real-model pass below. It needs DEEPSEEK_API_KEY, and is skipped without it.

Real-model pass

With SERO_TOOL_CHECK_REAL_MODEL=1, the run ends with one short task on deepseek/deepseek-flash at high thinking, at chat, Code Mode chat, subagent, Code Mode subagent and a Workflow step. The model writes a file, reads it back and runs git status --short. Each site passes on the file it wrote and on the app's own record of the session's model and tool calls, not on what the model says. The stub still drives every launch and planner, so a run costs under one cent. Each site prints its model, its tool calls and its cost.

Room members and the Architect owner are not covered yet (#665). Their model is fixed when the app starts.

To run only the real-model pass after building the desktop app:

SERO_TOOL_CHECK_REAL_MODEL=1 pnpm --filter @sero/desktop exec playwright test --project=tools --grep 'real-model pass'

This filtered run checks the five real-model tasks. It does not run the action pass or compare the coverage ledger. The tools project does not retry failed tests automatically.

In CI

Tool Check (.github/workflows/tool-check.yml) never runs on a pull request or a schedule. Start it from the Actions tab, or:

gh workflow run tool-check.yml --ref <branch> -f target=linux-host

The target is all, linux-host, linux-docker, macos-host or windows-host. Add -f real_model=true for the real-model pass, which reads the DEEPSEEK_API_KEY repository secret. Each target uploads its ledger and Playwright report as an artifact.

To run only the five real-model tasks on macOS:

gh workflow run tool-check.yml --ref <branch> -f target=macos-host -f real_model=true -f grep='real-model pass'

The optional grep input filters test titles. Leave it empty for a full run.

Evals command reference

CommandSource scriptWhen to useCost/auth
pnpm eval:snapshotnode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-snapshot.yaml --no-cacheFast prompt assembly/cache drift checkNo live LLM calls; low/no provider cost.
pnpm eval:searchnode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-search.yaml --no-cacheBash, FFF, Graphify, and combined search behavior, five runs per task and armRequires credentials and may cost money.
pnpm eval:file-toolsnode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-file-tools.yaml --no-cacheRuntime-backed file-edit behavior, batching metrics, and result feedbackRequires credentials and may cost money.
pnpm eval:memory-savenode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-memory-save.yaml --no-cacheMemory save-recall report: saved, missed and noise savesRequires credentials and may cost money.
pnpm eval:memory-searchnode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs eval --config eval/promptfoo-memory-search.yaml --no-cacheMemory recall hit rate and false-hit rate per thresholdNo LLM calls. Hybrid mode downloads the local embedding model once.
pnpm evalnode eval/patch-drizzle.cjs && node scripts/run-promptfoo.mjs evalReal agent behavior checksRequires credentials and may cost money.
pnpm eval:viewnode scripts/run-promptfoo.mjs viewInspect saved promptfoo resultsNo new model calls.

Snapshot evals

Snapshot evals use eval/promptfoo-snapshot.yaml and eval/snapshotProvider.ts. They assemble an approximation of the full Sero session prompt from real prompt-building functions and check:

  • SDK/base prompt block presence
  • CLI prompt block presence
  • container/subagent prompt guidance where applicable
  • prompt block ordering for cache stability
  • full prompt size against baseline
  • metadata completeness

Run snapshot evals before committing changes to prompt assembly, CLI instructions, container prompt blocks, subagent guidance, or session setup.

Real LLM evals

Real evals use promptfooconfig.yaml and eval/seroProvider.ts. They run through promptfoo with actual model calls. The default config uses the Sero provider with a 120s timeout and an Anthropic grading provider for rubric assertions.

The search eval uses eval/promptfoo-search.yaml. Its Bash, FFF, Graphify, and combined arms receive the same seeded repository and tasks. It covers ranked lookup, exhaustive lookup, dependency tracing, architecture explanation, and a profile-wide lookup in a second workspace. The provider records tool names, arguments, result text estimates, tool latency, total token usage, and total latency.

The Graphify arms load the real plugin hooks and query engine. The fixture seeds known graph files and profile state, so this eval does not run the Python indexer. Graph commands use the bridged sero-cli model tool. The Bash and FFF controls must report that profile-wide search is unavailable rather than search outside the current workspace. Promptfoo runs cases serially so temporary profiles and native indexes do not overlap across arms.

The file-tool eval sets toolMode: runtime in eval/promptfoo-file-tools.yaml. The provider builds its session from Sero's host file-tool factory through eval/runtimeFileTools.ts, so recorded calls exercise the runtime edit and write tools rather than Pi's built-ins. eval/assertions/editBatching.ts reports replacements per call, same-file edit runs, calls by tool name, result tokens, latency, and failures. eval/seroProvider.test.ts covers extension-loader isolation in the default and runtime modes.

Memory evals and metrics report

The memory checks measure memory; none of them gates a build.

  • Save-recall (eval/promptfoo-memory-save.yaml, eval/memorySaveProvider.ts, eval/assertions/memorySave.ts). The provider plays each test's conversation in a real Pi session with the memory plugin and a memory-only sero-cli tool. Turns are separated by a line that holds only ---. Each test lists moments, and each moment has marker words. A saved entry that holds all the markers of a moment counts for that moment, one entry per moment. Other entries, including a second save of the same fact, count as noise. The assertion reason holds the report line: saved: … | missed: … | noise saves: …. The model comes from the provider model config. Use deepseek/deepseek-v4-flash with DEEPSEEK_API_KEY, or openai-codex/gpt-5.6-luna on the OpenAI subscription.

  • Offline search (eval/promptfoo-memory-search.yaml, eval/memorySearchProvider.ts, eval/memory-search/). The provider writes the fixed entries in eval/memory-search/fixture.json to a temporary profile and scores each fixture query with the plugin's own search. It reports the hit rate and false-hit rate for a range of thresholds, in keyword mode and hybrid mode. The test passes when the default threshold keeps the hit rate at 0.8 or more and the false-hit rate at 0.1 or less. The defaults, 0.6 for hybrid and 0.5 for keyword search, come from this report.

  • Live metrics report (scripts/memory-metrics-report.mjs). The memory plugin writes one metrics-YYYY-MM-DD.jsonl file per day to <SERO_HOME>/debug/memory/. The script summarises them:

    node scripts/memory-metrics-report.mjs [--dir <path>] [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--json]

    A missing folder gives an empty report. Any other read error fails the script.

The memory plugin also has a test of the real search index under Electron's Node: run pnpm --filter @sero-ai/plugin-memory test:qmd. Plain pnpm test skips it.

Set SERO_EVAL_MODEL to use a specific model in all arms. Use a canonical provider/model value, for example:

SERO_EVAL_MODEL=anthropic/claude-haiku-4-5 pnpm eval:search

If you do not set it, the eval uses the default model in the Sero agent settings.

Auth/cost notes:

  • pnpm eval can consume paid provider tokens.
  • It expects provider credentials such as ANTHROPIC_API_KEY from the shell or eval environment handling.
  • The eval provider can apply env credentials as runtime API-key overrides before falling back to ~/.sero-ui/agent/auth.json.
  • Do not run live evals in CI or on PRs unless budget and credentials are explicitly intended.

Scenario matrix

Scenario fileTestsModeCoverage
eval/scenarios/prompt-stability.yaml7SnapshotPrompt block presence, ordering, size, and metadata.
eval/scenarios/file-ops.yaml3Real LLMCreate/read/edit file behavior and latency.
eval/scenarios/coding-tasks.yaml3Real LLMTypeScript/React generation, null-safety fixes, utility generation.
eval/scenarios/cli-ops.yaml4Real LLMsero-cli use for todos, workspace info, batch commands, and VCS status.
eval/scenarios/search-tools.yaml6 × 4 arms × 5 repeatsReal LLMTask completion, tool choice, cross-workspace coverage, follow-up count, result size, and latency for Bash, FFF, Graphify, and their combination.
eval/scenarios/file-edits.yaml7Real LLM (runtime file tools)Independent changes, a block move, duplicate text, a stale match, failure recovery, whole-file replacement, and identifier renaming.
eval/promptfoo-memory-save.yaml5Real LLMMemory saves for a correction, a preference, a decision with a reason, a surprise, and a conversation with nothing to save.
eval/promptfoo-memory-search.yaml1 × 2 modesOfflineMemory recall hit rate and false-hit rate per threshold, in keyword and hybrid mode.

To add scenarios, create/edit a YAML file under eval/scenarios/ and add it to the relevant promptfoo config.

Failure interpretation

FailureLikely next step
Snapshot says a block is missingInspect prompt assembly source and confirm the block is still intentionally included.
Snapshot ordering failsTreat as cache-sensitive; confirm the prompt order change was intentional.
Prompt size growth failsRemove accidental verbosity or update the baseline with an intentional prompt change.
pnpm eval auth failsCheck env credentials and stale profile auth under ~/.sero-ui/agent/auth.json.
Real eval times outInspect provider latency and scenario complexity; adjust timeout only when justified.
Tool-sequence assertion failsInspect context.providerResponse.metadata.toolCalls in the result viewer.
Search eval uses FFF for the exhaustive caseTreat it as a contract failure; the model must use bash with rg when completeness matters.
Graphify command appears in a Bash callTreat it as a prompt or bridge failure; Graphify commands must use the sero-cli model tool.
Memory save eval reports a miss or noiseRead the saved, missed and noise saves report line; the save rules are in the memory plugin's instructions.
Memory search eval fails at the default thresholdRead the per-threshold rows; a scoring change moved hits below the threshold or false hits above it.
LLM rubric failsRead the output; rubrics are useful but can be noisy.

Relationship to other tests

Risk areaBest current signalNotes
Prompt assembly / cache stabilitypnpm eval:snapshotLow-cost check for prompt block drift, ordering drift, and size regressions.
Agent file-editing behaviorpnpm evalExercises real tool use in isolated temp workspaces.
Runtime file-tool behaviorpnpm eval:file-toolsRuns the runtime edit and write tools with batching metrics and diff feedback.
Agent CLI usage patternspnpm evalChecks that the agent prefers sero-cli in supported scenarios.
Memory saves and recallpnpm eval:memory-save, pnpm eval:memory-search, metrics reportReports only; the metrics report measures real use.
Search and graph behaviorpnpm eval:searchCompares task completion and tool choice; use the FFF plugin benchmark for model-free latency samples.
Desktop startup/session wiringdesktop Vitest + Playwright CINot primarily an eval concern.
Plugin/runtime bridge regressionspackage tests + focused e2eBetter covered by targeted source tests.
Container lifecycle/full-render UXlocal/manual Playwright runsEnvironment-sensitive and not a generic promptfoo check.