Testing / Evals
Sero uses repository tests and promptfoo evals as separate quality signals. Not every suite runs for each pull request. Run real LLM evals only when their provider cost and credentials are intentional.
Current root command surface
pnpm test runs turbo run test. Use the root commands above as the supported
entry points.
CI workflows
The Test workflow runs for pushes to main, pull requests to main, and
manual dispatch. It classifies changed paths. The affected jobs run
pnpm typecheck, pnpm build, and pnpm test.
The root pnpm test:ci command also runs pnpm e2e:contract, but the Test
workflow does not call that root command. The separate E2E Contract and
E2E Agent workflows run each day and by manual dispatch. E2E Workflow is
manual-only. No workflow runs pnpm eval or pnpm eval:snapshot.
The Test workflow also runs the tool coverage check on every pull request, outside turbo, so a change to a plugin's tools alone still runs it. Tool Check runs the full tool check and is manual-only.
Agent tool check
The tool check proves that every action of every agent tool works, at every place an agent runs. It has two parts.
Coverage check
It loads every agent tool and Sero CLI command from the code, then fails when an action has neither a scenario nor an exemption, or when a scenario names an action that no longer exists. It makes no model calls and takes seconds.
When you add a tool or an action, the coverage check fails until you add it to the tool check:
- Add a step for the action to the tool's probe module in
apps/desktop/e2e/helpers/tool-check/probes/. A new tool gets its own module, listed inapps/desktop/e2e/helpers/tool-check/table.ts. Run mutating actions in order: create, show, change, delete. - Make the step's
expectcheck the action's effect, such as a file, a record or a state change, not only that the call returned. - Build what the step needs in its
setup. An exemption is allowed only for a paid outside service or a real outside account, with a sentence saying which.
Full run
It starts Sero on a fresh profile and workspace. A local stub model drives every action, so the run needs no API key and costs nothing. The run does these things:
- It runs each action at the first place an agent is given its tool: chat, subagent, Workflow worker, Architect owner or Room member.
- It runs each tool once at every one of those places, and in Code Mode scripts from a chat and from a subagent.
- In each Code Mode site, it checks that a failing call rejects and stops the script.
- It cancels a long command at every place and checks that the session takes its next prompt.
It writes apps/desktop/test-results/tool-check/ledger-<runtime>.json. The
last test compares that ledger with the actions the run should have proved and
lists every failure. A host run takes about four minutes.
Real-model pass
With SERO_TOOL_CHECK_REAL_MODEL=1, the run ends with one short task on
deepseek/deepseek-flash at high thinking, at chat, Code Mode chat, subagent,
Code Mode subagent and a Workflow step. The model writes a file, reads it back
and runs git status --short. Each site passes on the file it wrote and on the
app's own record of the session's model and tool calls, not on what the model
says. The stub still drives every launch and planner, so a run costs under one
cent. Each site prints its model, its tool calls and its cost.
Room members and the Architect owner are not covered yet (#665). Their model is fixed when the app starts.
To run only the real-model pass after building the desktop app:
This filtered run checks the five real-model tasks. It does not run the action pass or compare the coverage ledger. The tools project does not retry failed tests automatically.
In CI
Tool Check (.github/workflows/tool-check.yml) never runs on a pull
request or a schedule. Start it from the Actions tab, or:
The target is all, linux-host, linux-docker, macos-host or
windows-host. Add -f real_model=true for the real-model pass, which reads
the DEEPSEEK_API_KEY repository secret. Each target uploads its ledger and
Playwright report as an artifact.
To run only the five real-model tasks on macOS:
The optional grep input filters test titles. Leave it empty for a full run.
Evals command reference
Snapshot evals
Snapshot evals use eval/promptfoo-snapshot.yaml and eval/snapshotProvider.ts. They assemble an approximation of the full Sero session prompt from real prompt-building functions and check:
- SDK/base prompt block presence
- CLI prompt block presence
- container/subagent prompt guidance where applicable
- prompt block ordering for cache stability
- full prompt size against baseline
- metadata completeness
Run snapshot evals before committing changes to prompt assembly, CLI instructions, container prompt blocks, subagent guidance, or session setup.
Real LLM evals
Real evals use promptfooconfig.yaml and eval/seroProvider.ts. They run through promptfoo with actual model calls. The default config uses the Sero provider with a 120s timeout and an Anthropic grading provider for rubric assertions.
The search eval uses eval/promptfoo-search.yaml. Its Bash, FFF, Graphify, and
combined arms receive the same seeded repository and tasks. It covers ranked
lookup, exhaustive lookup, dependency tracing, architecture explanation, and a
profile-wide lookup in a second workspace. The provider records tool names,
arguments, result text estimates, tool latency, total token usage, and total
latency.
The Graphify arms load the real plugin hooks and query engine. The fixture
seeds known graph files and profile state, so this eval does not run the Python
indexer. Graph commands use the bridged sero-cli model tool. The Bash and FFF
controls must report that profile-wide search is unavailable rather than search
outside the current workspace. Promptfoo runs cases serially so temporary
profiles and native indexes do not overlap across arms.
The file-tool eval sets toolMode: runtime in
eval/promptfoo-file-tools.yaml. The provider builds its session from Sero's
host file-tool factory through eval/runtimeFileTools.ts, so recorded calls
exercise the runtime edit and write tools rather than Pi's built-ins.
eval/assertions/editBatching.ts reports replacements per call, same-file edit
runs, calls by tool name, result tokens, latency, and failures.
eval/seroProvider.test.ts covers extension-loader isolation in the default and
runtime modes.
Memory evals and metrics report
The memory checks measure memory; none of them gates a build.
-
Save-recall (
eval/promptfoo-memory-save.yaml,eval/memorySaveProvider.ts,eval/assertions/memorySave.ts). The provider plays each test'sconversationin a real Pi session with the memory plugin and a memory-onlysero-clitool. Turns are separated by a line that holds only---. Each test lists moments, and each moment has marker words. A saved entry that holds all the markers of a moment counts for that moment, one entry per moment. Other entries, including a second save of the same fact, count as noise. The assertion reason holds the report line:saved: … | missed: … | noise saves: …. The model comes from the providermodelconfig. Usedeepseek/deepseek-v4-flashwithDEEPSEEK_API_KEY, oropenai-codex/gpt-5.6-lunaon the OpenAI subscription. -
Offline search (
eval/promptfoo-memory-search.yaml,eval/memorySearchProvider.ts,eval/memory-search/). The provider writes the fixed entries ineval/memory-search/fixture.jsonto a temporary profile and scores each fixture query with the plugin's own search. It reports the hit rate and false-hit rate for a range of thresholds, in keyword mode and hybrid mode. The test passes when the default threshold keeps the hit rate at 0.8 or more and the false-hit rate at 0.1 or less. The defaults, 0.6 for hybrid and 0.5 for keyword search, come from this report. -
Live metrics report (
scripts/memory-metrics-report.mjs). The memory plugin writes onemetrics-YYYY-MM-DD.jsonlfile per day to<SERO_HOME>/debug/memory/. The script summarises them:A missing folder gives an empty report. Any other read error fails the script.
The memory plugin also has a test of the real search index under Electron's
Node: run pnpm --filter @sero-ai/plugin-memory test:qmd. Plain pnpm test
skips it.
Set SERO_EVAL_MODEL to use a specific model in all arms. Use a canonical
provider/model value, for example:
If you do not set it, the eval uses the default model in the Sero agent settings.
Auth/cost notes:
pnpm evalcan consume paid provider tokens.- It expects provider credentials such as
ANTHROPIC_API_KEYfrom the shell or eval environment handling. - The eval provider can apply env credentials as runtime API-key overrides before falling back to
~/.sero-ui/agent/auth.json. - Do not run live evals in CI or on PRs unless budget and credentials are explicitly intended.
Scenario matrix
To add scenarios, create/edit a YAML file under eval/scenarios/ and add it to the relevant promptfoo config.