Step 6 of 6
Demo Motors, fictionalHow the checks score
These are recorded results from the repo’s eval scripts, not live API calls. One author wrote the rules and every test sample, so a perfect score here is a ceiling, not proof of real-world accuracy.
12 of 12
self-constructedFixture recall
Enforcement fixtures modeled on public cases. Each expected verdict was written before the rules.
uv run python ../evals/run.py, from engine/
5 of 5
self-constructedMutation catch rate
Five planted engine bugs, run over 202 synthetic deals. Certify rejected every deal it priced where a bug changed a number.
uv run python -m certified_offer mutate, from engine/
36 of 36
self-constructedDemo Motors misconduct recall
200 synthetic Queens deals with 36 planted violations. No honest deal was accused, 0 of 164.
uv run python ../evals/run.py, from engine/
55, 54, and 54 of 55
self-constructedAI-only baseline recall
claude-sonnet-5-5 judged 100 items with no engine in each of 3 runs: fixtures, Demo Motors deals, mutant worksheets, and honest worksheets. It missed a rounding error in 2 of 3 runs and called an honest worksheet wrong in every run. Precision 0.965, 0.964, and 0.982. 3 of 100 items changed verdict between runs. certified-offer caught all 55 and accused no honest item, the same answer every run. Honest items are deals the rulebook allows: three NY deals whose doc fees exceed the $175 cap are left out: their math is right, but certify holds them, so the label would depend on the rule, not the numbers.
Responses: evals/results/baseline-llm-2026-11-08.json. Rerun, about $3 with an Anthropic key: uv run --project engine python evals/baseline_llm.py --runs 3 --effort high, from the repo root.