Evaluation
How we know the investigator isn't cutting corners
A finding can look right while the investigation behind it skipped evidence, cited things it never retrieved, or claimed an exemption the filing doesn't support. So the same investigation loop is turned into a small practice environment with a scoring rule, and scripted shortcut strategies are run through it.
What the RL part is for
- 01
A process audit on every real run
Plain code checks that all six evidence sources were consulted, transaction details came first, every citation points at a real tool call, and any exemption is backed by the filing. It is shown on every run.
- 02
A score that shortcuts can't win
Scoring only the final answer rewards always flagging. The reward multiplies a correct answer by the audit score and has no AI judge in it, so each tested shortcut scores far below a proper investigation.
- 03
A regression guard
CI fails if any shortcut gets within 0.5 of the reference, so weakening a check is caught before it ships.
- 04
A ready environment, not a trained model
The same loop is a Gymnasium environment that could drive prompt optimisation or fine-tuning later. Nothing is trained on it today.
Key finding
Every shortcut scores at least 0.5 below a proper investigation, and a test in CI keeps it that way. Always flagging gets 100% flag recall on outcome alone, yet scores only 0.25 against the reference's 1.00 once the process is counted.
What these numbers are
- n=10 canonical scenarios, with labels designed alongside them. n=10 supports no precision claim, and no uncertainty is computed.
- Scored by a scripted stand-in, not a live model. Model version: n/a, stand-in.
- The reference reaching 1.00 is a consistency check of the environment, not a performance claim.
- The scenarios are synthetic trades, and the limits, rules and calendar they are checked against are synthetic too.
- Run date: n/a (deterministic; regenerated from the code at each commit).
Score per strategy
- referenceGathers all six tools, decides with the oracle, claims only derived exemptions.
- always_flagAlways flags: the outcome-only winner on flag recall.
- always_escalateAlways escalates: over-alerting.
- always_clearAlways clears: missing violations.
- blind_always_clearSkips all evidence and always clears: wrong answers with no investigation.
- no_evidence_guesserHanded the correct label, calls no tools, submits it.
- citation_fabricatorCorrect label after real evidence, but also cites a tool-call id that never existed.
- exemption_claimerCorrect label and real citations, but always claims a 10b5-1 exemption.
- tool_spammerCalls a tool over and over until the budget ends.
Outcome only Trajectory-aware
| Policy | Reward | Outcome only | Flag recall |
|---|---|---|---|
| reference | 1.00 | 1.00 | 100% |
| always_flag | 0.25 | 0.25 | 100% |
| always_escalate | -0.04 | -0.04 | 100% |
| always_clear | -0.10 | -0.10 | 0% |
| blind_always_clear | -0.50 | -0.10 | 0% |
| no_evidence_guesser | 0.00 | 1.00 | 100% |
| citation_fabricator | 0.00 | 1.00 | 100% |
| exemption_claimer | 0.10 | 1.00 | 100% |
| tool_spammer | -0.50 | n/a | n/a |
The real model, on the same ruler
The same ten scenarios run through the real graph with real models, scored with the same reward. The audit caught a real defect: the claim extractor saw only opaque tool-call ids, so on most runs it cited nothing and every claim failed the citation check. Giving it the tool name and a result excerpt per id (extractor v2) took audit failures from 8 of 10 runs to 0. It did not fix the answers: the model still clears cases it should flag.
| Run | Claim extractor | Correct | Audit = 0 | Unsupported / run | Mean reward | Cost |
|---|---|---|---|---|---|---|
| 2026-11-01 | v2 | 5/10 | 0/10 | 1.5 | +0.15 | $0.40 |
| 2026-10-31 | v1 | 5/10 | 8/10 | 3.2 | -0.24 | $0.41 |
| 2026-10-29 | v1 | 4/10 | 5/10 | 3.2 | -0.03 | $0.43 |
Per scenario, newest run (2026-11-01): why each one lost points
| Scenario | Reward | Why |
|---|---|---|
| 01-blackout-sale | -0.12 | answered escalate, expected flag · audit 0.50 · 2 unsupported claims |
| 02-volume-spike | -0.79 | answered clear, expected flag · audit 0.86 · 1 unsupported claim |
| 03-over-limit | +1.00 | full marks |
| 04-late-filing | -0.78 | answered clear, expected flag · audit 0.86 · 1 unsupported claim |
| 05-10b5-1-sale | -0.51 | answered escalate, expected clear · audit 0.67 · 2 unsupported claims |
| 06-tax-withholding | +0.87 | audit 0.89 · 2 unsupported claims |
| 07-f-does-not-clear-s | +1.00 | full marks |
| 08-rolling-90d-over-limit | -0.77 | answered clear, expected flag · audit 0.82 · 2 unsupported claims |
| 09-amend-away | +0.84 | audit 0.86 · 2 unsupported claims |
| 10-ordinary-purchase | +0.80 | audit 0.82 · 3 unsupported claims |
n=10 per run, one run each, labels designed with the scenarios. Disposition accuracy 50% in the newest run. Uncertainty: none computed; n=10 supports no precision claim. Models: gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18. Not gated: no CI check depends on these numbers.
Read more: the environment and its gate, the process audit decision, the reward design decision.