← jason fetzerShift Investigatorthe golden set, case by case
Back to the investigatorDocsSynthetic data / no model calls

golden evals

How the numbers are checked

Every case below is a question with a written expectation: which tools must run with which arguments, what the answer must and must not say, and for the value cases what each number must equal when computed a second way. Open a case to run it on this server and see the tool trail it actually produced.

Score

JSON

Loading the golden set…

Offline cases pass or fail one by one. A live case is a recording of a real model’s tool choices, replayed here through the same agent, tools and validators; its type passes by pass rate. This page never calls a model: “Run” executes the offline planner or replays the recording.

Trend

One point per tracked run, from evals/history.jsonl as shipped with this build. The file changes only through a pull request; nothing on this page writes to it.

Narrative judge

The checks above score numbers and routing; nothing in them reads a model's prose. A DeepEval G-Eval judge scores each narrative a recorded run writes on five rubric lines, 0 or 1 each, never blended. It is a model call, so it runs only from the owner's machine (evals/run.py --judge), never in the CI gate and never from this page, and it is trusted only as far as it agrees with the owner's own grades of the same narratives.

Cases