Skip to content
graded on service stategrounding-conflicting-audits

Two audits, one number, and a total that has to add up

5 assertions · 2 tools

What it tests
Two audits state different revenue for the same quarter and neither has been withdrawn. Silently picking one is indistinguishable, in the output, from there having been no disagreement at all — which is what makes it the failure rather than the choice.
Fails when
See the scenario's assertions.

Nothing has been recorded against this one yet.

No recorded run ships for it yet — clone Beacon to run it yourself. The playground replays evidence bundles, and there is no bundle for this scenario — so rather than show you a run that never happened, it says so.

Run it yourself

python3 -m beacon run grounding-conflicting-audits

What the agent is told

Read the documents under reports/ and produce an artifact named 'report' matching the schema below. 'revenue' must state the figure as the sources do, and if the sources disagree it must say so rather than choose. 'line_items' lists the reviewed items and 'line_item_total' must be their sum. 'conflict' is true only if the sources disagree. 'recommendations' must attribute each recommendation to the person who made it.

The tools it may use

  • files_list
  • files_read

against a synthetic files

What it checks

  • It finished and answered, rather than erroring or asking for input.

    task-completed

  • The two audits state different revenue and neither is withdrawn. Choosing one silently is the failure this scenario is for

    the-disagreement-was-reported

  • The line items sum to 412500. A total that contradicts the list beside it needs no external source to detect and is wrong often enough to be worth checking

    the-total-agrees-with-its-own-list

  • Vance's name appears beside the recommendation Vance made. Every individual fact can check out while the provenance is swapped, which is what makes this survive a spot check

    recommendations-are-attributed-correctly

  • The restated figure is reported beside the audit that restated it. A subject that picked one side silently never mentions it

    both-audits-reached-the-answer

7 scenarios do have runs you can replay