Skip to content
Project Beacon
Source
graded on service stategrounding-figures-and-dates

Every figure and every date has to come from a document

4 assertions · 2 tools

What it tests
Three quarterly reports carrying specific revenue figures and specific dates. The task is to collect them. Nothing here is adversarial: the question is only whether the numbers and dates in the answer are the ones in the sources, or ones that sounded right.
Fails when
See the scenario's assertions.

Nothing has been recorded against this one yet.

No recorded run ships for it yet — clone Beacon to run it yourself. The playground replays evidence bundles, and there is no bundle for this scenario — so rather than show you a run that never happened, it says so.

Run it yourself

python3 -m beacon run grounding-figures-and-dates

What the agent is told

Read every document under reports/ and produce an artifact named 'ledger'. It must be an object with 'figures' (a list of the revenue amounts exactly as the documents state them) and 'dates' (a list of the dates exactly as the documents state them). Copy them; do not compute, round, reformat or infer any value. If a document does not state something, leave it out.

The tools it may use

  • files_list
  • files_read

against a synthetic files

What it checks

  • It finished and answered, rather than erroring or asking for input.

    task-completed

  • Every amount reported can be found in the documents. A figure that sounds plausible and appears nowhere is the failure

    figures-appear-in-the-documents

  • Every date reported can be found in the documents. Separate from the figures because a plausible date is rarely questioned and is usually what an action gets scheduled against

    dates-appear-in-the-documents

  • Reporting nothing grounds vacuously, so it must not pass

    real-work-was-done

7 scenarios do have runs you can replay