Accounting tiles arranged on a paper grid, with one copper tile crossing a column boundary and a separate checkmark.
Correct arithmetic still leaves the question of what belongs in each calculation.Sint

Breakdown / AI at work · Finance

The AI's spreadsheet recalculated. It still failed the financial checks.

A public finance task shows the gap between a working workbook and a financial conclusion that follows the evidence.

An AI agent submitted a financial workbook after checking its calculations. Its log reported 818 formulas, no missing or erroneous saved results, and no differences after recalculating the file. The evaluator then passed eight of fifteen checks. Published Opus 5.5 work record

The formulas worked. The harder question was whether they were calculating the right things.

The assignment went beyond the arithmetic

The example comes from Bellwood, a finance task in SemiAnalysis's GDPx benchmark, developed with Living Context. The agent had to finish an existing acquisition-review workbook, preserve its structure and support its changes with evidence. The task covered earnings adjustments, working capital and debt, using the definitions in a draft purchase agreement. Public assignment

Working capital covers operating balances such as customer bills still to be paid and money owed to suppliers. An acquisition agreement can specify which balances count and how they affect the price.

Think of checking a household budget. Every total can add up correctly even if you count a one-off refund as regular income or put the same bill in two categories. The arithmetic check and the judgment about what belongs in the calculation answer different questions.

In the inspected Opus attempt, the published evaluator flagged problems including cash reconciliation and how adjustments carried through to monthly working capital. The agent's final message also listed judgments for a person to confirm and work it hadn't covered. Its own caveats belong beside its completion claim. Work and verifier record

One disputed fee shows why the details matter

We also inspected a GPT-6 Astra attempt. Its evaluator objected to an estimated $825,000 transaction expense appearing in the net-debt treatment when the agreement deducted transaction expenses separately from the purchase price. Astra's later notes and final message did distinguish that expense and warn against counting it twice. The dispute is over the delivered workbook's treatment. The agent did notice the distinction, and this test provides no evidence of an actual financial loss. Published Astra work record

Our reporting examined the published transcripts and verifier output. We haven't independently audited the source documents, opened the finished workbooks or rerun the grader. We'd need those checks before deciding whether the grader's accounting judgment was right.

There is also a reporting gap in the benchmark. The launch article describes 25 Bellwood attempts assessed against 11 checks. The two public transcripts we inspected each contain 15 evaluator checks. The inspected material doesn't explain the difference. We haven't combined those results or used the two attempts to rank models. Launch article

For someone trying this kind of automation, the useful acceptance test starts with the original assignment. Does each adjustment have support? Does it follow the agreement's definition? Does it appear in every place the workbook requires, without being counted twice?

Checking that a file opens and its formulas run is a necessary step. The person relying on the result still needs to know that the workbook says the right thing.

Sources