
Breakdown / AI at work · Finance
The AI's spreadsheet recalculated. It still failed the financial checks.
A public finance task shows the gap between a working workbook and a financial conclusion that follows the evidence.
An AI agent submitted a financial workbook after checking its calculations. Its log reported 818 formulas, no missing or erroneous saved results, and no differences after recalculating the file. The evaluator then passed eight of fifteen checks. Published Opus 5.5 work record
The formulas worked. The harder question was whether they were calculating the right things.
The assignment went beyond the arithmetic
The example comes from Bellwood, a finance task in SemiAnalysis's GDPx benchmark, developed with Living Context. The agent had to finish an existing acquisition-review workbook, preserve its structure and support its changes with evidence. The task covered earnings adjustments, working capital and debt, using the definitions in a draft purchase agreement. Public assignment
Working capital covers operating balances such as customer bills still to be paid and money owed to suppliers. An acquisition agreement can specify which balances count and how they affect the price.
Think of checking a household budget. Every total can add up correctly even if you count a one-off refund as regular income or put the same bill in two categories. The arithmetic check and the judgment about what belongs in the calculation answer different questions.
In the inspected Opus attempt, the published evaluator flagged problems including cash reconciliation and how adjustments carried through to monthly working capital. The agent's final message also listed judgments for a person to confirm and work it hadn't covered. Its own caveats belong beside its completion claim. Work and verifier record
One disputed fee shows why the details matter
We also inspected a GPT-6 Astra attempt. Its evaluator objected to an estimated $825,000 transaction expense appearing in the net-debt treatment when the agreement deducted transaction expenses separately from the purchase price. Astra's later notes and final message did distinguish that expense and warn against counting it twice. The dispute is over the delivered workbook's treatment. The agent did notice the distinction, and this test provides no evidence of an actual financial loss. Published Astra work record
Our reporting examined the published transcripts and verifier output. We haven't independently audited the source documents, opened the finished workbooks or rerun the grader. We'd need those checks before deciding whether the grader's accounting judgment was right.
There is also a reporting gap in the benchmark. The launch article describes 25 Bellwood attempts assessed against 11 checks. The two public transcripts we inspected each contain 15 evaluator checks. The inspected material doesn't explain the difference. We haven't combined those results or used the two attempts to rank models. Launch article
For someone trying this kind of automation, the useful acceptance test starts with the original assignment. Does each adjustment have support? Does it follow the agreement's definition? Does it appear in every place the workbook requires, without being counted twice?
Checking that a file opens and its formulas run is a necessary step. The person relying on the result still needs to know that the workbook says the right thing.
Sources
- Bellwood finance assignmentSemiAnalysis / Living Context
- Work and verifier recordSemiAnalysis / Living Context
- Published Astra work recordSemiAnalysis / Living Context
- Launch articleSemiAnalysis