Evaluating accounting AI: how to test an agent on your real books
A demo shows what an agent can do once. A test on a closed quarter of your own books shows what it will do every day. Here is how to run one.
Manoj Tutika7 min read
Working on this yourself? Talk to Mark, our co-founder. We’re here to solve it with you.
On this page
You evaluate an accounting agent by replaying a period you have already closed and checked, letting the agent book it from the raw documents and bank lines, and comparing its result with what your accountant signed off. You measure how much it booked on its own, how often those bookings were right per account and per VAT code, and how far its trial balance and VAT return differ from yours, in euros.
If you are a controller, a CFO or a partner at an accounting firm, you have probably seen a demo where an agent books five invoices flawlessly. That tells you very little. Your books contain the supplier who changed entity, the subscription that is half personal, the credit note that refers to an invoice from last year. This post describes a test you can run on that reality, before you let any agent near your live ledger.
Why is a demo not an evaluation?
A demo is chosen by the vendor. The documents are clean, the suppliers are well known and the accounts are obvious. Your own books are different in three ways that matter.
- The mix is skewed. Most of your volume is repeat bookings that any tool gets right. The errors live in the long tail of one-off suppliers and unusual lines.
- The cost of errors is uneven. Putting €40 of software in the wrong expense account is a nuisance. Missing €4,000 of reverse-charge VAT is a correction on your VAT return.
- Your policies are specific. Capitalisation thresholds, cost-centre rules and how you treat mixed-use costs are yours, and no general model knows them.
An evaluation has to be built from your data and scored the way your errors actually cost you.
How do you build a golden set from your own books?
A golden set is a collection of past transactions where you know the right answer. The simplest source is a quarter that is closed, reviewed and filed. Its VAT return has gone in and your accountant has looked at it.
For each item in that quarter you keep two things: the input exactly as it arrived (the PDF, the bank line, the payment platform export) and the final booking after review (accounts, VAT codes, amounts, period, counterparty). The agent sees only the inputs. The bookings are the answer key.
Three rules make the difference between a useful golden set and a misleading one.
- Cut by time. The agent may learn from bookings before the test quarter, never from the quarter itself. If it can see the answer as a precedent, you are measuring memory, not judgement. This is the most common mistake in evaluating accounting software, and it flatters every result.
- Keep the mess. Include the credit notes, the duplicates your bookkeeper caught, the private expense paid with the company card. Removing them makes the test easy and the result worthless.
- Adjudicate disagreements. Your reference books are not perfect either. When the agent disagrees with the booking, someone qualified decides who was right before you count it as an error. Expect some disagreements to turn out to be inconsistencies in the original books rather than agent errors.
What should you measure?
A single accuracy percentage hides almost everything. We find five measures tell you what you need.
Coverage is the share of items the agent booked on its own instead of asking. Precision is the share of those autonomous bookings that were right. These two pull against each other: an agent that asks about everything has perfect precision and no value. You want high precision first and then as much coverage as the precision allows.
Per-field accuracy scores the account, the VAT code, the amount, the period and the counterparty separately. An agent can choose the right account with the wrong VAT code, and that is a different kind of problem.
Per-account accuracy breaks precision down by general ledger account. This is where weak spots show. An agent can be perfect on software subscriptions and unreliable on travel, and your overall number would never tell you.
Numeric diff compares the totals that matter: the trial balance per account and the boxes of your VAT return, agent books against reference books, in euros. This turns "eight errors" into "€412.00 in the wrong expense account and €37.80 of input VAT not reclaimed", which is what your CFO and your accountant care about.
What does a worked example look like?
Here is an illustrative replay of one quarter for a company with about 400 purchase documents and card transactions in it. The numbers are made up for this example, but they add up.
- The quarter holds 400 items.
- The agent booked 352 on its own, so coverage is 88.0%.
- It sent the other 48 to a person with a question.
- Of the 352 autonomous bookings, 344 matched the adjudicated answer, so precision is 97.7%.
- The remaining 8 autonomous bookings were wrong.
Broken down by account, the picture is uneven. Software and hosting is 118 of 118. Office supplies is 40 of 42, travel 55 of 57, professional fees 53 of 55, marketing 48 of 49 and literature 30 of 31. Six of the eight errors sit in three accounts where suppliers sell more than one kind of thing.
The numeric diff tells you what those eight errors cost. Office supplies is €312.00 too low, travel €150.70 too low (€188.50 that belonged there went elsewhere, partly offset by €37.80 of VAT that landed in it), professional fees €412.00 too high and marketing €88.50 too high. Input VAT is €37.80 too low, because one car rental invoice of €180.00 net was booked without its 21% VAT, putting the full €217.80 into travel. The differences net to zero across the trial balance, as they must, since every entry still balanced. But the VAT return would have reclaimed €37.80 less than it should. The one error in literature linked a correct booking to the wrong supplier, which moves no totals but would have shown up in your payables.
That one line is the most important result of the test. Misclassifications between expense accounts are a reporting issue you can fix at close. A wrong VAT code reaches a filing. It tells you exactly which rule or question to add before going live.
How do you keep an agent from getting worse?
An agent is not a fixed piece of software. Its rules change, the models underneath it change and your own policies change. Each of those can improve one account and quietly break another.
The answer is a regression gate: every change to the agent is replayed against the golden set before it reaches your books, and it only goes live if it passes thresholds you agreed in advance.
A sensible gate has three kinds of condition.
- Floors on precision overall and on the VAT code field, which should be strictest because VAT errors reach filings.
- No account gets worse. An improvement on average that makes one account worse should block the change, not pass it.
- Numeric limits. The VAT return boxes must match within a small tolerance, and the trial balance diff per account must stay below a materiality level you choose.
The golden set should also grow. Every correction you make in live use is a new test item with a known answer. After a few months, the set reflects your business better than any initial sample could.
At Thred we think about every agent this way: an agent earns autonomy on an account by its track record there, not by a general claim. That is also why each agent in Thred Layer can run in "Ask me first" or "Autonomous" mode, so you can start cautious and widen the scope as the numbers support it.
What should an accounting firm test differently?
If you run an accounting firm, the same method applies per client, with one addition. Test on several administrations at once, chosen to differ: a webshop, a consultancy, a holding with intercompany flows. An agent that does well on one client type can struggle on another, and per-client results show you where to start and where to wait.
Firms also have an advantage. Your closed and reviewed client files are an excellent golden set already, and your reviewers are the right people to adjudicate. Our page for accounting firms describes how Thred works across a client portfolio.
What questions should you ask a vendor?
Whoever builds the agent, these questions separate a serious answer from a sales answer.
- Can we replay a closed quarter of our own books before going live, with a time cut?
- Do you report precision and coverage separately, per account and per VAT code?
- Can you show the trial balance and VAT return diff in euros?
- What happens to an agent change that makes one account worse?
- Do our corrections become test cases?
If the answer to any of these is a single headline percentage, keep asking.
Books that are correct as they happen are only worth having if you can prove they are correct. To see how the Thred ledger holds up against your own closed quarter, book a demo.
Frequently asked questions
What is a golden set in accounting AI testing?
It is a set of past transactions where the correct booking is known, usually taken from a period that has been closed, reviewed and filed. The agent sees only the original inputs, and its bookings are scored against the reviewed answers.
Why is a single accuracy percentage misleading for bookkeeping software?
Most transactions are easy repeats, so a high overall score can hide weak accounts. Errors also differ in cost: a misclassified expense is fixed at close, while a wrong VAT code reaches a filing. Per-account, per-field and euro-based measures show where the real risk is.
What is the difference between coverage and precision?
Coverage is the share of items the agent booked without asking a person. Precision is the share of those autonomous bookings that were correct. You want high precision first, and then as much coverage as that precision allows.
How do you stop an accounting agent from getting worse after an update?
Replay every change against the golden set before it reaches your books and only release it if it meets agreed thresholds, such as minimum precision, minimum VAT code accuracy, no account getting worse and VAT return boxes matching within a small tolerance.
Can an accounting firm test an agent across several clients?
Yes, and it should. Pick closed files from clients that differ, such as a webshop, a consultancy and a holding, and score each separately. Results per client type show where an agent is ready and where it needs more rules.
Sources
Read next
- How we measure bookkeeping without a human: the 99.5% method
- Real-time books: what changes when your ledger is always current
- Exceptions are the real work: how finance teams should handle what automation can't
Where Thred comes in
Thred’s own general ledger, built for agents. Every transaction is booked as it happens, under your rules, and traceable back to its source.



