Thred
All posts

Duplicates, retries and idempotency: keeping the ledger right when data arrives twice

Bank feeds, payment platforms and syncs all deliver some data twice. Here is how a ledger should recognise a repeat, book it once and prove it with the bank balance.

Manoj Tutika7 min read

Working on this yourself? Talk to Mark, our co-founder. We’re here to solve it with you.

Book a demo
On this page

Idempotency means that processing the same piece of data twice leaves your books exactly as if it had been processed once. For a ledger fed by bank feeds, payment platforms and integrations, it is the property that stops a re-delivered bank line, a retried payout notification or a twice-sent invoice from being booked twice.

If you are a controller or finance lead, you have met the symptom even if not the word. The bank account in the ledger is €1,728.00 off from the bank. Someone spends an afternoon scrolling, finds a software subscription booked twice in March, and deletes one. This post explains where those duplicates come from, how a well-built ledger recognises them before they land, and what you can check yourself.

Why does the same transaction arrive twice?

Duplicates are not a bug in one tool. They are a normal property of how financial data moves between systems. Most integrations promise "at least once" delivery, which is a polite way of saying "sometimes more than once". The usual sources are these.

  • Overlapping bank syncs. A feed reconnects after a consent renewal and re-fetches the last 30 or 90 days to be safe. Every line in that window comes in again.
  • Two channels for one account. A daily bank feed and a statement file (in the EU often a CAMT.053 file, the ISO 20022 statement format) both carry the same lines, sometimes with slightly different descriptions.
  • Pending becomes booked. A card payment appears as pending, then reappears days later as booked, sometimes with a different date or, for currency conversions and hotel deposits, a different amount.
  • Retried notifications. Payment platforms resend a notification if your side does not confirm receipt in time. Stripe, for example, documents that it retries undelivered webhook events for up to three days in live mode.
  • Lost confirmations. Your system writes an entry to the accounting software, the connection drops before the reply arrives, and the job tries again. The first write had succeeded.
  • The same invoice, sent twice. A supplier emails the PDF and later uploads it to a portal, or sends a reminder with the invoice attached.

None of these can be prevented at the source. So the ledger has to be built on the assumption that it will see every piece of data more than once.

What is a transaction key, and why does it matter?

To recognise a repeat, you first need to say what makes two records "the same transaction". That definition is the transaction key: a small set of fields that together identify one real-world event, and that do not change when the data is re-sent.

The best key is an identifier the source gives you and promises to keep stable. Banks include references in their statements. ISO 20022 has an end-to-end identification that travels with a SEPA transfer from payer to payee, and a reference the bank assigns to each entry. Payment platforms give every charge, refund and payout an ID. When such an identifier exists, use it.

When it does not, you build a fingerprint from fields that should not change: the bank account (IBAN), the booking date, the amount in cents, the currency, the counterparty's IBAN and the payment reference, after normalising spaces and case. A fingerprint is just those fields combined into one value that can be compared quickly.

Fingerprints have one trap. Two real transactions can be identical. Two €4.50 coffees on the same card on the same day share every field. A naive rule that drops identical lines would delete a real expense. The fix is to count occurrences: the first €4.50 line of that day in that statement is occurrence 1, the second is occurrence 2. A re-delivered statement produces the same two keys, so nothing duplicates. A third genuine coffee produces occurrence 3, so nothing is lost.

How should a ledger make posting idempotent?

Recognising a repeat at the door is half the job. The other half is making sure every step after it is safe to run again. We think of it as three guards, each in a different place.

1. One source record per key

Every incoming bank line, payout, invoice or payment is stored once under its key. The database itself refuses a second record with the same key, so even two processes running at the same moment cannot both insert it. A repeat with new information, such as a pending card payment that is now booked, updates the existing record instead of creating a second one.

2. One journal entry per source record and purpose

A bank line produces at most one entry for "record this payment", whichever agent, rule or person touches it. If the posting job crashes and restarts, it finds the entry already there and stops. This is the guard that makes retries harmless.

3. Writes to other systems carry your own reference

When a finished entry is pushed to an external ledger or payment API, the request carries a unique reference generated before the first attempt, often called an idempotency key. Stripe's API, for instance, accepts an Idempotency-Key header and returns the original result if the same key is sent again. Where a target system has no such feature, put your reference in a field you can search, and look it up before retrying. Never retry a write blind.

Can the bank balance prove you have no duplicates?

Yes, and it is the check we would put first on any controller's list. The bank reports a balance per account per day. Your ledger has a balance for the same bank account. If every line has been booked exactly once, they agree. A duplicate or a missing line shows up as a difference, and the size of the difference usually points straight at the cause.

Here is an illustrative bank account for one week.

  • Opening balance €12,480.00
  • Customer payment received +€4,210.00
  • Software annual subscription −€1,728.00
  • Office rent −€349.00
  • Coffee −€4.50
  • Coffee −€4.50
  • Closing balance per bank €14,604.00

Suppose the feed reconnects mid-week and delivers the software line again, and the ledger has no key check. The ledger now shows €12,876.00 for that account. The difference with the bank is €1,728.00, exactly one subscription payment. With keys in place, the re-delivered line matches the key of the first one, is recognised as a repeat, and the ledger stays at €14,604.00. Both coffees stay, because they are occurrences 1 and 2 of the same fingerprint.

Run that comparison daily, per bank account, not once a month. A difference found the next morning has one or two candidate lines. A difference found at month-end has hundreds.

What about duplicate supplier invoices?

Invoices need the same thinking with a different key. The natural key is the supplier plus the invoice number, normalised so that "INV-0042", "inv 42" and "0042" are recognised as one. A second invoice with that key is a hard duplicate and should never be booked or paid.

Then there are soft duplicates: same supplier, same amount, a date a few days apart, but a different or missing number. That could be a resent invoice with a typo, or two genuine monthly charges. Here software should not decide alone. It should ask a person, showing both documents side by side.

This matters most at payment time. Paying a supplier twice is the duplicate that costs real cash, and recovering it takes weeks of emails. In Thred Layer, the Payer agent's job is described as paying bills on time, never twice, and the checks above are how that promise is kept.

How should a duplicate that slipped through be corrected?

Not by deleting it. If a duplicate entry reached the ledger, the correction is a reversing entry that cancels it, with a note pointing at the original and the reason. The record then shows what happened and when it was fixed, which is what your auditor or accountant will want to see. Deleting the entry makes the balance right and the history wrong.

At Thred, the ledger is built so that every entry can be reviewed and reversed and nothing is deleted from the record. Each entry links back to its source, so you can see which bank line or document it came from and whether that source was seen before.

A short checklist for your own setup

Whatever software you use, you can test how it handles repeats in an afternoon.

  • Reconnect a bank feed, or import a statement file for a period that is already loaded. Count the lines before and after.
  • Compare the ledger balance with the bank's closing balance for every bank account, every day, and look at how the tool reports differences.
  • Find two genuinely identical small transactions on one day and check that both survive a re-import.
  • Upload the same supplier invoice twice, once with its number written differently, and see what happens.
  • Ask your vendor what reference is sent with each write to your accounting software, and what happens when a write times out.

If any of these produce a duplicate without a warning, someone on your team is cleaning up after the software every month.

The takeaway

Data arriving twice is normal. Books recording it twice should never happen. The fix is boring and reliable: stable keys at the door, one entry per source record, references on every write, and a daily bank balance check that proves it.

See how Thred's continuously updated ledger books each transaction once and traces it to its source, or book a demo to watch a re-delivered bank feed land without a single duplicate.

Frequently asked questions

Why do bank transactions show up twice in my bookkeeping?

Usually because the same line was delivered twice: a feed reconnected and re-fetched recent days, a statement file was imported for a period the feed already covered, or a pending card payment came back as booked. The software then lacked a reliable way to recognise the repeat.

What does idempotency mean in accounting software?

It means processing the same data twice has the same effect as processing it once. A re-sent bank line, payout notification or invoice is recognised and ignored or used to update the existing record, so nothing is booked twice.

How can I tell if my ledger has duplicate transactions?

Compare the ledger balance of each bank account with the bank's closing balance for the same day. If every line was booked exactly once they agree. A difference the size of a single transaction usually points straight at a duplicate or a missing line.

Should I delete a duplicate journal entry?

It is better to reverse it with a correcting entry that refers to the original and states the reason. That keeps the history intact for your accountant or auditor while bringing the balance back in line.

How do you avoid paying the same supplier invoice twice?

Treat the supplier plus a normalised invoice number as a unique key and block a second invoice with the same key. Send near matches, such as the same supplier and amount a few days apart, to a person with both documents side by side before anything is paid.

Sources

  1. Idempotent requests (Stripe API reference)
  2. Receive Stripe events in your webhook endpoint (Stripe docs)
  3. The Idempotency-Key HTTP Header Field (IETF draft)

Read next

Where Thred comes in

Thred Ledger

Thred’s own general ledger, built for agents. Every transaction is booked as it happens, under your rules, and traceable back to its source.

Keep reading

Finance, handled.

Book a demo, or request a callback and we will call you.

Book a demo