Jugal Sheth  ·  The Data Factory
Production lessons

One Approval, Two Writes

The principle: a deduplication ID that is unique every time deduplicates nothing. If one approval can reach the write function through two paths, it eventually will — and the worker has to be the last line of defense.

One click. Two invocations.

An approver clicked approve once. The system triggered the write function twice.

A 90-day audit of the logs found that about one in four approved requisitions triggered the ERP write function twice. The approvers saw nothing wrong. The jobs were green. The orders went through.

That is the failure mode. Not a crash. A system doing the same work twice and calling it success.

How it was found

We run an internal procurement app. An approver approves a requisition, and the system hands it off to the ERP to create the purchase order.

The audit was a routine reliability check: 90 days of logs for the write function. What it found was not an event. It was a pattern.

Duplicate invocations — and almost all of them arrived through two doors at once: the queue and the API gateway, for the same approval. One business action, two triggers, two runs through the creation logic.

Each duplicate run was a second attempt at the same write — the system had no strong gate before the external call. Nothing had exploded yet. That was the scary part. Nothing had to explode for this to be broken.

The mechanism

Four layers. Each one alone looks harmless.

First, two doors in. The approval route sent a message to the queue — and it also had a fallback that called the write function directly through the API gateway. The fallback was a safety net, there in case the queue path failed. But nothing stopped both from running for the same approval. A safety net that fires alongside the main path is not a safety net. It is a second trigger.

Second, the dedupe ID was a timestamp. The queue was FIFO with deduplication. Deduplication works by giving each message an ID: identical IDs inside a window collapse into one delivery. The ID was built from the current time — Date.now() — so every message looked brand new. The dedupe window had nothing to collapse. The queue was doing exactly what it was told. It was told wrong.

A dedupe ID built from the current time is not a dedupe ID. It is a serial number.

Third, the worker didn't check first. The function that writes to the ERP didn't reliably ask whether a purchase order already existed for that requisition before writing one. So a duplicate invocation became a duplicate write attempt. The queue was supposed to prevent the duplicate. The worker was supposed to catch whatever got through. Neither had been given the information to do its job.

Fourth, retries were invisible. Only terminal failures surfaced in the logs on many paths. The attempts in between were missing. So the system could run the same write twice and leave a log trail that looked like a single clean failure — or a single clean success. Support spent more time proving what happened than fixing it.

And this was the same handoff that had already failed twice. A week earlier, the same pipeline had silently dropped work under burst traffic. A month before that, the same write path failed because item IDs arrived as decimal strings — "425.00000" instead of "425" — and the ERP rejected them.

Different symptoms, same handoff: the place where an approval becomes a write in an external system is where the system keeps lying, because it is the place with the most moving parts and the least visibility.

The fix

Four changes, in the order that matters.

1. One trigger path. User-facing approval flows use the queue only. The API gateway fallback is gone from the normal path. The admin fix-and-retry direct path stays — deliberately, as the recovery flow, not as a second trigger for normal approvals. One business action, one door in.

2. Stable dedupe IDs. Built from the business key: the requisition plus the action. Same approval, same ID, every time. Now the queue's dedupe window has something to collapse. The dedupe ID finally means what its name says.

3. A pre-create check in the worker. Before any write, the worker looks up the warehouse record for the requisition. If a purchase order already exists, it skips. If one is already in progress, it waits and rechecks once. The gate sits before the external call, not after it. A check after the write is a report. A check before the write is a safeguard.

4. An atomic claim before the write. The worker marks the requisition in-progress in a single atomic step before calling the ERP, so two overlapping runs can't both slip through the gap between checking and writing. And every log line now carries a correlation ID, with retry attempts logged at the point they happen — so the next incident can be reconstructed from the logs instead of from memory.

What it bought — and what it didn't prove yet

The fix was validated with safe tests that deliberately never touch the ERP: an empty payload, fake requisitions through both the queue-shaped and API-shaped paths, and a completed requisition that correctly skipped instead of writing again.

But none of those tests prove the full success path. A real end-to-end test needs real writes, and that hasn't happened. The monitoring window is two weeks, with a target of near-zero duplicates. And the fix prevents new duplicates — it doesn't clean up historical ones. That work is separate, and still open.

I am writing that down because it matters. The fix is real. The proof is partial. A draft that pretends otherwise would be the same kind of lie the pipeline told.

What to check in your own system

If you run anything where one user action triggers a write in an external system — and you do — these are worth an afternoon:

  1. Count the doors in. For each business action, list every path that can start the write. If there is a fallback, ask when both can fire. A fallback that can run alongside the main path is a second trigger with a reassuring name.
  2. Read your dedupe ID. If it contains a timestamp, a random value, or anything unique per attempt, it dedupes nothing. Build it from the business key — the thing you are protecting — so the same action always produces the same ID.
  3. Put a gate before the external write. Check whether the thing already exists before you create it. The check has to happen before the call, not after. The gate has to live in the worker too. The queue is a filter; the worker is the last line of defense.
  4. Claim before you write. Mark the work in-progress atomically, in the same step as the check, so two overlapping runs can't both pass the gate. A check without a claim is a race condition with a polite name.
  5. Make retries visible. Log every attempt. If you only log the final failure, you can't reconstruct what happened to one record from the logs. Then you don't have observability. You have a log file.

The lesson

A queue with deduplication feels safe. It is safe only if the ID means something.

Build the ID from the thing you're protecting. Give the worker a gate before the write. And count the duplicates — because the system will not tell you it's doing the work twice. It will just do it, quietly, and call it success.


Sources

Numbers in this article, traced to the work they came from (internal counts deliberately left out of the body):

  • Roughly a quarter of approved requisitions triggered the write function twice; 90-day audit window — corpus file 019 (ADR-010, 2026-04-14). The exact internal counts behind the rate are internal-only and were not used in the article body.
  • Dual trigger via queue + API gateway; dedupe IDs built from Date.now(); no pre-create idempotency check; atomic in-progress claim; correlation IDs; two-week monitoring window with near-zero target; admin fix-and-retry path kept as intentional exception — corpus file 019.
  • Silent failures under burst traffic a week earlier (restrictive concurrency cap, no gateway logging, dead-letter queue with no app feedback loop) — corpus file 018 (incident summary, 2026-04-08).
  • Item IDs arriving as decimal strings ("425.00000") failing ERP PO creation a month earlier; fix was one idempotent normalize helper at every read/write boundary — corpus file 013 (ADR-003, 2026-03-10).

No company or client names, internal table/procedure names, requisition identifiers, or commit SHAs appear in this article. No detail was invented: the validation limits (safe tests only, no real ERP writes; historical duplicates not retroactively repaired) are stated in corpus file 019, sections 6 and 8.


Enjoyed this? I write one short, practical data engineering lesson every day in THE DELTA.