Jugal Sheth  ·  The Data Factory
The idea

Green Jobs Lie.

Green jobs lie: a job can succeed while the data is wrong. Every incident below passed its checks. The dashboards were green. The data was wrong.

Not the environmental kind. I mean the green checkmark — the green dot on the dashboard, the green "success" on the pipeline run, the green status that tells the room everything is fine.

It is not fine. A pipeline job can succeed and still return the wrong answer. A query can run without errors and still drop the rows that matter. A dashboard can be green while the business decides on numbers that were never true.

That is what "green jobs lie" means. And it is the failure mode I keep finding in production data systems — not crashes, not timeouts, but silent wrongness. The system did exactly what it was told. What it was told was wrong.

Four failures that prove it

01 — The approval queue. Every job was green. About 15% of the purchase orders waiting for approval were invisible to the people supposed to approve them, because an INNER JOIN on a cross-system ID silently drops the rows that don't match. The fix: a LEFT JOIN and boundary-aware fuzzy matching. Visibility went 85% → 100%. Full story →

02 — A round number is a fingerprint. Salaries were stuck at exactly 5,600. Not 5,612. Not 5,589. Exactly 5,600 — which is 28 batches × 200. When a count lands on a round number, suspect pagination, not luck. A full page that returns exactly the page size is a warning, not a result.

03 — Claim before you send. 952 emails went to 242 people. One admin got 8 copies. Two workers reading the same queue will both send — unless one claims the work atomically first. Idempotency isn't a feature. It's the design. Full story →

04 — Never trust an exact-match join across two systems of record. An INNER JOIN quietly ate 280 expense reports — and $181M+ got falsely flagged in the process. The query didn't fail. It just returned less truth.

What to check in your own system

Monitor the result, not the job. A green job tells you the code ran. It tells you nothing about whether the data is right.

  1. Count what should be there, not just what arrived. Reconcile against the source of truth — a query cannot audit itself.
  2. Track match rate as a metric. The day a format changes upstream, your match rate is the first thing that moves. If nobody is watching it, the queue just gets quietly shorter.
  3. Give every join a fallback. A missing dimension should degrade a row, never delete it.
  4. Suspect round numbers. A count that lands exactly on a page size or a batch multiple is a signal, not a coincidence.

The lesson

Jobs lie by omission. The fix isn't more monitoring on the job. It's a parity check on the result: count what should be there, and make the gap visible.

I tell one of these stories every Wednesday in the "Green Jobs Lie" series on @jugal.builds — and one short, practical data engineering lesson every day in THE DELTA.


Enjoyed this? I write one short, practical data engineering lesson every day in THE DELTA.