Green Jobs Lie.
Green jobs lie: a job can succeed while the data is wrong. Every incident below passed its checks. The dashboards were green. The data was wrong.
Not the environmental kind. I mean the green checkmark — the green dot on the dashboard, the green "success" on the pipeline run, the green status that tells the room everything is fine.
It is not fine. A pipeline job can succeed and still return the wrong answer. A query can run without errors and still drop the rows that matter. A dashboard can be green while the business decides on numbers that were never true.
That is what "green jobs lie" means. And it is the failure mode I keep finding in production data systems — not crashes, not timeouts, but silent wrongness. The system did exactly what it was told. What it was told was wrong.
Four failures that prove it
01 — The approval queue. Every job was green. About 15% of the purchase orders waiting for approval were invisible to the people supposed to approve them, because an INNER JOIN on a cross-system ID silently drops the rows that don't match. The fix: a LEFT JOIN and boundary-aware fuzzy matching. Visibility went 85% → 100%. Full story →
02 — A round number is a fingerprint. Salaries were stuck at exactly 5,600. Not 5,612. Not 5,589. Exactly 5,600 — which is 28 batches × 200. When a count lands on a round number, suspect pagination, not luck. A full page that returns exactly the page size is a warning, not a result.
03 — Claim before you send. 952 emails went to 242 people. One admin got 8 copies. Two workers reading the same queue will both send — unless one claims the work atomically first. Idempotency isn't a feature. It's the design. Full story →
04 — Never trust an exact-match join across two systems of record. An INNER JOIN quietly ate 280 expense reports — and $181M+ got falsely flagged in the process. The query didn't fail. It just returned less truth.
What to check in your own system
Monitor the result, not the job. A green job tells you the code ran. It tells you nothing about whether the data is right.
- Count what should be there, not just what arrived. Reconcile against the source of truth — a query cannot audit itself.
- Track match rate as a metric. The day a format changes upstream, your match rate is the first thing that moves. If nobody is watching it, the queue just gets quietly shorter.
- Give every join a fallback. A missing dimension should degrade a row, never delete it.
- Suspect round numbers. A count that lands exactly on a page size or a batch multiple is a signal, not a coincidence.
The lesson
Jobs lie by omission. The fix isn't more monitoring on the job. It's a parity check on the result: count what should be there, and make the gap visible.
I tell one of these stories every Wednesday in the "Green Jobs Lie" series on @jugal.builds — and one short, practical data engineering lesson every day in THE DELTA.