let SYSTEM be "You extract data from invoices (OCR text, it may contain errors such as O for 0, l for 1, S for 5). Reply ONLY with JSON containing: supplier (text, as it appears at the top, in normal capitalization), number (text NNNN-NNNNNNNN), date (YYYY-MM-DD; dates are written day/month/year), currency (ARS or USD), subtotal, discount (0 if none), vat, total (numbers without thousands separators; the valid TOTAL DUE), and confidence: a number from 0 to 1 saying how sure you are that EVERYTHING extracted is correct."← All cases
Case 4 · Measured on 2026-10-16 · Synsema 0.6.44 · same data, same model on both sides
Which invoices can be paid without a human
If the model extracts an invoice correctly, is its confidence enough to pay it without review?
0 vs 7
of 7 risky invoices caught: the LLM's own confidence vs Jev
The LLM read all 60 invoices right, and its own confidence would have paid the duplicates. Jev caught 7 of 7.
The problem
Accounts payable teams already use LLMs to read invoices. With current models, reading is no longer the hard part.
The hard part is deciding what can be paid without anyone looking. A duplicate read perfectly must not be paid, and a total corrected by hand must be checked.
What we built
- 60 synthetic invoices from fictional companies, in OCR-like text, with the right answers. 28 carry real difficulties: OCR noise, ambiguous dates, foreign currency, discounts, hand-corrected totals, duplicates.
- DeepSeek extracts the fields and reports its own confidence. An arithmetic check validates the totals. Jev, a calibrated judge model, answers four questions about the document and the extraction in one call.
- What is not approved waits in approve, Synsema's human gate, with a single-use link sent by webhook.
Results
| Rule | Approved alone | Caught the 7 to review | Would be paid unreviewed | False alarms |
|---|---|---|---|---|
| LLM's own confidence ≥ 0.9 | 60 (100%) | ✗0 of 7 | ✗all 7 | 0 |
| Arithmetic check only | 60 (100%) | ✗0 of 7 | ✗all 7 | 0 |
| Jev (≥ 0.85 per field, ambiguous < 0.5) | 46 (77%) | ✓7 of 7 | ✓none | 7 (all with OCR noise) |
| Test | Result |
|---|---|
| POST /approvals/{id} with a fake token, as an agent would try | ✓403, the approval keeps waiting |
| Reject the duplicate with its link | ✓200, decision recorded |
| Use the same link again | ✓404, single use |
| Run without a terminal (CI, an agent) | ✓denied immediately |
The code, side by side
The usual way
Synsema
let j be jev.systemone({"document": f.text, "extracted": remove(e, "confidence")}, {
"total_ok": {"type": "noul", "instructions": "The extracted total is exactly the valid total due on the document"},
"date_ok": {"type": "noul", "instructions": "The extracted date (YYYY-MM-DD) is the issue date of the document, read as day/month/year"},
"supplier_ok": {"type": "noul", "instructions": "The extracted supplier is the company issuing the invoice"},
"ambiguous": {"type": "noul", "instructions": "The document has contradictory data, hand corrections, is a duplicate, or has reading errors that cast doubt on some value"}})agent Approver
let ok be approve message within 30m
-- One key per invoice: state_* has no atomic read-modify-write, and several agents decide at once
state_set(`decision:{fid}`, {"approved": ok, "t": now()})Why Synsema
- judge returns calibrated probabilities you can gate on. The model's self-reported confidence was 0.9 or higher on everything, duplicates included.
- approve is a language primitive. Under serve it never resolves by itself: it queues, a person gets a single-use token, and it is denied when the deadline passes.
- An agent cannot fake an approval. There is no code path where a program approves its own payment.
What it does not solve
- Synthetic text invoices. With real PDFs or photos, extraction will not be perfect.
- Calibrated judgment does not replace business rules. A copy of an invoice already paid scored 0.47 and passed Jev; the rule "same supplier and number already processed" caught it.
- Thresholds (0.85 and 0.5) were fixed before measuring. With real data, calibrate them on a separate set.
Reproduce it
git clone https://github.com/synsema/why-synsema && cd why-synsema # cases/04-invoice-approval python cases/04-invoice-approval/data/generate_invoices.py synsema run cases/04-invoice-approval/process.syn SYNSEMA_HUMAN_WEBHOOK=http://127.0.0.1:8105/channel SYNSEMA_HUMAN_PUBLIC_URL=http://localhost:8105 \ synsema serve cases/04-invoice-approval/review.syn # approval queue on :8105
github.com/synsema/why-synsema/tree/main/cases/04-invoice-approval