The review queue

2026-Q4

If the output can move money, a customer, or an auditor, a person still signs off. We put that in the PoC, not in a “phase 2 UX” slide. A live demo where someone nods at a chatbot is not a review workflow. The system is: document in, model proposes, human accepts or edits, result lands in the real tool (ERP, ticket, archive). Skip the middle box and you have a parlor trick.

What the reviewer has to see

The original page (or the span the field came from), the proposed value, a way to edit it without retyping the whole form, and a way to reject the file back to a pile. For extraction that is field-by-field. For retrieval it is the cited passage next to the answer, with “this citation is wrong” as a first-class action. If they only see the model’s prose, they will rubber-stamp.

Confidence is a routing hint, not a score to print in tiny grey type. High: maybe sample 10% if that is already how the team works. Low or unknown: every file. We would rather over-queue in week one than invent a threshold on three demos. Minutes per document for the reviewer is a PoC metric. If the queue is slower than the old process, the model is not helping yet.

Who owns a wrong accept

Write down who is on the queue (not “the business”), who they escalate to, and what happens when they catch an error after export. Chat logs are not an audit trail. We keep the proposed value, the final value, who clicked, and the model/prompt id. That is enough to reprocess when the template changes. It is also enough to see that one reviewer accepts everything.

What you win from the extra work

Proper review is not only risk control. It is how you get a dataset. Each click is a labeled pair: the page (or span), what the model said, what a person accepted or typed instead. That pile is valuable because it is their documents and their corrections, not a public benchmark. A chat demo leaves you with nothing you can train on. A queue leaves you with rows.

You use those rows immediately for prompt work you can measure: group the misses, change the prompt or the schema, rerun a holdout, watch the edit rate. Same table, later, is fine-tune data (document in, fields or JSON out) and training data for a router or an embedder (invoice vs contract). Compare a fine-tune to the prompt-only baseline on a later week, not on the examples you just trained on. That is a real improvement loop. None of it exists if nobody sat in a queue and disagreed with the model.

Internal search and messy exploration can live without a queue. Invoices, audit packs, medical paperwork, and anything that hits a customer should not. Related: OCR, layout models, and Azure Document Intelligence, Local inference vs an API.