How we score a 4-8 week PoC

2026-Q2

A PoC answers one question that a person in the business already knows how to argue about. “Can we extract requirement IDs from this audit pack well enough that a reviewer spends less time?” is a question. “Explore GenAI on our documents” is exploration, not a PoC. We lock the task sentence before week one and refuse to expand it in week three when someone wants a chatbot on the same corpus.

Samples

We want on the order of 20 real files, not three clean PDFs from the vendor deck. For document work that means the ugly ones: scans, mixed German/English, tables, stamps. If they cannot give us 20, we do not have a PoC; we have a demo. Hold out a few files the model never sees while we tune prompts. People leak the holdout by “just testing” in the same chat thread. Separate folder, separate credentials if you can.

Speech is the same rule with hours instead of pages: a few hours of their calls or dictation, including the accent and the canteen noise. Whisper-small on LibriSpeech is meaningless for a Berlin support line. Measure word error on that audio, then decide if the transcript is good enough for the next step (search, summary, CRM fields).

A metric they already use

Field-level exact match (or fuzzy match with a written threshold) for extraction. Count of missed requirement IDs for audit packs. Minutes a reviewer still spends per document. Tokens and euros per 100 pages, with the prompt we actually ship, not a 200-token toy prompt. If the business today samples 10% of files by hand, we compare to that 10%, not to a leaderboard.

Chat quality as a vibe check is allowed as a side note. It is not the go/no-go. We have killed PoCs that looked great in a live demo and failed on the holdout table. We have also said “use a script” when 80% of the files were one SAP export layout. That is a successful PoC: you learned not to buy a model.

What goes on the one-pager

Proceed, pause, or solve it without a model. Accuracy on the holdout, cost at the volume they quoted, who reviews errors (see the review queue), where the data sat (API vs local), and what would have to be true to harden it (monitoring, retries, a person who owns the prompt). If we used open weights, the card facts from the model-card note go in an appendix so nobody “upgrades” to a Gemma size that has no vision, or turns on tools because a sibling checkpoint had them.

Timebox is 4-8 weeks on purpose. If evaluation is still “we’ll know it when we see it” in week five, the question was never specified. Stop and rewrite the question rather than adding another model.