unrealistsOCR, layout models, and Azure Document Intelligence
“Document intelligence” is at least three jobs. Read the characters. Recover reading order, tables, stamps, headers. Then pull fields a reviewer already knows (invoice total, requirement ID). People buy one API or one VLM and expect all three. We split them, because the cheap path is different for each.
Often you do not need OCR
Born-digital PDFs already have a text layer. We try that first (pdfminer, PyMuPDF, whatever the stack already has). If you can select the text in a PDF viewer, OCR is wasted GPU and it will scramble hyphenation. Scans, faxes, photographed pages, and “PDF” that is only images: then OCR.
Traditional line OCR is still the default local engine for printed German/English scans. Tesseract on CPU, with the right tessdata and page-segmentation mode, is boring and operable. PaddleOCR (or RapidOCR, same models on ONNX) is what we reach for when the page is rotated, mixed script, or Tesseract’s line boxes fall apart. EasyOCR shows up in prototypes; we do not leave it as the production binary without a reason. None of these understand a table. They give you strings and boxes.
Layout is a separate model
If the output has to keep table structure or reading order, we run a layout detector, then OCR (or a recognition VLM) on the regions. Paddle PP-DocLayoutV3 is the one we see wired into current pipelines: blocks plus order, including messy scans. Docling’s local layout models (Heron and friends) are the other stack we actually install when the job is “PDF to Markdown for retrieval,” not “draw boxes on an invoice.”
GLM-OCR is the small VLM we try when line OCR loses formulas, seals, or nested tables: 0.9B, MIT on the weights, text / formula / table prompts, optional JSON extraction if you give it a schema. The full SDK pipeline still calls PP-DocLayoutV3 (Apache 2.0) for layout, then GLM-OCR for recognition. Do not paste OmniDocBench scores into a client deck; run it on their 20 ugly pages. Also do not confuse the open weights with Zhipu’s hosted MaaS: that path leaves the building.
When we call Azure
Azure Document Intelligence (old name: Form Recognizer) is the API we use when the sample is allowed to hit a Microsoft region and the pages are the kind Tesseract hates: skew, handwriting, checkboxes, invoices that almost match a template. The layout model returns text, tables, selection marks, and structure. Prebuilt invoice/receipt models and a custom extraction model are for repeating forms, not for “chat with 40,000 mixed PDFs.”
We pick an EU endpoint when that is the constraint, write down retention, and cost it per page at the volume they quoted, not at three demo files. High-volume clean digital PDFs do not belong here. A mailroom of scans often does, at least for the first PoC, because we get a structured JSON without standing up GPUs. If the PDFs cannot leave the network, Azure is off the table and we stay on Tesseract/Paddle plus layout plus GLM-OCR, with the same field metric as in How we score a 4-8 week PoC.
Hybrid we actually ship: text-layer extract locally; OCR locally for the rest; Azure only for pages that fail a confidence check or for a custom model on one form type; LLM (API or local) on the extracted text, not on raw page images, unless the task is visual (stamps, handwriting the OCR already lost). Related: What we read on a model card before a PoC.