Local inference vs an API

2026-Q3

Default for a first PoC is an API in an EU region with a DPA, logging we can turn down, and a sample that is allowed to leave the building. Most mid-market teams do not have a spare 24GB card, a person who will patch CUDA, or a reason to self-host yet. “We might be regulated later” is not a reason. “These PDFs cannot hit a vendor” is.

When we actually keep weights on-prem

Audit packs, medical/research IT, and finance paperwork are the usual triggers. If the sample cannot be anonymized without destroying the task (names are the thing you are extracting), we run on their GPU or a VPC they control. Then the model card license has to survive that deployment, not just a laptop demo. Gemma ToU and Llama Community are usable; they are not Apache. Fish Speech NC and XTTS CPML are a hard no for a product voice.

Hardware: we stay in roughly 270M-27B total, and we plan MoE on active params. Qwen3-8B or Mistral 7B Instruct v0.3 on a single 24GB card is a known shape. Mixtral at 12.9B active can fit; Mixtral-as-47B-dense in a quote will get the wrong box ordered. FunctionGemma 270M and SmolLM2 1.7B are for edge or cheap tool routers, not for “chat with the quality handbook.” Embeddings (nomic, BGE, EmbeddingGemma) almost always run local even when the LLM is an API. They are small and the corpus usually should not be shipped twice.

Hybrids we use more than pure local LLM

OCR and layout on-prem (see OCR, layout models, and Azure Document Intelligence), LLM on redacted text via API. Whisper or Parakeet on a batch of call recordings, then an API model on the transcript with speaker names stripped. A local classifier or regex for document type, API only for the long-tail types. That split is boring and it is how you get a go/no-go in 4-8 weeks without standing up an inference cluster.

Streaming speech is the exception where local (or a dedicated speech API) shows up early. Whisper-small is fine for overnight batches (~244M, card). Live captions want something like Parakeet unified-en in NeMo, or an on-device artifact like Whistle (16.9 MB, de/en among others). Qwen2-Audio is audio-in, text-out; it will not replace TTS. Qwen2.5-Omni will speak, and it wants more GPU than a Whisper job.

If IT cannot restart a container and read a VRAM graph, we do not leave them a local 14B as the production plan. We leave a working API path and a written trigger for when local would pay off (volume, data boundary, or latency they have measured). Related: How we score a 4-8 week PoC.