unrealistsWhen the website is the API
Two jobs get lumped into “scraping.” Crawling: follow links and store what you find, hoping structure repeats. Targeted extraction: given a source list and a schema, pull the fields the business already named (price, vacancy date, filing section, part number). Crawling is exploratory inventory. Extraction is a contract with downstream systems. Same HTTP stack, different success metric. If you crawl first without a schema, you end up with a tarball nobody can join to the ERP.
When crawling is the wrong default
Crawling pays off when you genuinely do not know what is on a domain (compliance audit of public pages, sitemap discovery before scoping extraction). It does not pay off when the question is “what is the value in cell B12 every Tuesday?” or “list every open role in these three municipalities.” That is extraction: fixed endpoints or URLs, parsers per layout, validation before load. A recursive spider without a field list is how you miss a ToS clause, burn budget on duplicate pages, and still fail the join key the warehouse needs.
If the stakeholder cannot name the fields and refresh cadence, you are still in exploration. Lock the task before you tune concurrency.
Websites as APIs
A lot of “scraping” is integration work where nobody shipped an API. The contract is the same as a REST job: inputs, outputs, error codes, refresh cadence, and an owner when the upstream changes. You implement the missing layer: stable identifiers, normalized rows, retries, and logs that look like any other service call.
Public registers are the clean example. A government insolvency or bankruptcy portal may list new filings every day with search and detail pages, but no bulk download and no documented API. The business still needs “new cases since yesterday” in a warehouse or a credit workflow. That is extraction against a fixed query pattern, not a site-wide crawl: you record the search parameters, fetch detail pages for new IDs, parse the fields legal already named, and validate before anything triggers a downstream action.
The other shape is operational: procurement or vendor eligibility checks behind an archaic web form (dropdowns, hidden fields, a POST that returns HTML). An automated job may need that answer several times a day before quoting, onboarding, or payment can continue. Here the website really is the API: session handling, form tokens, timeouts, and a queue so bursts do not look like abuse. You design idempotent “check status” calls, cache negative results for a sane TTL, and alert when the form layout shifts, same as you would when a vendor deprecates a JSON field.
Legality and data rights
We start with sources the client is allowed to use: vendor APIs, bulk exports, licensed feeds, SFTP drops, and portals behind credentials they own. Terms of use and robots rules are read as constraints, not puzzles to dodge. Personal data (EU): purpose, basis, retention, and whether the extract belongs in production or a one-off study. We write down what we relied on (API key scope, contract paragraph, public-domain claim) so legal can sign off without reverse-engineering the repo later.
“Scrape because there is no button” is a last resort after someone has checked product docs, partner programs, and whether an official dataset already exists (open government portals, industry aggregators, marketplace seller APIs). When HTML is the only surface the terms permit, extraction still needs the same field contract as an API job: rate limits, attribution, and a stop condition if the operator asks you to use their feed instead.
Layout drift and operational reality
Sites change. Class names move, A/B tests swap markup, PDF tables reflow, and “temporary” JSON endpoints disappear. Crawlers that only counted URLs go quiet. Extraction pipelines break loudly on validation: null rate spikes, checksum failures on primary keys, or a regex that suddenly matches the footer instead of the price. You need monitoring, versioned parsers, and a human who knows which source is allowed to be stale for a day.
Common obstacles are engineering problems with boring names: pagination that is not page=1..n, session cookies on a B2B portal the client logs into, JavaScript-rendered tables when static HTML is empty, CAPTCHA only on the signup path, not on the data path. The fix is rarely “more aggressive.” It is often an export job on the client side, a webhook, a smaller poll interval with backoff, or splitting “fetch” from “parse” so you can replay last week’s HTML when today’s DOM shifts. Same discipline as extraction from documents: schema first, replayable artifacts, metrics that finance can read.
How we stay on top of the flow
We do not hand over a script and vanish. The engagement stays a consulting thread: source discovery workshops (what already exists in IT, what vendors never mentioned), architecture for refresh and alerts, and a written go/no-go when a source is too brittle or too risky. Implementation is paired with decisions: which fields are worth a fragile parser, when to pay for the vendor API, when to accept manual CSV until contract renewal.
Obstacle list we actually walk through: authentication the client controls, polite rate limits, idempotent runs, diffing against yesterday, and a fallback path when the layout changes (secondary selector, OCR on a screenshot region, human queue for the long tail). Success looks like data in the target system with an owner who knows how it is produced, not like a hero script that only one engineer can restart. For PoC framing and metrics, see How we score a 4-8 week PoC.