Why your scraper dies on day three

Nearly every scraper rescue we take on starts the same way. Somebody wrote something that worked, shipped it, and three days later the data stopped arriving. Nothing in the code changed, so the assumption is that the site "blocked us". Usually it did not.

The four things that actually changed

Your fingerprint became boring. The first requests looked like a browser because the headers were copied from one. What was never copied is the TLS handshake — cipher order, extensions, ALPN. A default HTTP client announces itself in the first packet, long before any header is read. Two days of identical handshakes from one address is a pattern, and patterns get scored.

The rate was fine until it wasn't. Most rate limits are not a fixed number of requests per minute. They are a budget that refills, and a burst that empties it inside a second gets treated differently than the same volume spread out. Your test run had no burst because you were watching it.

A single class name moved. Markup changes constantly, and the failure is silent: the selector matches nothing, the parser returns an empty list, and the pipeline writes zero rows without raising anything. The scraper is not broken in any way a health check can see.

Session state expired. The token you got on the first run had a lifetime nobody documented. Everything after it returned a 200 with an empty body, which is the worst possible failure mode: it looks like success.

What to build instead

The fix is rarely a cleverer bypass. It is making failure loud.

• Assert on shape, not on status. A run that returns fewer rows than the last one is a failure, even at 200. Compare against the previous run and alert on the delta.
• Separate fetching from parsing. Store the raw response. When a selector breaks, you re-parse yesterday's data instead of re-crawling for it.
• Rotate on identity, not on address. A pool of addresses sharing one TLS fingerprint is one client wearing different hats.
• Budget your bursts. Spread requests deliberately. It costs you minutes and buys you months.
• Alert on silence. No data is a louder signal than an error, and almost nobody monitors for it.

None of this is exotic. It is the difference between a script and a system, and it is most of what we do when someone hands us a scraper that used to work.