Field notes
What we learn running scrapers at scale - anti-bot arms races, browser internals, compliance, and the occasional war story.
The SerpApi ruling, and why the robots.txt answer flips in Europe
A US court just gutted Google's DMCA case against SerpApi, and the coverage is getting the reasoning wrong. Here is what the order actually holds, and why the EDPB is moving in the opposite direction on the exact same question.
Monitoring, without the cron-job graveyard
Recurring scrapes are easy to start and hard to run well: missed windows, overlapping runs, silent failures when you run out of budget. Here's how we built monitoring so none of that is your problem - and so you get told when a page actually changes.
TLS fingerprints are the new User-Agent
Spoofing the User-Agent header stopped working years ago. Here's why your HTTP client gets blocked before it sends a single byte of HTTP, and what to do about it.
When to reach for a real browser (and when not to)
A headless browser is the most expensive way to fetch a page. Here's a practical decision tree for picking the lightest engine that actually works on your target.
Scraping and the GDPR: what a French SaaS actually has to do
Public doesn't mean free-for-all. A grounded look at where web scraping sits under the GDPR, written by a company that has to answer the questionnaire.