Simon Willison September 7, 2026 1 min signal 2026-09-07

Creepy crawlies

Archived text mirror, captured for personal reading when the Signal ran. Formatting is approximate — read the original at the source.

Simon Willison’s Weblog

Sponsored by: Portnox — Shadow AI is the new shadow IT. On Sept. 10, Forrester Research and Portnox share practical steps to regain AI agent visibility, access management, and policy enforcement. Register today

7th September 2026 - Link Blog

Creepy crawlies (via) Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel:

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.

Posted 7th September 2026 at 11:08 pm

Simon Willison’s Weblog

Recent articles