claudetbot-ifixit-1m-requests-jul2024

Summary

In July 2024, Anthropic's automated web scraper pulled content from the iFixit site roughly a million times a day even though iFixit had explicitly told AI crawlers to stay away. This highlights that a small website's stated scraping preferences carried essentially no enforcement power against a major AI company's appetite for training data, and that robots.txt functions more as a polite suggestion than a binding rule when the two sides are so unequal in scale.