25th July 2024 - Link Blog
AI crawlers need to be more respectful (via) Eric Holscher:
At Read the Docs, we host documentation for many projects and are generally bot friendly, but the behavior of AI crawlers is currently causing us problems. We have noticed AI crawlers aggressively pulling content, seemingly without basic checks against abuse.
One crawler downloaded 73 TB of zipped HTML files just in Month, racking up $5,000 in bandwidth charges!
Recent articles
- Conceptual integrity and counting lines of code - 19th August 2026
- Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things - 16th August 2026
- Now we have a timeline of the OpenAI accidental attack against Hugging Face - 7th August 2026