Why does Google use “Allow” in robots.txt, when the standard seems to be “Disallow?”
4th February 2012
My answer to Why does Google use “Allow” in robots.txt, when the standard seems to be “Disallow?” on Quora
The Disallow command prevents search engines from crawling your site.
The Allow command allows them to crawl your site.
If you’re using the Google Webmaster tools, you probably want Google to crawl your site.
Am I misunderstanding your question?
More recent articles
- ChatGPT Containers can now run bash, pip/npm install packages, and download files - 26th January 2026
- Wilson Lin on FastRender: a browser built by thousands of parallel agents - 23rd January 2026
- First impressions of Claude Cowork, Anthropic's general agent - 12th January 2026