Simon Willison’s Weblog

Subscribe

Items tagged xrobotstag, yahoo in Jun, 2008

Filters: Year: 2008 × Month: Jun × xrobotstag × yahoo × Sorted by date


The X-Robots-Tag HTTP header. News to me, but both Google and Yahoo! have supported it since last year. You can add per-page robots exclusion rules in HTTP headers instead of using meta tags, and Google’s version supports unavailable_after which is handy for content with a known limited shelf-life. # 9th June 2008, 9:21 am

Types

Years

Months

Tags