Simon Willison’s Weblog

Hmm, you know I was all set to say this would be easier with wget, but it turned out to be harder:

wget -S --spider slashdot.org 2>&1 | grep X- | grep -v X-powered

Bill Stilwell - 6th February 2004 00:14 - #

The python one-liner version is fun too:

python -c "import urllib; i = urllib.urlopen('http://slashdot.org').info(); print ['%s: %s' % (k[2:].title(), i[k]) for k in i.keys() if k in ('x-fry', 'x-bender')][0]"

Simon Willison - 6th February 2004 00:29 - #

Well, I won't claim the perl one-liner is pretty, but it was fun to concoct:

perl -MLWP::UserAgent -e '$ua=LWP::UserAgent->new(env_proxy=>1); $r=HTTP::Request->new(GET=>"http://slashdot.org/") ; $h=$ua->request($r)->headers; $h->scan(sub {print "$_[1]\n" if $_[0] =~ /X-(Bender|Fry)/;});'

Bill Stilwell - 6th February 2004 01:04 - #

Let's add curl for good measure:

curl -Is http://slashdot.org | egrep '^X-(Bender|Fry)'

David Lindquist - 6th February 2004 01:14 - #

Someone alert the Wget and Curl Weblog.

Adrian Holovaty - 6th February 2004 01:29 - #

lynx -mime_header http://slashdot.org/ | head -n 6 | tail -1 | cut -d : -f 2

Who needs that silly X-Bender: bit?

James - 6th February 2004 01:58 - #

Ooh I didn't know about cut. You don't want to cut off everything before the colon though as you need to know if the quote comes from Fry or Bender. This seems to work well though:

lynx -mime_header http://slashdot.org/ | head -n 6 | tail -1 | cut -d - -f 2-

Simon Willison - 6th February 2004 02:05 - #

Heh, Slashdot are going to hate you for this ;)

Andrew Donaldson - 6th February 2004 08:42 - #

Heh -- will this be Slashdot being hit by the Simon Willison effect? ;-)

Nick - 6th February 2004 10:06 - #

Why use three processes when one will do:

lynx -mime_header http://slashdot.org/ | sed -n '6s/^X-//p'

Sed - 30 years old and still going strong :-)

Mark Russell - 6th February 2004 10:47 - #

Add the -head parameter and save slashdot a bunch of traffic :) (it's faster, too! ) lynx -head -mime_header http://slashdot.org/ | sed -n '6s/^X-//p' One more thing: How did you find that? And DON'T tell me that after x years of web development you are not using a browser anymore, but instead talking raw HTTP with the servers... *G*

Sencer - 6th February 2004 13:13 - #

OK, the sed method wins. How much more arcane can you get?

I think I first spotted those headers last year when I was debugging mod_gzip support for my HTTPClient class, but I've heard about it from a few places - they've been doing it for quite a long time and it's pretty much a part of internet folklore now.

Considering their track record of bringing lesser sites to their knees, Slashdot is also one of the few sites for which wasting their bandwidth really doesn't feel that immoral ;)

Simon Willison - 6th February 2004 14:58 - #

Quote: X-Bender: Woohoo, I'm popular!

Hetta - 6th February 2004 21:37 - #

Well, I don't have lynx (not my box), but the admins did install wget at least. Here's a version for all the poor sods like me. A bit brittle, but hey, it's just a quick hack:

wget -S --spider slashdot.org 2>&1 | sed -n 's/^ 6 X-//p'

Keith Gaughan - 9th February 2004 20:52 - #

You lot are fuckin GEEKS PUMP PUMP SQUIRT

Jodoodle - 18th May 2006 15:38 - #