Simon Willison’s Weblog

83 items tagged “http”


Traffic Server. Mark Nottingham explains the release of Traffic Server, a new Apache Incubator open source project donated by Yahoo! using code originally developed at Inktomi around a decade ago. Traffic Server is a HTTP proxy/cache, similar to Squid and Varnish (though Traffic Server acts as both a forward and reverse proxy, whereas Varnish only handles reverse). # 1st November 2009, 12:15 pm

High-end Varnish-tuning. Tuning the Varnish HTTP cache to serve 27K requests/second on a single core 2.2GHz Opteron. # 20th October 2009, 9:25 am

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm

PostBin. Handy debugging tool for webhooks—create a TinyURL-style URL, then see a log of any POST requests made to that address. # 21st September 2009, 11:03 pm

We experimented with different async DB approaches, but settled on synchronous at FriendFeed because generally if our DB queries were backlogging our requests, our backends couldn’t scale to the load anyway. Things that were slow enough were abstracted to separate backend services which we fetched asynchronously via the async HTTP module.

Bret Taylor # 11th September 2009, 5:31 pm

rather baffling finding: POST requests, made via the XMLHTTP object, send header and body data in separate tcp/ip packets [and therefore,] xmlhttp GET performs better when sending small amounts of data than an xmlhttp POST

Iain Lamb # 18th August 2009, 12:27 pm

PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub. # 1st July 2009, 8:22 pm

The Resource Expert Droid. Like the HTML Validator but for your server’s HTTP headers—extremely useful. # 25th June 2009, 10:06 am

Facebook Usernames and OpenID

Today’s launch of Facebook Usernames provides an obvious and exciting opportunity for Facebook to become an OpenID provider. Facebook have clearly demonstrated their interest in becoming the key online identity for their users, and the new usernames feature is their acknowledgement that URL-based identities are an important component of that, no doubt driven in part by Twitter making usernames trendy again.

[... 760 words]

Styling buttons to look like links. Nat has a neat trick for styling submit buttons to look like regular links—so there’s absolutely no excuse for using a “delete” link when you should be using a POST request. # 10th June 2009, 2:11 am

A rev=“canonical” HTTP Header. Chris Shiflett proposes optionally exposing rev=canonical information in an HTTP header, thus allowing sites to discover shorter URLs using just a HEAD request and removing the need to parse HTML. The pingback specification also uses this shortcut. # 12th April 2009, 12:33 pm

Concurrence. Exciting: a Python framework for “creating massively concurrent network applications” (the tutorial benchmarks a Hello World web server at over 8,000 requests a second). It’s implemented on top of libevent using pyrex, can run on either Stackless Python or Greenlets from the py library and ships with a WSGI server, an HTTP client and a DBAPI 2.0 compliant MySQL driver. # 15th March 2009, 1:28 pm

django-springsteen and Distributed Search. Will Larson’s Django search library currently just talks to Yahoo! BOSS, but is designed to be extensible for other external search services. Interestingly, it uses threads to fire off several HTTP requests in parallel from within the Django view. # 25th February 2009, 10:28 pm

Tokyo Cabinet: Beyond Key-Value Store. Useful overview of Yet Another Scalable Key Value Store. Interesting points: multiple backends (hash table, B-Tree, in memory, on disk), a “table” engine which enables more advanced queries, a network server that supports HTTP, memcached or its own binary protocol and the ability to extend the engine with Lua scripts. # 14th February 2009, 11:17 am

Pragmatism, purity and JSON content types

I started a conversation about this on Twitter the other day, but Twitter is a horrible place to have an archived discussion so I’m going to try again here.

[... 555 words]

Ehy IE8, I Can Has Some Clickjacking Protection? (via) IE8 has built-in protection against clickjacking, but it’s opt-in (with a custom HTTP header) and IE only. It turns out the usual defence against clickjacking (using framebusting JavaScript) doesn’t work in IE as it can be worked around with a security=“restricted” attribute on an iframe. # 29th January 2009, 1:39 pm


ETags And Modification Times In Django. Part of Malcolm’s series of tutorials on implementing advanced HTTP concepts in Django. # 13th December 2008, 9:49 am

ptth (Reverse HTTP) implementation in a browser using Long Poll COMET. Donovan Preston experiments with the cleverly named idea of ptth, where servers send HTTP requests to clients. # 8th December 2008, 5:22 pm

Response Splitting Risk. Important reminder that you should always ensure strings used in HTTP headers don’t contain newlines. # 19th October 2008, 11:58 pm

Versioning REST Web Services. Peter Williams suggests using a vendor MIME media type in the Accept header to specify a required API version, because embedding the API version in the URL itself leads to a single resource ending up with many different URLs, one for each API version. # 13th October 2008, 12:45 pm

Robust Defenses for Cross-Site Request Forgery [PDF]. Fascinating report which introduces the “login CSRF” attack, where an attacker uses CSRF to log a user in to a site (e.g. PayPal) using the attacker’s credentials, then waits for them to submit sensitive information or bind the account to their credit card. The paper also includes an in-depth study of potential protection measures, including research that shows that 3-11% of HTTP requests to a popular ad network have had their referer header stripped. Around 0.05%-0.10% of requests have custom HTTP headers such as X-Requested-By stripped. # 24th September 2008, 9:40 am

Flickr Developer Blog: API Responses as Feeds (via) Flickr API calls that return a “standard photos response” (e.g. and flickr.favorites.getList) can now output eight different feed formats as well, including Atom, RSS flavours, geoatom, geordf and KML. Error codes are returned as X-FlickrErrCode HTTP headers. # 25th August 2008, 10:20 pm

Gears API Blog: Gears 0.4 is here! New features are Geolocation, a Blob API for dealing with arbitrary binary data, onprogress() events for tracking HTTP downloads and uploads (meaning progress indicators) and the built-in Gears dialogs localized to 40 languages. # 22nd August 2008, 10:14 am

If it’s easy to make all your calls conform to the RESTful verb architecture, then that’s good, I guess. But if not, then just use a POST as an RPC call, keep it as simple as possible and be done with it. And don’t spend another minute worrying about being RESTful or not.

Damien Katz # 15th August 2008, 8:07 am

IE8 Security Part IV: The XSS Filter (via) IE8 will include an XSS filter to identify and neutralise “reflected” XSS attacks (where malicious code in a query string is rendered to the page), turned on by default. Sounds like a good idea to me, and site authors can disable it using Yet Another Custom HTTP header (X-XSS-Protection: 0). # 3rd July 2008, 9:37 am

Enough Already with the Connections! Comet doesn’t mean making long-lived HTTP connections (which most browsers do anyway thanks to HTTP keep-alive), it means making long-held HTTP requests. I’m guilty of spreading this misinformation in the past. # 30th June 2008, 9:27 am

mod_rpaf for Apache. A more secure alternative to Django’s equivalent middleware: sets the REMOTE_ADDR of incoming requests from whitelisted load balancers to the X-Forwarded-For header, without any risk that if the load balancers are missing attackers could abuse it to spoof their IP addresses. # 24th June 2008, 5:02 pm

ncache. A squid-style caching system built on top of nginx. Supports the HTTP PURGE method for cache invalidation. # 18th June 2008, 8:09 pm

The X-Robots-Tag HTTP header. News to me, but both Google and Yahoo! have supported it since last year. You can add per-page robots exclusion rules in HTTP headers instead of using meta tags, and Google’s version supports unavailable_after which is handy for content with a known limited shelf-life. # 9th June 2008, 9:21 am

Yahoo!’s Latest Performance Breakthroughs. 20 new performance tips to join the previously published 14. Flushing the buffer while the backend code is still working to cause the browser to start loading CSS earlier is interesting. # 20th March 2008, 3:17 pm