Simon Willison’s Weblog

Lou’s Pseudo 3d Page. Spectacularly detailed exploration of the road graphics used in racing games prior to true 3D. This is a potential gold mine for anyone looking for a fun project to try out with canvas. Bonus points for comet integration—I’m still looking forward to the first real-time multiplayer game in the browser using comet and canvas. 8th February 2010, 11:21 am

Comet (long polling) for all browsers using ScriptCommunicator. More Comet from the Plurk team: 80 lines of dependency free JavaScript implementing long polling using script tags (hence working cross-domain) across IE6+, Firefox, WebKit and Opera. The clever bit is the code to detect loading errors. It doesn’t try to fix the infinite loading indicator problem—is that still a cromulent usability concern? 3rd February 2010, 12:37 am

Plurk: Instant conversations using Comet (via) Plurk’s comet implementation sounds pretty amazing. They’re using a single quad-core server with 32GB of RAM running 8 Node.js instances to serve long-polled comet to 100,000+ simultaneous users. They switched to Node from Java JBoss/Netty and found the new solution used 10 times less memory. 1st February 2010, 10:13 am

Web Sockets in Tornado. Bret Taylor has a simple class making it trivial to experiment with the Web Sockets protocol (now shipping in Chrome) using the scalable Tornado application server. He also raises the million dollar question: what will existing load balancers and proxies make of the new protocol? 31st December 2009, 11:54 am

Real time online activity monitor example with node.js and WebSocket. A neat exploration of Node.js—first hooking a “tail -f” process up to an HTTP push stream, then combining that with HTML 5 WebSockets to achieve reliable streaming. 8th December 2009, 11:07 pm

SPDY: The Web, Only Faster. Alex Russell explains the benefits of Google’s SPDF proposal (a protocol that upgrades HTTP)—including header compression, multiplexing, the ability to send additional resources such as images and stylesheets down without needing the data:uri hack and Comet support built in to the core assumptions of the protocol. 13th November 2009, 1 pm

nginx_http_push_module. More clever design with webhooks—here’s an nginx module that provides a comet endpoint URL which will hang until a back end process POSTs to another URL on the same server. This makes it much easier to build asynchronous comet apps using regular synchronous web frameworks such as Django, PHP and Rails. 17th October 2009, 4:48 pm

Diesel. Yet Another Asynchronous Python Comet Library, of interest because this is the first one I’ve seen that uses Python’s generator coroutines, taking advantage of the return value of the yield statement to feed messages in to a generator function. Currently only works on Python 2.6 on Linux due to a dependency on 2.6’s epoll support. 23rd September 2009, 5:15 pm

Tornado Web Server (via) An extremely exciting addition to the Python web landscape, Tornado is the open sourced version of FriendFeed’s custom web stack. It’s a non-blocking (epoll) Python web server designed for handling thousands of simultaneous connections, perfect for building Comet applications. The web framework is cosmetically similar to web.py or App Engine’s webapp but has decorators for writing asynchronous request handlers. The template language uses Django-style syntax but allows you to use full Python expressions. FriendFeed have benchmarked it handling 8,000 requests a second running as four load-balanced processes on a 4 core server. 10th September 2009, 9:32 pm

svnpubsub.py (via) A Twisted/Python powered comet API for pushing out Subversion commits, built for Apache Foundation projects. 6th September 2009, 9:50 pm

EtherPad. Outstanding implementation of an online real-time collaborative text editor—basically SubEthaEdit in your browser. I can see myself using this a lot. 24th July 2009, 12:35 am

Webhooks behind the firewall with Reverse HTTP. Hookout is a Ruby / rack adapter that lets you serve a web application from behind a firewall, by binding to a Reverse HTTP proxy running on the internet (such as the free one provided by reversehttp.net). Useful for far more than just webhooks, this means you can easily expose any Ruby web service to the outside world. An implementation of this as a general purpose proxy server would make it useful for applications written in any language. 22nd July 2009, 1:46 pm

Reverse HTTP Demo (via) This is a bit of a brain teaser—a web server running in JavaScript in your browser which uses long polling comet to respond to incoming HTTP requests channelled through a “Reverse HTTP” proxy. 21st July 2009, 3:54 pm

Paul Buchheit: Make your site faster and cheaper to operate in one easy step. Paul promotes gzip encoding using nginx as a proxy, and mentions that FriendFeed use a “custom, epoll-based python server” as their application server. Does that mean that they’re serving their real-time comet feeds directly from Python? 17th April 2009, 5:19 pm

Visualising Radio, pushing, not pulling. The BBC’s new radio player uses Comet over a Flash XMLsocket connection transport, with an ActiveMQ message queue behind the scenes. I’d like to know what server they’re using to broadcast out to the XMLsocket connections. 13th January 2009, 12:59 am

ptth (Reverse HTTP) implementation in a browser using Long Poll COMET. Donovan Preston experiments with the cleverly named idea of ptth, where servers send HTTP requests to clients. 8th December 2008, 5:22 pm

FriendFeed launch a real-time API. This is huge: JSONP plus long polling Comet, with “everything since X” tokens to ensure you don’t miss anything. This is the first open Comet API I’ve seen anywhere. Combine this with FriendFeed’s regular API (which allows arbitrary message posting) and you’ve got a really powerful tool for hackers who want to experiment with Comet without rigging up their own infrastructure. 22nd October 2008, 2:18 pm

A Million-user Comet Application with Mochiweb, Part 1. Richard Jones explores Mochiweb, Erlang and linux kernel tuning for building a high performance comet server. Does this mean real-time web features are coming to last.fm? 16th October 2008, 2:16 pm

Private Messages with cometD Chat. cometd-java (a Java servlet reference implementation of the Bayeux protocol) can be extended with BayeuxService subclasses that run within the server itself. 16th October 2008, 2:14 pm

View your FriendFeed in real-time. FriendFeed become the latest site to enable real-time updates using the long-polling variant of Comet. The real-time Web was something of a theme at this year’s FOWA, with talks on message queues, XMPP and scaling Comet at Meebo. 16th October 2008, 2:06 pm

Spawning + Django. The latest version of Spawning (a fast Python web server built on top of the Eventlet non-blocking coroutine networking library) can run Django applications out of the box, using threads and processes to work around the blocking nature of the ORM’s database drivers. Eric Florenzano reports better performance than Apache and mod_wsgi, and is now hosting his site on it. 31st July 2008, 10:56 am

How Comet Brings Instant Messaging to meebo. “What started off as a hack appears to be fulfilling one of the most basic needs of the Web, which is live synchronous interaction”—Jian Shen 27th July 2008, 11:18 am

Independence Day: HTML5 WebSocket Liberates Comet From Hacks. The HTML5 spec now includes WebSocket, a TCP-style persistent socket mechanism between client and server using an HTTP handshake to work around firewalls. The Orbited comet implementation provides a WebSocket compatible API to existing browsers today, and can also act as a firewall/proxy between WebSocket and regular TCP sockets, allowing browsers to talk to things like XMPP servers using Orbited to bridge the gap. 4th July 2008, 9:54 am

Enough Already with the Connections! Comet doesn’t mean making long-lived HTTP connections (which most browsers do anyway thanks to HTTP keep-alive), it means making long-held HTTP requests. I’m guilty of spreading this misinformation in the past. 30th June 2008, 9:27 am

Comet (programming) on Wikipedia on 4th June 2008 (via) The last useful version (which I had pointed many people to) before it was gutted down to just a couple of paragraphs by infuriating deletionists. 16th June 2008, 9:34 am

Saturday Mornings: Going Live! Some 1980s/1990s British nostalgia. I pinched a video of the theme tune from here for my talk on Comet at Brighton SkillSwap. 12th June 2008, 12:14 am

Google Finance Comet. Google Finance now shows live stock quotes, updated by Comet. 4th June 2008, 8:36 am

On-board vs. Off-board Comet. Useful distinction. On-board comet runs on the same server as the rest of your application; Off-board comet is served from a separate server (generally a subdomain) and a separate stack. If you want to stick with PHP, Rails or Django for the rest of your site off-board comet looks like the way to go. 22nd May 2008, 5:02 pm

Engineering @ Facebook: Facebook Chat. The new Facebook Chat uses Comet (long polling with a hidden iframe) against a custom web / chat server written in Erlang, designed to handle a launch to all 70 million users at once. It was tested using a “dark launch” period where live pages simulated chat request traffic without showing any visible UI. 15th May 2008, 7:55 am

Persevere adds Comet Support. Persevere sounds neat: a RESTful HTTP/JSON data store (the interface reminds me of CouchDB) which recently gained the ability to “subscribe” to a resource and receive notifications of updates via comet. 13th May 2008, 8:09 am

Comet at the Highland Fling. I thoroughly enjoyed the Highland Fling yesterday. Here are the slides from my talk on Comet. 4th April 2008, 10:13 am

A Toy Chat Server with Eventlet and Mulib (via) Eventlet (the Python non-blocking IO library originally written for Second Life) is ideally suited to building Comet servers; Chuck Thier demonstrates a simple chat server in a small amount of code. 21st March 2008, 3:28 am

Internet Explorer 8 Readiness Toolkit. The new built-in development tools look similar enough to Firebug to make me very happy. Also of interest: Selectors API (for fast getElementsBySelector), CSS 2.1 support, support for XHTML style namespaces in HTML, an interesting Web Slices feature based on the hAtom microformat and 6 connections per host (up from 2) which should make Comet easier. 5th March 2008, 6:28 pm

Gears 0.2 Released! New modules are HttpRequest and Timer, both for use within workers (which provide Erlang-style message passing concurrency). Particularly interesting is that the Gears HttpRequest module can be used for much cleaner Comet implementations in IE. 5th March 2008, 12:21 am

django-evserver. Marek Majkowski got Comet working with Django using a custom WSGI server that wraps libevent using ctypes. 19th January 2008, 12:15 pm

$.comet (via) The first Comet (with Bayeux) plugin I’ve seen for jQuery—currently only handles long-polling over XMLHttpRequest, but still a promising start. 9th January 2008, 8:31 am

Why we switched to Jetty. Zimbra (recently acquired by Yahoo!) are using Jetty for Comet. It sounds like they are using Bayeux as well. 8th January 2008, 5:12 am

20,000 Reasons Why Comet Scales. Greg Wilkins coaxes Jetty and Bayeux in to supporting 20,000 simultaneous users per server while maintaining sub-second latency, using Amazon EC2 to run the benchmark. 7th January 2008, 8:32 am

Django and Comet. How to build a chat application using Django and the Orbited comet server. Orbited can be set up to proxy most requests through to a Django backend while handling any comet requests itself. 26th December 2007, 9:05 pm

The Future of Comet: Part 1, Comet Today. Absolutely the best summary I’ve seen of all of the current Comet techniques in one place. 11th December 2007, 1:13 pm

JavaScript Internationalisation, explained by reindeer. “Santa even spooked Comet recently by talking about him as if he were some pushy web server.” 8th December 2007, 2:04 pm

Call for Participation for XTech 2008. XTech 2008 will be in Dublin, Ireland from the 6th to the 9th of May. Lots of really interesting topics in the CfP (OpenID, OAuth, Comet, CouchDB...)—deadline for submissions is the 25th of January. 5th December 2007, 3:28 pm

Jetty WebServer. Jetty 6.1 was the only cometd / Bayeux implementation I tried which worked out of the box. 27th November 2007, 6:43 pm

A Taxonomy of Event- and REST-based Comet. Kris Zyp describes a conceptual model for Comet messages based on REST semantics (so you can send a PUT referencing a specific URI down to a client to represent an idempotent state change). 21st November 2007, 8:18 pm

IE ActiveX(“htmlfile”) Transport, Part II. Fascinating tricks for working around IE memory leaks using explicit CollectGarbage() calls and setInterval() to an empty function. 19th November 2007, 11:48 am

Ten New Things in WebKit 3. Does “incremental updates for persistent server connections” for XMLHttpRequest mean Safari now has native support for Comet? 16th November 2007, 1:19 am

Orbited: The Orbit Event Daemon. HTTP daemon designed for long-lasting comet connections, written in Python using pyevent on top of libevent. 9th November 2007, 11:01 pm

Comet Daily. New regularly updated site covering Comet, the Ajax-like umbrella term for JavaScript server-push techniques. Already a bunch of great stuff on there. 7th November 2007, 10:53 am

Bayeux. Comet might just make Java relevant for web development again. 7th August 2006, 11:51 am