Simon Willison’s Weblog

Making SQLite extensions npm install’able for Node.js, and on deno.land/x for Deno (via) Alex Garcia figured out how to get his “pip install X” trick for distributing compiled SQLite extensions to work for Node too! Now you can “npm install” 10 of his extensions, including sqlite-regex and sqlite-xsv and sqlite-http and sqlite-html and more, and attach them to a node-sqlite3 or better-sqlite3 connection. He’s bundled them for Deno too! # 29th March 2023, 10:13 pm

I pushed 20 more of my projects to GitHub. Some great Node.js stuff here from Peteris Krumins, including modules for processing PNG, JPEG and animated GIFs. # 23rd September 2010, 1:18 am

nodejitsu’s node-http-proxy (via) Exactly what I’ve been waiting for—a robust HTTP proxy library for Node that makes it trivial to proxy requests to a backend with custom proxy behaviour added in JavaScript. The example app adds an artificial delay to every request to simulate a slow connection, but other exciting potential use cases could include rate limiting, API key restriction, logging, load balancing, lint testing and more besides. # 28th July 2010, 11:34 pm

Multi-node: Concurrent NodeJS HTTP Server. Kris Zyp’s library for spawning multiple Node child processes (one per core is suggested) for concurrent request handling, taking advantage of Node’s child_process module. This alleviates the need to run multiple Node instances behind an nginx load balancer in order to take advantage of multiple cores. # 15th July 2010, 8:22 am

DNode: Asynchronous Remote Method Invocation for Node.js and the Browser. Mind-bendingly clever. DNode lets you expose a JavaScript function so that it can be called from another machine using a simple JSON-based network protocol. That’s relatively straight-forward... but DNode is designed for asynchronous environments, and so also lets you pass callback functions which will be translated in to references and used to make remote method invocations back to your original client. And to top it off, there’s a browser client library so you can perform the same trick over a WebSocket between a browser and a server. # 11th July 2010, 2:27 pm

pdf.js. A JavaScript library for creating simple PDF files. Works (flakily) in your browser using a data:URI hack, but is also compatible with server-side JavaScript implementations such as Node.js. # 17th June 2010, 7:39 pm

Parsing file uploads at 500 mb/s with node.js. Handling file uploads is a real sweet spot for Node.js, especially now it has a high performance Buffer API for dealing with binary chunks of data. Felix Geisendörfer has released a new library called “formidable” which makes receiving file uploads (including HTML5 multiple uploads) easy, and uses some clever algorithmic tricks to dramatically speed up the processing of multipart data. # 2nd June 2010, 3:57 pm

jed’s fab. Spectacular web framework for Node.js which, despite using nothing but regular JavaScript, has syntax that is easily confused with Lisp. General consensus at work is that truly understanding how this works is a crucial step on the path to JavaScript enlightenment. # 18th May 2010, 6:50 pm

Understanding node.js. A king providing orders to his army of servants is a much better analogy than my hyperactive squid. # 18th May 2010, 6:44 pm

Realtime Election Tweets. Jay Caines-Gooby’s realtime election tweet service, using Node.js, nginx and WebSocket with a Flash fallback. # 6th May 2010, 9:20 pm

A HTTP Proxy Server in 20 Lines of node.js. Proxying is definitely a sweet spot for Node.js. Peteris Krummins takes it a step further, adding host blacklists and an IP whitelist as configuration files and using Node’s watchFile method to automatically reload changes to them. # 28th April 2010, 1:24 pm

tempalias.com development diary (via) tempalias.com is a e-mail forwarding service that lets you create an address that will only work for a few days (or a limited number of messages) and will forward messages on to your real account. It’s implemented using Node.js and Redis and the code is released under an MIT license. Philip Hofstetter, the developer, maintained a detailed development diary throughout which is worth reading if you’re interested in Node.js. # 23rd April 2010, 7:36 pm

webhook-relay. Another of my experiments with Node.js: webhook-relay is a self-contained queue and webhook request sending agent. Your application can POST to it specifying a webhook alert to be sent off, and webhook-relay will place that request in an in-memory queue and send it on its own time, avoiding the need for your main application server to block until the outgoing request has been processed. # 19th March 2010, 10:17 am

Node.js, redis, and resque (via) Paul Gross has been experimenting with Node.js proxies for allowing web applications to be upgraded without missing any requests. Here he places all incoming HTTP requests in a redis queue, then has his backend Rails servers consume requests from the queue and push the responses back on to a queue for Node to deliver. When the backend application is upgraded, requests remain in the queue and users see a few seconds of delay before their request is handled. It’s not production ready yet (POST requests aren’t handled, for example) but it’s a very interesting approach. # 28th February 2010, 11:02 pm

kriszyp’s node-promise. Another elegant approach to managing asynchronous flows in Node, including running things both in parallel and serial. # 28th February 2010, 3:50 pm

“Do” it fast! Tim Caswell’s Do library has been upgraded for compatibility with Node v0.1.30, and now has a clever Do.convert() method which wraps Node’s low-level APIs with the Do libraries “continuable” abstraction. # 22nd February 2010, 7:02 pm

node-v0.1.30 (via) A very significant new release of Node.js: the Twisted/Dojo-style Promise abstraction has been removed entirely, causing backwards incompatible changes to a bunch of core APIs. This means the pseudo-blocking Promise.wait() method is gone too, making it even harder to accidentally block your event loop. Instead, user-level libraries are encouraged to add Promise-style abstractions. I’m pleased to see Node sticking to the low-level stuff. # 22nd February 2010, 7 pm

do. A library for Node that adds a higher level abstraction for dealing with chained and parallel callbacks. # 17th February 2010, 5:43 pm

How To Node. New blog about Node.js, with a superb series of tutorials aimed at both experienced and new JavaScript developers. The stuff on managing callbacks (including running them in both series and parallel) is pretty eye-opening. # 17th February 2010, 5:42 pm

dogproxy. Another of my experiments with Node.js—this is a very simple HTTP proxy which addresses the dog pile effect (also known as the thundering herd) by watching out for multiple requests for a URL that is currently “in flight” and bundling them together. # 3rd February 2010, 1:05 pm

Plurk: Instant conversations using Comet (via) Plurk’s comet implementation sounds pretty amazing. They’re using a single quad-core server with 32GB of RAM running 8 Node.js instances to serve long-polled comet to 100,000+ simultaneous users. They switched to Node from Java JBoss/Netty and found the new solution used 10 times less memory. # 1st February 2010, 10:13 am

Hot Code Loading in Node.js. Blaine Cook’s patch for Node.js that enables Erlang-style hot code loading, so you can switch out your application logic without restarting the server or affecting existing requests. This could make deploying new versions of Node applications trivial. I’d love to see a Node hosting service that allows you to simply upload a script file and have it execute on the Web. # 31st January 2010, 1:57 pm

Real time online activity monitor example with node.js and WebSocket. A neat exploration of Node.js—first hooking a “tail -f” process up to an HTTP push stream, then combining that with HTML 5 WebSockets to achieve reliable streaming. # 8th December 2009, 11:07 pm

Perl: Love it, or hate it, but don’t ignore it. Phillip Smith calls me out for omitting Perl from my list of Node.js event loop alternatives (I only mentioned Twisted and EventMachine). No conspiracy here, I’m just not connected enough to the Perl community to know what the popular event loop libraries are. To Perl’s credit, Perlbal was the first piece of software I saw that showed me how a single threaded, event loop based system could massively outperform a threaded alternative. # 27th November 2009, 7:51 am

Request Routing With URI Templates in Node.JS. I quite like this approach (though the implementation is a bit “this” heavy for my taste). JavaScript has no equivalent to Python’s raw strings, so regular expression based routing ala Django ends up being a bit uglier in JavaScript. URI template syntax is more appealing. # 24th November 2009, 9:06 am

node.js at JSConf.eu (PDF). node.js creator Ryan Dahl’s presentation at this year’s JSConf.eu. The principle philosophy is that I/O in web applications should be asynchronous—for everything. No blocking for database calls, no blocking for filesystem access. JavaScript is a mainstream programming language with a culture of callback APIs (thanks to the DOM) and is hence ideally suited to building asynchronous frameworks. # 17th November 2009, 6:07 pm

Going evented with Node.js. Comprehensive Node.js tutorial—from basic principles to installation and writing a simple Twitter search command-line client application. # 17th November 2009, 1:09 pm

node.js. “Evented I/O for V8 JavaScript”—a JavaScript environment built on top of the super-fast V8 engine which provides event-based IO functionality for building highly concurrent TCP and HTTP servers. The API design is superb—everything is achieved using JavaScript events and callbacks (even regular file IO) and the small standard library ships with comprehensive support for HTTP and DNS. Overall it’s very similar to Twisted and friends, but JavaScript’s anonymous function syntax feels more natural than the Python equivalent. It compiles cleanly on Snow Leopard. Definitely a project to watch. # 9th November 2009, 11:25 pm