Simon Willison’s Weblog

How does Sidekiq really work? (via) I really like this category of blog post: Dan Svetlov took the time to explore the Sidekiq message queue’s implementation and then wrote it up in depth. # 5th February 2024, 5:20 pm

Build impossible programs. Delightful talk by Julia Evans describing how she went about building a Ruby profiler in Rust despite having no knowledge of Ruby internals and only beginner’s knowledge of Rust. # 19th September 2018, 6:38 pm

Ruby-style Blocks in Python. Yes, yes, yes, yes. A proposal for muli-line lambda support in Python that doesn’t trip up on significant whitespace. If this gets in before the proposed feature freeze I’ll be a very happy Pythonista. UPDATE: This is a post from over a year ago, and it looks like the proposal has since stalled. # 23rd April 2010, 11:19 am

grammar.coffee (via) The annotated grammar for CoffeeScript, a new language that compiles to JavaScript developed by DocumentCloud’s Jeremy Ashkenas. The linked page is generated using Jeremy’s Docco tool for literate programming, also written in CoffeeScript. CoffeeScript itself is implemented in CoffeeScript, using a bootstrap compiler originally written in Ruby. # 8th March 2010, 7:27 pm

twitter-text-conformance (via) This is a neat idea: Twitter have released open source libraries for parsing standard tweet syntax in Ruby and Java, but they’ve also released a set of YAML unit tests aimed at anyone who wants to implement the same parsing logic in other languages. # 6th February 2010, 3:39 pm

rlisagor’s freshen. A Python clone of Ruby’s innovative Cucumber testing framework. Tests are defined as a set of plain-text scenarios, which are then executed by being matched against test functions decorated with regular expressions. Has anyone used this or Cucumber? I’m intrigued but unconvinced—are the plain text scenarios really a useful way of defining tests? # 5th January 2010, 7:30 pm

clarity. A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine. # 4th November 2009, 10:36 pm

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI. # 4th November 2009, 8:20 pm

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine. # 21st October 2009, 9:14 pm

MySQL backups with EBS snapshots. Assaf Arkin’s 45 line ruby script shows how to lock tables / XFS freeze / create an EBS snapshot / unfreeze and unlock, with hourly snapshots preserved for the past 24 hours and daily snapshots for the past week. Is an EBS snapshot enough to restore your data to somewhere other than EC2 though? # 13th October 2009, 12:34 pm

XSS Protection by Default in Rails 3.0. Fantastic news—congratulations, Rails core team. # 8th October 2009, 4:35 pm

I like Unicorn because it’s Unix. Ryan Tomayko analyses Unicorn, a new, pre-forking Ruby HTTP server that makes extensive use of Unix syscalls and idioms, and asks why dynamic language programmers don’t take advantage of these more often. # 7th October 2009, 11:42 am

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm

homebrew. Exciting alternative to MacPorts for compiling software on OS X—homebrew avoids sudo and defines packages as simple Ruby scripts, shared and distributed using Git. # 21st September 2009, 6:51 pm

Webhooks behind the firewall with Reverse HTTP. Hookout is a Ruby / rack adapter that lets you serve a web application from behind a firewall, by binding to a Reverse HTTP proxy running on the internet (such as the free one provided by reversehttp.net). Useful for far more than just webhooks, this means you can easily expose any Ruby web service to the outside world. An implementation of this as a general purpose proxy server would make it useful for applications written in any language. # 22nd July 2009, 1:46 pm

Phusion Passenger for nginx. Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet? # 20th April 2009, 4:53 am

Twitter: blaming Ruby for their mistakes? The comments on the entry include replies from Twitter employees and the RabbitMQ consultant they brought in, and provide a full rebuttal to the various accusations of NIH that were thrown around recently. # 6th April 2009, 11:06 am

Ruby on Rails 2.3 Release Notes. I’m impressed with how thoroughly Rails has embraced Rack (Ruby’s standardised web framework API, inspired by Python’s WSGI). # 15th March 2009, 1:22 pm

CloudMade: A Summary of the Future of Mapping. CloudMade are now offering commercially supported APIs on top of OpenStreetMap, including geocoding, routing and tile access libraries in Python/Ruby/Java and a very neat theming tool that lets you design your own map styles. This is really going to kick innovation around OpenStreetMap up a notch. # 17th February 2009, 11:25 am

juno. An ultra-lightweight Python web framework inspired by Ruby’s Sinatra. # 4th February 2009, 10:48 am

Prawn (via) Really nice PDF generation library for Ruby, used to generate Dopplr’s beautiful end of year reports. # 16th January 2009, 4:04 pm

why’s potion. why’s latest project is a small, fast language (JIT to x86/x86-64) which seems to take ideas from Ruby, Lua, Python and who knows where else. Everything is based around objects, closures and mixins, with the delightful inclusion of scoped mixins so you can modify an object only within a certain module (hence avoiding Ruby’s action-at-a-distance problems). # 8th January 2009, 6:37 pm

Merb gets merged into Rails 3! Huge news. Of particular interest is the new focus on “framework agnosticism”, whereby Rails will aim to play well with people wishing to use alternative ORMs, template mechanisms and so forth. Rails has previously suffered from a reputation for getting in your way if you deviate from its opinions. # 23rd December 2008, 8:32 pm

On packaging. James Bennett discusses the problems with setuptools (and ruby gems), and recommends Ian Bicking’s pip as a setuptools replacement. # 14th December 2008, 4:57 pm

Reia. The most common complaint I see about Erlang is the syntax. Reia is a Python-style scripting language (with a dash of Ruby) that runs on the Erlang virtual machine. Looks promising. # 25th September 2008, 6:12 pm

DoS vulnerability in REXML. Ruby’s REXML library is susceptible to the “billion laughs” denial of service attack where recursively nested entities expand a single entitity reference to a billion characters (kind of like the exploding zip file attack). Rails applications that process user-supplied XML should apply the monkey-patch ASAP; a proper gem update is forthcoming. # 23rd August 2008, 11:11 am

Ruby’s Vulnerability Handling Debacle. The critical Ruby vulnerabilities are over a week old now but there’s still no good official patch (the security patches cause segfaults in Rails, leaving the community reliant on unofficial patches from third parties). Max Caceres has three takeaway lessons, the most important of which is to always keep a “last-known-good” branch to apply critical patches to. # 2nd July 2008, 10:39 am

Whitespace Sensitivity. Amusingly, Ruby is actually far more sensitive about whitespace than Python is. # 1st July 2008, 2:50 pm

Arbitrary code execution vulnerabilities in Ruby (via) Sounds nasty—integer overflows and unsafe allocs leading to memory corruption. Definite DoS and possible code injection as well. Upgrade to Ruby “1.8.5-p231, or 1.8.6-p230, or 1.8.7-p22” ASAP. # 21st June 2008, 8:44 am

Spicing Up Embedded JavaScript. John Resig collects the various ways in which a JavaScript interpreter can be hosted by Python, PHP, Perl, Ruby and Java. There are full JS implementations in PHP, Perl and Java; Ruby and Python both have modules that use an embedded SpiderMonkey. # 15th June 2008, 11:32 am