Simon Willison’s Weblog

Why Facebook open-sourced its datacenters. Jon Stokes speculates that Facebook plan to use open source hardware to compete with Google at datacenter efficiency . This isn’t a new pattern. Years ago when I worked at Yahoo! I was furiously jealous of the secret sauce technologies that allowed Google to build big applications faster than anyone else, such as BigTable and map/reduce. Today, the open source world has created better, free alternatives—sponsored in part by Facebook, Yahoo! and other Google competitors. 9th April 2011, 7:54 am

Slide, Inc.—open source. slide.com have open sourced a whole bunch of interesting Python libraries, most of them involving C extensions or greenlet non-blocking I/O. wirebin (fast binary serialization of native Python types) and meminfo (an extension for finding precise in-memory sizes of Python objects) look particularly interesting. No documentation yet—not even a readme. 17th June 2010, 8:05 pm

tobeytailor’s gordon. Another Flash runtime in pure JavaScript project, released back in January. Not quite as advanced as Smokescreen yet (it doesn’t have an audio implementation) but already available as open source under an MIT license. 29th May 2010, 11:57 am

Django 1.2 release notes (via) Released today, this is a terrific upgrade. Multiple database connections, model validation, improved CSRF protection, a messages framework, the new smart if template tag and lots, lots more. I’ve been using the 1.2 betas for a major new project over the past few months and it’s been smooth sailing all the way. 17th May 2010, 9:11 pm

RE2: a principled approach to regular expression matching. Google have open sourced RE2, the C++ regular expression library they developed for Google Code Search, Sawzall, Bigtable and other internal projects. Unlike PCRE it avoids the potential for exponential run time and unbounded stack usage and guarantees that searches complete in linear time, mainly by dropping support for back references. 12th March 2010, 9:28 am

Symbian Operating System, Now Open Source and Free. With Symbian now open source, are there any widely used operating systems left (besides Windows) that don’t have an open source core? 4th February 2010, 8:38 am

The Maximal Usage Doctrine for Open Source. Yehuda Katz shares my own philosophy on Open Source licensing—stick BSD or MIT on it to maximise the number of people who can use it. The projects I work on are small enough that I don’t care if someone makes big private improvements and refuses to share them. I can see how much larger projects like Linux would disagree though. 6th January 2010, 5:23 pm

Semantic Versioning. Tom Preston-Werner provides a name, specification and URL describing the relatively widely used Major.Minor.Patch versioning system. This is really useful—by giving something a name and a spec, people can say “this project uses semantic versioning” and skip having to explain their backwards compatibility policy in full. 15th December 2009, 9:53 pm

EtherPad is Back Online Until Open Sourced. Fantastic news. EtherPad just got acquired by Google and announced the team would be joining the Google Wave effort and the existing service would be shut down. Lots of people complained, so they’re going to keep it alive until they’ve open sourced the code! 6th December 2009, 9:08 am

Opening Up Librelist.com Code, Looking For Volunteers. Zed Shaw’s Librelist is a new service for open source project mailing lists, aiming to be donation supported in a similar way to Freenode IRC. The code is all available, and is written in Lamson and Django. 4th December 2009, 9:25 am

Announcing Kong: A server description and deployment testing tool. An ultra simple website monitoring tool written in Django which makes it easy to manage a list of Twill scripts for testing different sites. It was developed at the Lawrence Journal-World—Eric showed me a demo if this a year or so ago and I’ve been hoping they would open source it. 18th November 2009, 12:47 pm

Introducing the YUI 3 Gallery. Write a plugin for YUI3, BSD license it and sign a CLA and Yahoo! will push your module out to their CDN and make it loadable using the YUI().use() statement. They’re coordinating the submissions using GitHub. 4th November 2009, 11:14 pm

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI. 4th November 2009, 8:20 pm

Traffic Server. Mark Nottingham explains the release of Traffic Server, a new Apache Incubator open source project donated by Yahoo! using code originally developed at Inktomi around a decade ago. Traffic Server is a HTTP proxy/cache, similar to Squid and Varnish (though Traffic Server acts as both a forward and reverse proxy, whereas Varnish only handles reverse). 1st November 2009, 12:15 pm

Introducing Cloudera Desktop. It’s a GUI for Hadoop, and under the hood is a whole stack of open source software, including Python, Django, MooTools, Twisted, lxml, CherryPy, Mako, Java and AspectJ. 21st October 2009, 6:48 pm

Kung Fu People (via) The first site to launch based on the open source Django code from djangopeople.net! 19th August 2009, 11:37 am

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub. 13th August 2009, 1:51 pm

Django 1.1 release notes (via) Django 1.1 is out! Congratulations everyone who worked on this, it’s a fantastic release. New features include aggregate support in the ORM, proxy models, deferred fields and some really nice admin improvements. Oh, and the testing framework is now up to 10 times thanks to smart use of transactions. 29th July 2009, 9:34 am

NASA NEBULA Services (via) NASA’s new NEBULA cloud computing platform appears to be built entirely on open source infrastructure, including Python, Django, Fabric, Eucalyptus, RabbitMQ, Trac and Solr. 28th July 2009, 12:10 pm

Popfly Shutting Down. Yet another reminder that building stuff on a closed-source platform (especially a hosted service) is risky business, even from a vendor as large as Microsoft. This certainly won’t help them make the case for Azure. 17th July 2009, 9:32 am

Twenty questions about the GPL. Jacob kicks off a fascinating discussion about GPLv3. 13th July 2009, 11:59 pm

BBC: Glow (via) The BBC have released Glow, their jQuery-like JavaScript library developed in house over the past few years. It’s open source under the Apache license. 8th July 2009, 3:25 pm

Yahoo! proposal to open source “Traffic Server” via the ASF. Traffic Server is a “fast, scalable and extensible HTTP/1.1 compliant caching proxy server” (presumably equivalent to things like Squid and Varnish) originally acquired from Inktomi and developed internally at Yahoo! for the past three years, which has been benchmarked handling 35,000 req/s on a single box. No source code yet but it looks like the release will arrive pretty soon. 7th July 2009, 12:37 pm

From Microsoft: C# and CLI under the Community Promise. Microsoft’s assurance that it won’t “assert its Necessary Claims” against alternative (including open source) implementations of the ECMA C# and CLR specifications. The promise doesn’t cover implementations of .NET, WinForms etc- so the Mono team have announced they will be splitting their project in to two packages—a safe, ECMA based package and a package containing everything else. 7th July 2009, 11:15 am

Stellarium. Really lovely open source planetarium application, for Linux, OS X and Windows. 7th July 2009, 12:37 am

EveryBlock source code released. EveryBlock’s Knight Foundation grant required them to release the source code after two years, under the GPL. Lots of neat Django / PostgreSQL / GIS tricks to be found within. 1st July 2009, 8:01 pm

PostgreSQL Development Priorities. The top two for 8.4 are “Simple built-in replication” and “Upgrade-in-place”, Josh Berkus is seeking feedback on priorities for future work on 8.5. 28th May 2009, 8:08 pm

djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in. 4th May 2009, 6:12 pm

Reducing XSS by way of Automatic Context-Aware Escaping in Template Systems (via) The Google Online Security Blog reminds us that simply HTML-escaping everything isn’t enough—the type of escaping needed depends on the current markup context, for example variables inside JavaScript blocks should be escaped differently. Google’s open source Ctemplate library uses an HTML parser to keep track of the current context and apply the correct escaping function automatically. 14th April 2009, 9:26 am

Ext Core 3.0 Beta Released. The Ext JavaScript team have just released the core library (similar to jQuery or Prototype) under an MIT license. The rich GUI elements that go on top are still under the GPL. 5th April 2009, 8:17 pm

DB2 support for Django is coming. From IBM, under the Apache 2.0 License. I’m not sure if this makes it hard to bundle it with the rest of Django, which uses the BSD license. 18th February 2009, 10:58 pm

Whoosh. A brand new, pure-python full text indexing engine (think Lucene). Claims to offer performance in the same league as wrappers to C or Java libraries. If this works as well as it claims it will be an excellent tool for adding search to projects that wish to avoid a dependency on an external engine. 12th February 2009, 12:49 pm

Apache Qpid. A new open source AMQP message queue with implementations in C++ and Java, developed by engineers from Red Hat, IONA and JP Morgan Chase. Anyone tried this yet? Looks pretty good on paper. 5th February 2009, 11:01 pm

google-mobwrite. Neal Fraser’s terrifyingly clever differential synchronization algorithm (for SubEthaEdit-style collaboration over the web) is now available as an open source Python and JavaScript library. 24th January 2009, 11:55 pm

Project Voldemort. Yet Another “big, distributed, persistent, fault-tolerant hash table”—this time from LinkedIn, released under the Apache 2.0 license. The approach to consistency is interesting—instead of using distributed transactions, they use versioning and “resolve inconsistencies at read time”. It also uses consistent hashing (as seen in libketama) to select servers. The design document has lots more information. 17th January 2009, 7:45 pm

OurDelta Builds for MySQL (via) A community supported “alternative distro” of MySQL, incorporating new features from Google and other sources by maintaining a clean set of patches against the MySQL source tree (which I guess is why it’s not considered a fork). I recognise some of the patches from the excellent “High Performance MySQL, 2nd Edition”. 8th December 2008, 4:20 pm

License Hacking. Wikipedia is making the switch to a CC license, by asking the Free Software Foundation to include that as an option in the latest version of the Free Documentation License which Wikipedia currently uses and which includes an auto-upgrade clause. Devious. 10th November 2008, 10:46 pm

Windows Server and SQL Server on EC2 (via) Launched today, the pricing includes rental of the Windows license. Regular Windows is 25% to 50% more expensive than Linux, but SQL Server comes in at a hefty $1.10 per hour, which is $9636 per year (nearly three times as much as a Linux server running an open source database). 23rd October 2008, 3:54 pm

Cappuccino Web Framework. Now open source (LGPL)—the Objective-C-in-JavaScript web application toolkit from 280 North, who are speaking at this year’s FOWA in October. Beautiful logo. 5th September 2008, 3:27 pm

Chromium. Google Chrome is out! Here’s the open source project, including the code for the new V8 JavaScript virtual machine. 2nd September 2008, 9:06 pm

Free licenses upheld by US “IP” court. Free software and CC licenses which dictate conditions that, when violated, turn you in to a copyright infringer now have precedence in US law. 14th August 2008, 9:33 am

The Price of Anonymity: Our Principles? Alex Russell calls for a constructive step towards better gender balance in open source: make it clear that misogynistic, offensive and lewd behaviour will not be tolerated by open source communities and bake that policy in to community codes of conduct. 28th July 2008, 12:44 am

FLOSS Weekly 34: Django. Randal Schwartz interviewed Jacob Kaplan-Moss at OSCON for the consistently excellent FLOSS Weekly podcast. 27th July 2008, 9:47 am

What the Heck is the Open Web? Brad Neuberg is seeking a two sentence definition. Bonus points for answering the following: “If Adobe were to open source Flex/Flash, or Microsoft Silverlight, would that be the Open Web? If so, why? If not, why not?” 22nd July 2008, 1:33 am

Protocol Buffers: Google’s Data Interchange Format. Open sourced today. Highly efficient binary protocol for storing and transmitting structured data between C++, Java and Python. Uses a .proto file describing the data structure which is compiled to classes in those languages for serializing and deserializing. 3-10 times smaller and 20-100 times faster than XML. 8th July 2008, 8:20 am

Ruby’s Vulnerability Handling Debacle. The critical Ruby vulnerabilities are over a week old now but there’s still no good official patch (the security patches cause segfaults in Rails, leaving the community reliant on unofficial patches from third parties). Max Caceres has three takeaway lessons, the most important of which is to always keep a “last-known-good” branch to apply critical patches to. 2nd July 2008, 10:39 am

Reddit release their codebase. Under the same Common Public Attribution License used by Facebook for their recent source release. 18th June 2008, 2:32 pm

Facebook Open Platform. Facebook have open-sourced (under a modified MPL, does it still fit the OSI definition?) the code for the Facebook Platform, including their implementations of FBML, FQL and FBJS. This is no small release; the tarball weighs in at 40MB and includes libfbml, which depends on Firefox 2.0.0.4 for its HTML parser! 3rd June 2008, 12:21 am

The Carbon Account. The carbon calculator project I contributed to at Torchbox last year has launched, and they’ve made the code available as open source. 30th March 2008, 7:04 pm

Standing in Line. Simon Wistow coins “CLAMP” for LAMP + Cache, and expresses the need for a dirt-simple, high performance open source queue system. 19th March 2008, 9:41 am

Flickr Uploadr: Open Source and Powered by XULRunner. Quietly released a few months ago; it’s really nice. 13th February 2008, 11:30 pm

Photo Matt: Act Two. Automattic is an excellent case-study of building a business on top of an open source project. 23rd January 2008, 10:42 am

Sun To Acquire MySQL. Sun also employ Josh Berkus, one of the lead developers of PostgreSQL. 16th January 2008, 1:55 pm

Good architectural layering, and Bzr 1.1. Mark Shuttleworth on the growing importance of plug-in architectures as an open source project evolves, as they allow new developers to release their own components without needing commit access to the project. Django is pretty good for this, but more hooks (and a faster event dispatch system) would be useful. 9th January 2008, 2:06 pm

The Dark Side Of The Moon (via) Robert O’Callahan believes that Moonlight is a strategic mistake, because it gives credibility to Microsoft’s entry to a new market which they will use to “keep the competition on a treadmill”; Moonlight can also never be entirely free due to the need for a proprietary codec (VC-1) available only as a binary blob. 4th January 2008, 12:41 pm

The future of web standards. Nice analysis from James Bennett, who suggests that successful open source projects (Linux, Python, Perl etc) could be used as the model for a more effective standards process, and points out that Ian Hickson is something of a BDFL for the WHAT-WG. 17th December 2007, 1:16 pm

A Little Laptop With Big Ambitions. I hadn’t realised how much competition OLPC faced from Microsoft and Intel’s Classmate. It would be amazing to see a generation grow up understanding that computers are open tools that they can control themselves rather than closed black boxes. 24th November 2007, 10:47 pm

Proprietary Software Does Not Scale. I’ve been thinking this for a while: if you’re using software with a per-CPU license you can’t just roll it out as an image across a bunch of virtual machines when you need to. 18th November 2007, 12:30 am

/trunk/jl/scraper. journa-list.com is open source, and the screen scrapers are written in Python. 11th October 2007, 4:10 pm

Kosmos Distributed File System (via) New open source distributed filesystem similar to Google’s GFS. 28th September 2007, 9:12 am

The Rubinius Sprint. Sun are throwing a ton of resources at Ruby, because as Tim Bray says, “it’s not fast enough”. Imagine where they’d be if they’d invested this kind of support in Jython five years ago... 21st September 2007, 11:32 pm

Why Tamarin instead of... Justification for Tamarin in Mozilla over Mono and the JVM. It mainly comes down to license compatibility and overall size. 9th August 2007, 12:43 pm

Sweet Gig. SitePen seek “R&D Associate” to have fun hacking on Open Source software and researching whatever they think is important. 7th August 2007, 2:47 pm

Legal advice for businesses that are scared of open source (via) Useful for pointing people to. 31st July 2007, 9:59 pm

Django weekly roundup: July 30. Every active open source project needs something like this. 30th July 2007, 5:03 pm

Grub. Jimmy Wales just announced at OSCON that Wikia have acquired Grub from LookSmart, and will be releasing it as open source. 27th July 2007, 5:24 pm

gSculpt. Powerful open source modelling software, written in Python and demonstrated (to much applause) as the last lightning talk of EuroPython 2007. 11th July 2007, 11:48 pm

Implementing Silverlight in 21 Days. Absolutely incredible feat of software engineering by Miguel de Icaza and the Moonlight team. 21st June 2007, 11:10 am

Croquet. Open-source collaborative virtual world environment built on top of Squeak, a bit like a decentralised version of Second Life. 10th June 2007, 10:50 am

google-diff-match-patch (via) Robust algorithms to perform the operations required for synchronizing plain text, in Java, JavaScript and Python. 9th June 2007, 6:15 pm

Semi-synchronous replication for MySQL (via) Google’s patch for MySQL which enables more reliable master-slave replication (a transaction isn’t committed until at least one slave has replicated the data). 5th June 2007, 10:07 pm

Ten Reasons The World Needs Patent Covenants (via) Sun just made their OpenID patent covenant official. Simon Phipps explains why these are a Good Idea. 22nd May 2007, 5:09 pm

Arduino. Open source hardware hacking. It’s way easier than you would think. 17th May 2007, 6:30 pm

VirtualBox. GPL licensed virtualization software; they recently released an OS X version. 8th May 2007, 9:35 pm

Migrating Microsoft Hotmail from FreeBSD to Microsoft Windows 2000. I’d like to see them try that with Yahoo!’s 100+ properties. 4th May 2007, 5:54 pm

MSFT and Yahoo: two icebergs, roped together. Yahoo!’s engineering platform and culture is Open Source pretty much all the way down. Microsoft’s isn’t. I wonder how that would pan out. 4th May 2007, 5:50 pm

Adobe open sources Flex. Ted Leung says that this might indicate the possibility of Adobe open sourcing Flash itself in the future. 26th April 2007, 11:24 am

New Open Source Utility Library for the Google Maps API (via) Google are taking a hybrid approach to development on their Maps API—an open source utility library layered on top of their closed source, obfuscated core code. 23rd March 2007, 9:09 pm

opensource @ Joost. Joost is built on top of Mozilla, Redland, SQLite and a bunch of other bits and pieces of Open Source infrastructure. 12th March 2007, 1:29 pm

Mono-based device wins Best-of-Show at CES. “The Sansa Connect is running Linux as its operating system, and the whole application stack is built on Mono, running on an ARM processor.” 17th January 2007, 11:21 pm

Correo. New open-source OS X mail client, based on Thunderbird but with a Camino-style native interface. 12th January 2007, 11:36 am

The Second Life Viewer is now open-source (via) I’d heard that the biggest barrier to this was the need to protect the SL economy from malicious disruption. The FAQ is fascinating, and a real tribute to open-source principles. 8th January 2007, 6:47 pm

GWT 1.3 Release Candidate is 100% Open Source. At least you can see how the code generator works now. 12th December 2006, 5:50 pm

Microsoft versus FOSS Configuration Management. Why the Free Software world’s source control works and Vista’s apparently doesn’t. 7th December 2006, 9:28 am

I want my OpenID! Open ID code bounties worth $5,000. Could this kickstart adoption? 28th July 2006, 4:02 pm

Yahoo! UI Library. Open Source JavaScript widgets and libraries. 14th February 2006, 1:12 am

DHS Funding Open Source Security. Paying for “source code analysis technology” coverage of Linux, Apache, PostgreSQL and more. 17th January 2006, 10:18 pm