Simon Willison’s Weblog

Test-Driven Heresy. Tim Bray advocates TDD for maintenance development, but argues that it may not be as useful during the exploratory, greenfield development phase of a project.

# 11:03 am / tdd, unittests, tim-bray, testing

Four crowdsourcing lessons from the Guardian’s (spectacular) expenses-scandal experiment. Michael Andersen from the Nieman Journalism Lab interviewed me about the MP expenses crowdsourcing site.

# 3:31 pm / mpsexpences, guardian, interviews, crowdsourcing

The Resource Expert Droid. Like the HTML Validator but for your server’s HTTP headers—extremely useful.

# 10:06 am / http, headers, validator, resourceexpertdroid, mark-nottingham

What’s New In Python 3.1. Lots of stuff, but the best bits are an ordered dictionary type (congrats, Armin), a Counter class for counting unique items in an iterable (I do this on an almost daily basis) and a bunch of performance improvements including a rewrite of the Python 3.0 IO system in C.

# 3:02 pm / python, python3, python31, performance, releases, armin-ronacher

BashReduce. Map/Reduce in Bash is no longer a joke project (if it ever was)—Richard Crowley is extending it and using it for analysis at OpenDNS.

# 3:03 pm / mapreduce, bash, bashreduce, richardcrowley, opendns

Twitter, an Evolving Architecture. The most detailed write-up of Twitter’s current architecture I’ve seen, explaining the four layers of cache (all memcached) used by the Twitter API.

# 3:09 pm / caching, twitter, memcached, architecture

cache-money. A “write-through caching library for ActiveRecord”, maintained by Nick Kallen from Twitter. Queries hit memcached first, and caches are automatically kept up-to-date when objects are created, updated and deleted. Only some queries are supported—joins and comparisons won’t hit the cache, for example.

# 3:17 pm / activerecord, caching, rails, twitter, cachemoney, memcached

Firefox 3.5 for developers. It’s out today, and the feature list is huge. Highlights include HTML 5 drag ’n’ drop, audio and video elements, offline resources, downloadable fonts, text-shadow, CSS transforms with -moz-transform, localStorage, geolocation, web workers, trackpad swipe events, native JSON, cross-site HTTP requests, text API for canvas, defer attribute for the script element and TraceMonkey for better JS performance!

# 6:08 pm / firefox, html5, dragndrop, audio, video, offlineresources, fonts, textshadow, csstransforms, localstorage, geolocation, webworkers, json, crossdomain, canvas, tracemonkey, javascript, performance, browsers, mozilla, firefox35

MongoDB. Lots of discussions about this at EuroPython today—it’s a document database, very similar to CouchDB but significantly faster and suggested for production use. Best of all, trying it out on OS X is as easy as extracting the tarball and running “bin/mongod --dbpath /tmp/test-mongo-db run”.

# 7:13 pm / couchdb, europython, mongodb, nonrelationaldatabase, keyvaluestore, documentstore, json, osx

Using Mongo for Real-Time Analytics. MongoDB supports an “upsert” query, which when combined with the $inc operator can cause counter fields to be incremented if they exist and created otherwise. This makes it a great fit for real-time analytics applications (one increment per page view), something that regular relational databases aren’t particularly good at.

# 7:28 pm / mongodb, increment, counters, databases, upsert