Simon Willison’s Weblog

Items in Aug, 2010

Filters: Year: 2010 × Month: Aug ×

RasterWeb: Lanyrd. Pete Prodoehl calls me out on Lanyrd’s integration with the Twitter auth API at the expense of OpenID. I’ve posted a comment with my justification—essentially, tying to Twitter’s ecosystem means I can actually implement the features I’ve been talking about building on top of OpenID for years, with far less engineering effort. # 31st August 2010, 8:49 pm

Lanyrd—the social conference directory. Nat and my new project, launched today and doing pretty well despite some early server hiccups. Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events. We’re particularly keen on helping people build up a detailed profile of their previous talks, so adding older conferences is encouraged. # 31st August 2010, 7:41 pm

LWPx::ParanoidAgent. Every programming language needs an equivalent of this library—a robust, secure way to make HTTP requests against URLs from untrusted sources without risk of tarpits, internal network access, socket starvation, weird server errors, or other nastiness. # 31st August 2010, 2:30 am

If you are not paying for it, you’re not the customer; you’re the product being sold.

blue_beetle on MetaFilter # 27th August 2010, 12:58 pm

What is the largest production deployment of Server Side JavaScript?

I believe Flickr used to use Rhino for scripting the image processing (resizing, thumbnailing, sharpening) that was applied to every single uploaded photo. No idea if that’s still the case though.

[... 47 words]

How do Solr, Lucene, Sphinx and Searchify compare?

Lucene is a Java library for creating and searching through a full text index. If you want to make use of it, you’ll need to write your own Java code that integrates with it.

[... 109 words]

Does SQLAlchemy depend on MySQLdb?

No. SQLAlchemy can talk to all sorts of different DB-API compliant backends, including MySQL Connector/J (Jython only), MySQL Connector/Python, mysql-python (the MySQLdb module) and OurSQL—plus backends for many other databases. See the full list here:

[... 50 words]

Django (web framework): Why did stop using Drupal?

They wrote about their reasons in detail in a post to the Django sub-reddit a while ago:

[... 165 words]

What is the best way to learn about setting up server software for Python based web apps?

I’m a big fan of Fabric for automated deployment scripts. Start by reading this tutorial:

[... 40 words]

In what circumstances should one use “magic quotes” in PHP?

Absolutely never. Magic quotes was a badly designed feature, and PHP has been trying to escape its legacy for years. If you are constructing SQL strings using string concatenation you’re asking for trouble—use prepared statements or a library that interpolates and correctly escapes variables for you.

[... 65 words]

Why do some people disable JavaScript in their browser?

For security reasons.

[... 159 words]

Which major companies are using Solr for search?

The Guardian newspaper uses Solr for its Open Platform Content API.

[... 27 words]

Which Solr app for Django is better: Haystack or django-solr-search (solango)?

I’d go with Haystack—while it supports multiple backends, I get the feeling Solr is the principle backend it was developed for. It’s extremely well documented in my opinion, and the SearchQuerySet API it gives you makes running low-level queries really easy if the higher level class-based view it provides don’t do quite what you want.

[... 109 words]

What is the largest production deployment of CouchDB for online use?

The BBC have a pretty big CouchDB cluster, which they use mostly as a replicated key-value store. It’s used by their new identity platform which includes customisation features for iPlayer.

[... 47 words]

What is the highest traffic website built on top of Django?

My best guess would be Disqus. Instagram are pretty enormous these days as well.

[... 31 words]

What is the history of the Django web framework? Why has it been described as “developed in a newsroom”?

I was there!

[... 670 words]

What is the history of Django? I’ve been playing with Quora—it’s a really neat twist on the question-and-answer format, which makes great use of friends, followers and topics and has some very neat live update stuff going on (using Comet on top of Tornado). I just posted quite a long answer to a question about the history of Django. # 24th August 2010, 5:31 pm

Readme Driven Development (via) Tom Preston-Werner advocates for writing the readme before any other code. “Until you’ve written about your software, you have no idea what you’ll be coding.” # 23rd August 2010, 8:20 pm

Using Freebase Gridworks to Create Linked Data. A very handy tutorial from’s Jeni Tennison. # 23rd August 2010, 8:11 pm

PNGStore—Embedding compressed CSS & JavaScript in PNGs. Cal did some further analysis on the CSS/JS to PNG compression trick (including producing some interesting images of jQuery compressed using different image packing techniques) and found it to be slightly less effective than regular GZipping. # 23rd August 2010, 9:47 am

10K Apart Contest: Cheating by Compressing Your JavaScript and CSS to PNG Images. Fascinating hack: transform your JS and CSS in to coloured pixels, save the result as a PNG to benefit from PNG’s built in compression algorithms, then read the data back out of the PNG and convert it back to text using JavaScript and canvas—all to reduce the on-disk filesize when entering the 10K app competition. Alex’s GithubFinder entry is worth checking out too. # 23rd August 2010, 9:45 am

A little deeper investigation showed that nothing I had posted on Buzz had gone public since August 6. Nothing. [...] No one noticed. Not even me. It makes me feel like everything I’ve posted over the past four years on Twitter, Jaiku, Friendfeed, Plurk, Pownce, and, yes, Google Buzz, has been an immense waste of time. I was shouting into a vast echo chamber where no one could hear me because they were too busy shouting themselves.

Leo Laporte # 22nd August 2010, 6:43 pm

Undelete! How to undelete a file accidentally removed using rm on Linux, by grepping through the raw bytes on the hard drive searching for a unique string that was contained in the file. “grep -a -B 25 -A 100 ’some string in the file’ /dev/sda1 > results.txt” # 21st August 2010, 10:56 am

Polymaps. Absurdly classy: “a JavaScript library for image- and vector-tiled maps using SVG”. It can pull in image tiles from sources such as OpenStreetMap, then overlay SVG paths specified using GeoJSON. The demos make use of GeoJSON tiles for US states and counties hosted on AppEngine. The library is developed by Stamen and SimpleGeo, and released under a BSD license. SVG support in the browser is required. # 20th August 2010, 6:46 pm

A More Royal Royal Opera House. Beautiful piece of work updating the branding for the Royal Opera House, including a strikingly modern take on the original crest. # 20th August 2010, 12:08 pm

Surfin’ Safari: Announcing... MathML! MathML is now supported by the WebKit nightlies. Worth checking out for the typographical discussion that’s broken out in the comments. # 18th August 2010, 1:49 pm

Pictos. Here’s something new: a for-sale font containing a set of beautiful royalty-free icons (like Wingdings, but good) designed to be embedded in web applications using @font-face. Small file sizes, scalable vectors without SVG. Not sure about the accessibility implications though. # 17th August 2010, 8:54 pm

Yahoo! Developer Network: Important API Updates and Changes. Some important (and potentially worrying) news about Yahoo! APIs. The BOSS (Build your Own Search Service) API will no longer be free—not an enormous surprise, and hopefully the pricing will be sensible. Most of the other search APIs (including web, news and image search) are being turned off with no replacement, while term extraction and spelling suggestions will be YQL-only. Most worrying, changes to Geo, Maps and Local APIs will be announced in September, with some set to close. I really hope this doesn’t affect the GeoPlanet APIs. # 17th August 2010, 6:14 pm

Human pylons carry electricity across Iceland. An entry in the “Icelandic High-Voltage Electrical Pylon International Design Competition” proposes giant human-shaped electricity pylons. “The figures can be placed into different poses, with the suggestion that the landscapes could inform the position that the sculpture is placed into. For example, as a power line ascends a hill, the pylons could look as if they’re climbing. The figures could also stretch up to gain increased height over longer spans.” # 17th August 2010, 1:38 pm

Writing Bulletproof Apps with API Errorpoints. This is a very good idea: Web APIs should offer special API endpoints for simulating each of the possible errors that might be returned by the production API. # 16th August 2010, 7:12 pm