398 items tagged “projects”
Posts about projects I have worked on.
2010
World Government Data. Launched last week, this is the Guardian’s meta-search engine for searching and browsing through data from four different government data sites (with more sites planned). Under the hood it’s Django, Solr, Haystack and the Scrapy crawling library. The application was built by Ben Firshman during an internship over Christmas.
Applications: the real stars of the data.gov.uk launch. A write-up of the data.gov.uk launch event at the Guardian. I demonstrated the Guardian’s World Government Data search engine and a small data.gov.uk inspired feature on WildlifeNearYou.
Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page.
Owls, Otters, Monkeys and Lions Near You.com. It’s not just Owls—we also registered ottersnearyou.com, monkeysnearyou.com and lionsnearyou.com. We’ll probably stop there though, or this could turn in to a very expensive marketing gimmick.
owlsnearyou.com. Nat and I built this over the weekend. It asks for your location, then tells you where your nearest Owl is (using sightings data people have entered on WildlifeNearYou.com). If you’re using Firefox 3.6 or an iPhone it grabs your location using the W3C geolocation API so you don’t have to type anything at all.
WildlifeNearYou: Help identify animals in other people’s photos. The first of a number of crowdsourcing-style features we have planned for WildlifeNearYou—users can now help identify the animals in each other’s photos, and photo owners get a simple queue interface to approve or reject the suggestions.
WildlifeNearYou: It began on a fort...
Back in October 2008, myself and 11 others set out on the first /dev/fort expedition. The idea was simple: gather a dozen geeks, rent a fort, take food and laptops and see what we could build in a week.
[... 558 words]countdown_to_newyear.py. A quick Python / OS X script I knocked up last night to count in the new year (using the OS X “say” command).
2009
Crowdsourced document analysis and MP expenses
As you may have heard, the UK government released a fresh batch of MP expenses documents a week ago on Thursday. I spent that week working with a small team at Guardian HQ to prepare for the release. Here’s what we built:
[... 2,081 words]Django ponies: Proposals for Django 1.2
I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.
[... 1,674 words]Investigate your MP’s expenses. Launched today, this is the project that has been keeping me ultra-busy for the past week—we’re crowdsourcing the analysis of the 700,000+ scanned MP expenses documents released this morning. It’s the Guardian’s first live Django-powered application, and also the first time we’ve hosted something on EC2.
optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome!
geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services.
djng—a Django powered microframework
djng is nearly two weeks old now, so it’s about time I wrote a bit about the project.
[... 1,501 words]djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in.
rev=canonical bookmarklet and designing shorter URLs
I’ve watched the proliferation of URL shortening services over the past year with a certain amount of dismay. I care about the health of the web and try to ensure that URLs I am responsible will last for as long as possible, and I think it’s very unlikely that all of these new services will still be around in twenty years time. Last month I suggested that the Internet Archive start mirroring redirect databases, and last week I was pleased to hear that Archiveteam, a different organisation, had already started crawling.
[... 920 words]Oscars 2009: the interactive results | guardian.co.uk. My latest project for the Guardian, put together on very short notice. Updates live as the results are announced, and allows Twitter users to vote on their favourite for each category by sending a specially formatted message to @guardianfilm—jQuery and Ajax polling against S3 under the hood.
Rate limiting with memcached
On Monday, several high profile “celebrity” Twitter accounts started spouting nonsense, the victims of stolen passwords. Wired has the full story—someone ran a dictionary attack against a Twitter staff member, discovered their password and used Twitter’s admin tools to reset the passwords on the accounts they wanted to steal.
[... 910 words]2008
lightningtimer.net. I’m fed up of having to dig out or knock up a timer script every time I manage lightning talks, so I’ve given one a domain name. You can use lightningtimer.net/#90 to set a different start time for the counter.
Tweetersation. Nat and my latest side project: a JSONP API powered tool to more easily follow conversations between people on Twitter, by combining their tweets in to a single timeline.
csrf_protect.php. A PHP class for applying CSRF protection to existing PHP applications, using output buffering to rewrite any POST forms on a page. Heavily inspired by Django’s CSRF middleware. Tell me if you spot any bugs!
backup_to_s3.py. I wrote Yet Another S3 backup script today. It’s a thin wrapper about boto that doesn’t do anything particularly impressive, but it fits my brain.
django-html. A small project I’m working on to make Django behave better with regards to HTML v.s. XHTML.
Announcing dmigrations
The team at Global Radio (formerly GCap Media) is the largest group of Django developers I’ve personally worked with, consisting of 14 developers split into two scrum teams, all contributing to the same overall codebase.
[... 625 words]json-tinyurl. Because sometimes you want to be able to create a shorter version of a URL directly from JavaScript without hosting your own server-side proxy.
Get Lat Lon now has a “Get my location (by IP)” button. It took all of five minutes to add using the new google.loader.ClientLocation API. The button is only visible if your location can be resolved.
json-head. I’ve deployed another App Engine mini-app, which provides a JSON-P API for running HEAD requests against an arbitrary URL (useful for checking things like Content-Length and Content-Type headers and whether a URL returns 200). App Engine’s urlfetch limitations mean it can only deal with port 80 and 443 requests.
jsontime. Nat and I threw this together this morning—it runs on Google App Engine and exposes Python’s pytz timezone library over JSONP.
2007
Get Lat Lon. I finally got fed up of hunting around for simple latitude/longitude tools when messing around with mapping APIs, so I built my own with a memorable URL. I plan to add new features as and when I need them.
idproxy.net: Use your Yahoo! account as an OpenID
In an ideal world, some or all of the sites with large user databases (Yahoo!, AOL, Google, Amazon and so on) would act as OpenID providers, allowing their users to sign in to OpenID supporting sites around the Web. Until that happens, people who want to use OpenID need to sign up for Yet Another Account to do so.
[... 414 words]