Items tagged projects
Filters: projects × Sorted by date
Datasette: instantly create and publish an API for your SQLite databases
I just shipped the first public version of datasette, a new tool for creating and publishing JSON APIs for SQLite databases.
[... 968 words]simonw/csvs-to-sqlite. I built a simple tool for bulk converting multiple CSV files into a SQLite database. # 13th November 2017, 6:49 am
Implementing faceted search with Django and PostgreSQL
I’ve added a faceted search engine to this blog, powered by PostgreSQL. It supports regular text search (proper search, not just SQL“like” queries), filter by tag, filter by date, filter by content type (entries vs blogmarks vs quotation) and any combination of the above. Some example searches:
[... 3103 words]Find conferences to speak at with Lanyrd. We just launched calls for participation on Lanyrd. You can list calls for any conference, browse them by topic, and subscribe to an Atom feed of calls for your area of interest. # 24th November 2010, 2:38 am
Welcome to Lanyrd | The Lanyrd Blog. We’ve started a blog for Lanyrd, our social conference directory project. We’re off to a great start: “Lanyrd is now listing 1,508 conferences and 5,167 individual speaker profiles. 5,637 people have signed in to the site and made 13,293 edits to our data.” # 11th September 2010, 9:32 pm
Lanyrd—the social conference directory. Nat and my new project, launched today and doing pretty well despite some early server hiccups. Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events. We’re particularly keen on helping people build up a detailed profile of their previous talks, so adding older conferences is encouraged. # 31st August 2010, 7:41 pm
getlatlon.com commit dae961a... I’ve finally added an OpenStreetMap tab to getlatlon.com—here’s the diff, it turns out adding a custom OpenStreetMap layer to an existing Google Maps application only takes a few lines of boilerplate code. # 10th July 2010, 12:22 pm
webhook-relay. Another of my experiments with Node.js: webhook-relay is a self-contained queue and webhook request sending agent. Your application can POST to it specifying a webhook alert to be sent off, and webhook-relay will place that request in an in-memory queue and send it on its own time, avoiding the need for your main application server to block until the outgoing request has been processed. # 19th March 2010, 10:17 am
dogproxy. Another of my experiments with Node.js—this is a very simple HTTP proxy which addresses the dog pile effect (also known as the thundering herd) by watching out for multiple requests for a URL that is currently “in flight” and bundling them together. # 3rd February 2010, 1:05 pm
They Write For You. I helped put together this visualisation of stories written by MPs for various newspapers at last Friday’s ’Hackers and Hacks" hack day. # 2nd February 2010, 9:27 am
World Government Data. Launched last week, this is the Guardian’s meta-search engine for searching and browsing through data from four different government data sites (with more sites planned). Under the hood it’s Django, Solr, Haystack and the Scrapy crawling library. The application was built by Ben Firshman during an internship over Christmas. # 27th January 2010, 12:27 pm
Applications: the real stars of the data.gov.uk launch. A write-up of the data.gov.uk launch event at the Guardian. I demonstrated the Guardian’s World Government Data search engine and a small data.gov.uk inspired feature on WildlifeNearYou. # 27th January 2010, 12:23 pm
Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page. # 25th January 2010, 12:36 am
Owls, Otters, Monkeys and Lions Near You.com. It’s not just Owls—we also registered ottersnearyou.com, monkeysnearyou.com and lionsnearyou.com. We’ll probably stop there though, or this could turn in to a very expensive marketing gimmick. # 19th January 2010, 2:54 pm
owlsnearyou.com. Nat and I built this over the weekend. It asks for your location, then tells you where your nearest Owl is (using sightings data people have entered on WildlifeNearYou.com). If you’re using Firefox 3.6 or an iPhone it grabs your location using the W3C geolocation API so you don’t have to type anything at all. # 19th January 2010, 2:45 pm
WildlifeNearYou: Help identify animals in other people’s photos. The first of a number of crowdsourcing-style features we have planned for WildlifeNearYou—users can now help identify the animals in each other’s photos, and photo owners get a simple queue interface to approve or reject the suggestions. # 15th January 2010, 1:35 am
WildlifeNearYou: It began on a fort...
Back in October 2008, myself and 11 others set out on the first /dev/fort expedition. The idea was simple: gather a dozen geeks, rent a fort, take food and laptops and see what we could build in a week.
[... 558 words]countdown_to_newyear.py. A quick Python / OS X script I knocked up last night to count in the new year (using the OS X “say” command). # 1st January 2010, 4:24 pm
Crowdsourced document analysis and MP expenses
As you may have heard, the UK government released a fresh batch of MP expenses documents a week ago on Thursday. I spent that week working with a small team at Guardian HQ to prepare for the release. Here’s what we built:
[... 2081 words]Django ponies: Proposals for Django 1.2
I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.
[... 1674 words]Investigate your MP’s expenses. Launched today, this is the project that has been keeping me ultra-busy for the past week—we’re crowdsourcing the analysis of the 700,000+ scanned MP expenses documents released this morning. It’s the Guardian’s first live Django-powered application, and also the first time we’ve hosted something on EC2. # 18th June 2009, 11:16 pm
optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome! # 28th May 2009, 7:38 pm
geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. # 27th May 2009, 10:02 am
djng—a Django powered microframework
djng is nearly two weeks old now, so it’s about time I wrote a bit about the project.
[... 1501 words]djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in. # 4th May 2009, 6:12 pm
rev=canonical bookmarklet and designing shorter URLs
I’ve watched the proliferation of URL shortening services over the past year with a certain amount of dismay. I care about the health of the web and try to ensure that URLs I am responsible will last for as long as possible, and I think it’s very unlikely that all of these new services will still be around in twenty years time. Last month I suggested that the Internet Archive start mirroring redirect databases, and last week I was pleased to hear that Archiveteam, a different organisation, had already started crawling.
[... 920 words]Oscars 2009: the interactive results | guardian.co.uk. My latest project for the Guardian, put together on very short notice. Updates live as the results are announced, and allows Twitter users to vote on their favourite for each category by sending a specially formatted message to @guardianfilm—jQuery and Ajax polling against S3 under the hood. # 23rd February 2009, 2:19 am
Rate limiting with memcached
On Monday, several high profile “celebrity” Twitter accounts started spouting nonsense, the victims of stolen passwords. Wired has the full story—someone ran a dictionary attack against a Twitter staff member, discovered their password and used Twitter’s admin tools to reset the passwords on the accounts they wanted to steal.
[... 910 words]lightningtimer.net. I’m fed up of having to dig out or knock up a timer script every time I manage lightning talks, so I’ve given one a domain name. You can use lightningtimer.net/#90 to set a different start time for the counter. # 12th November 2008, 4:43 pm
Tweetersation. Nat and my latest side project: a JSONP API powered tool to more easily follow conversations between people on Twitter, by combining their tweets in to a single timeline. # 2nd October 2008, 5:08 pm