Simon Willison’s Weblog

51 items tagged “search”

Digg Search: Now With 99.987% Less Suck. Really nice implementation of faceted search, still using Lucene and Solr under the hood. # 10th April 2009, 10:17 pm

Sphinx 0.9.9-rc2 is out. Interesting new feature: the Sphinx search server now supports the MySQL binary protocol, so you can talk to it using a regular MySQL client library and fire off search queries using SELECT syntax and the new SphinxQL query language. # 8th April 2009, 1:59 pm

Guardian + Lucene = Similar Articles + Categorisation. Alf Eaton loaded 13,000 Guardian articles tagged Science in to Solr and Lucene and is using Solr’s MoreLikeThisHandler to find related articles and automatically apply Guardian tags to Nature News articles. # 11th March 2009, 12:53 pm

How search.twitter.com uses Varnish. Includes examples of the configuration options they use. # 2nd March 2009, 5:08 pm

django-springsteen and Distributed Search. Will Larson’s Django search library currently just talks to Yahoo! BOSS, but is designed to be extensible for other external search services. Interestingly, it uses threads to fire off several HTTP requests in parallel from within the Django view. # 25th February 2009, 10:28 pm

Xapian performance comparision with Whoosh. Whoosh appears to be around four times slower than Xapian for indexing and empty cache searches, but Xapian with a full cache blows Whoosh out of the water (5408 searches/second compared to 26.3). Considering how fast Xapian is, that’s still a pretty impressive result for the pure-Python Whoosh. # 14th February 2009, 1:15 pm

Whoosh. A brand new, pure-python full text indexing engine (think Lucene). Claims to offer performance in the same league as wrappers to C or Java libraries. If this works as well as it claims it will be an excellent tool for adding search to projects that wish to avoid a dependency on an external engine. # 12th February 2009, 12:49 pm

Introduction to Information Retrieval (via) This looks excellent—a modern guide to implementing search engines written by some of the engineers behind Yahoo! Search. The full text is available online, but it looks like it’s well worth investing in the dead tree edition. # 9th February 2009, 8:54 pm

Announcing the Article Search API. The most interesting API from the NYTimes yet—search against 2.8 million articles from 1981 until today using 35 searchable fields and get back detailed metadata as well as the first paragraph of the articles themselves. # 5th February 2009, 11:06 pm

solango. Another attempt at a Django/Solr integration library, based on code written for “a top 20 newspaper site” (I’d love to know which one). This is well documented, uses a registration model clearly inspired by the Django admin which keeps search related metadata out of your regular models and includes management commands for re-indexing and generating Solr schema.xml files. # 4th February 2009, 12:22 pm

All you ever wanted to know about writing bloom filters. This helped me understand a key use case for bloom filters: reducing the impact of the “worst case search is when there are no matching results so everything gets scanned” problem. # 30th January 2009, 8:26 am

How-to: Full-text search in Google App Engine. Use search.SearchableModel instead of db.Model—it’s pretty rough at the moment which is probably why it’s still undocumented. # 27th June 2008, 8:25 am

Google AJAX Search API: Flash and Server Side Access. Over a year after Google shot down their SOAP Search API, they’ve quietly released a JSON based one under the guise of supporting “Flash and other non JavaScript environments”. Comes with the strange requirement that an HTTP referer be sent with every request; the API key is optional. # 22nd April 2008, 7:16 pm

In-Depth django-sphinx Tutorial. Another neat Django extension from the guys at Curse: easy integration with the sphinx full text search engine. # 5th March 2008, 12:03 am

pysolr. Python wrapper for Solr, the search web service wrapper for Lucene. One thing I’m not clear on: do you need to configure Solr with the fields you’ll be indexing in advance, or can Solr create new fields on the fly to match the data you send it? # 9th January 2008, 8:50 pm

Opera 9.5 alpha, Kestrel, released. “With history search, Opera creates a full-text index of each and every page you visit, and when you go to the address bar, you can simply start entering words you know have been on pages you’ve visited before, and items matching your search show up.” I just tried this; it’s magic. I’m switching back to Opera from Camino. # 16th September 2007, 8:34 pm

django-sphinx (via) More code from Curse Gaming; this time a really nice API for adding Sphinx full-text search to a Django model. # 9th September 2007, 12:35 am

Grub. Jimmy Wales just announced at OSCON that Wikia have acquired Grub from LookSmart, and will be releasing it as open source. # 27th July 2007, 5:24 pm

Apache Solr 1.1. Solr is the search Web Service built on top of Lucene. The latest release introduces JSON, Python and Ruby response formats in addition to XML. # 13th January 2007, 1:16 am

Giving away the index

My final year project is due in two weeks, and I’m going to be running on silent for most of them. I have, however, upgraded to Tiger and playing with Spotlight has given me plenty to think about.

[... 414 words]

Google cruft

New Google feature: Google Movies. Displays aggregated movie reviews (like Rotten Tomatoes), looks up local movie times based on your zip code saved in Google Local (more evidence of the fabled Google cookie), and even handles recommendations.

[... 120 words]