Simon Willison’s Weblog

Xapian performance comparision with Whoosh. Whoosh appears to be around four times slower than Xapian for indexing and empty cache searches, but Xapian with a full cache blows Whoosh out of the water (5408 searches/second compared to 26.3). Considering how fast Xapian is, that’s still a pretty impressive result for the pure-Python Whoosh.

# 14th February 2009, 1:15 pm / whoosh, xapian, search, fulltextsearch, python, richardboulton

Whoosh. A brand new, pure-python full text indexing engine (think Lucene). Claims to offer performance in the same league as wrappers to C or Java libraries. If this works as well as it claims it will be an excellent tool for adding search to projects that wish to avoid a dependency on an external engine.

# 12th February 2009, 12:49 pm / search, fulltextindexing, python, whoosh, lucene, open-source

Introduction to Information Retrieval (via) This looks excellent—a modern guide to implementing search engines written by some of the engineers behind Yahoo! Search. The full text is available online, but it looks like it’s well worth investing in the dead tree edition.

# 9th February 2009, 8:54 pm / search, yahoosearch, books, freebooks

Announcing the Article Search API. The most interesting API from the NYTimes yet—search against 2.8 million articles from 1981 until today using 35 searchable fields and get back detailed metadata as well as the first paragraph of the articles themselves.

# 5th February 2009, 11:06 pm / new-york-times, apis, newspapers, search

solango. Another attempt at a Django/Solr integration library, based on code written for “a top 20 newspaper site” (I’d love to know which one). This is well documented, uses a registration model clearly inspired by the Django admin which keeps search related metadata out of your regular models and includes management commands for re-indexing and generating Solr schema.xml files.

# 4th February 2009, 12:22 pm / solr, django, python, lucene, search

All you ever wanted to know about writing bloom filters. This helped me understand a key use case for bloom filters: reducing the impact of the “worst case search is when there are no matching results so everything gets scanned” problem.

# 30th January 2009, 8:26 am / bloom-filters, search, jonathan-ellis

How-to: Full-text search in Google App Engine. Use search.SearchableModel instead of db.Model—it’s pretty rough at the moment which is probably why it’s still undocumented.

# 27th June 2008, 8:25 am / googleappengine, appengine, fulltextsearch, python, search

Google AJAX Search API: Flash and Server Side Access. Over a year after Google shot down their SOAP Search API, they’ve quietly released a JSON based one under the guise of supporting “Flash and other non JavaScript environments”. Comes with the strange requirement that an HTTP referer be sent with every request; the API key is optional.

# 22nd April 2008, 7:16 pm / google, soap, ajax, json, search, web-services, apis

In-Depth django-sphinx Tutorial. Another neat Django extension from the guys at Curse: easy integration with the sphinx full text search engine.

# 5th March 2008, 12:03 am / django, curse, python, sphinx-search, search, david-cramer

pysolr. Python wrapper for Solr, the search web service wrapper for Lucene. One thing I’m not clear on: do you need to configure Solr with the fields you’ll be indexing in advance, or can Solr create new fields on the fly to match the data you send it?

# 9th January 2008, 8:50 pm / solr, search, apache, python, lucene, pysolr

Opera 9.5 alpha, Kestrel, released. “With history search, Opera creates a full-text index of each and every page you visit, and when you go to the address bar, you can simply start entering words you know have been on pages you’ve visited before, and items matching your search show up.” I just tried this; it’s magic. I’m switching back to Opera from Camino.

# 16th September 2007, 8:34 pm / opera, camino, browsers, history, search, fulltext, kestrel

django-sphinx (via) More code from Curse Gaming; this time a really nice API for adding Sphinx full-text search to a Django model.

# 9th September 2007, 12:35 am / django, python, orm, search, sphinx-search, fulltext, cursegaming, david-cramer

Grub. Jimmy Wales just announced at OSCON that Wikia have acquired Grub from LookSmart, and will be releasing it as open source.

# 27th July 2007, 5:24 pm / search, oscon07, looksmart, open-source, oscon, jimmywales, grub, wikia

Apache Solr 1.1. Solr is the search Web Service built on top of Lucene. The latest release introduces JSON, Python and Ruby response formats in addition to XML.

# 13th January 2007, 1:16 am / json, python, ruby, xml, webservice, search, lucene, solr

Implementing Full Text Indexing with PostgreSQL (via) TSearch2 tutorial.

# 13th August 2004, 3:44 pm / postgresql, search