Beanstalkd. This is the light-weight cross-language queue I’ve been waiting for. Similar to Starling but your workers don’t need to poll for new jobs; you can call the blocking “reserve” call instead.
Sign in with 
OpenID
Beanstalkd. This is the light-weight cross-language queue I’ve been waiting for. Similar to Starling but your workers don’t need to poll for new jobs; you can call the blocking “reserve” call instead.
So, I was wondering, how does beanstalkd compare with Gearman?
FWIW, the first version of Starling had blocking reads, but it made distribution across many queue servers much more difficult (i.e., you need to do a blocking read across all of them).
Blaine Cook - 12th May 2008 15:34 - #
I'm currently using a homebrew solution called
MemcacheQueue.It's main advantage is it's decentralization - since it's mainly used under Django, I've just built it around Django's low level cache framework, and I can push and pull jobs from one of the ten
memcachedservers we're currently running. The key's value is simply a Python list, and I have the clients pop it's first item. Works outstandingly well.I'm not sure if beanstalkd is quite what you're looking for either. Unless it's changed in recent releases, it didn't persist the queue. So restarts meant you lost all your messages. Fine if it's easy to regenerate the job queue, but painful when you can't.
Persistence is next on my to-do list. It'll happen right after beanstalkd 1.0 is ready.
Doing blocking reads across more than one queue server isn't terribly hard -- only the client library needs to be concerned with details of how it happens. To my knowledge none of the existing beanstalk client libraries do this yet, but it'll happen sooner or later.
Keith Rarick - 13th May 2008 05:25 - #
For my purposes persistence isn't hugely important - I plan to use beanstalkd for interactive tasks (e.g. someone requests that my application go away and retrieve an Atom feed) - the client will be polling for completion via Ajax, so I can include logic that times out after 30 seconds or so and assumes something went wrong with the queue.
Simon, you may also be interested in the python client (if you haven't already found it).
http://code.google.com/p/pybeanstalk/
Tim Yardley - 13th May 2008 20:57 - #