cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity.
Sign in with 
OpenID
Not to dampen your enthusiasm, but have you used http://gearman.org/ ? Can you make any comparisons?
Jeremy Dunck - 22nd September 2009 03:01 - #
I tried to get it running it a few years ago and failed, haven't looked again again (and I've heard it's changed tremendously since then). Oddly enough, CrowdCloud feels to me a bit like a Brad Fitzpatrick design - that's why I'm so excited about it, it's simple, elegant and I instantly understood almost everything about how it worked. I've been complaining about how hard I find it to run workers, dish out jobs to them and keep them up and running for ages. CrowdCloud seems to solve that problem for me (or it will once they ship an init script for the node server).
Gearman's docs have gotten much better recently. But I'm definitely having a look at DocCloud, too.
Jeremy Dunck - 22nd September 2009 16:30 - #
We've been using Gearman for a few months without much complaint.
I do like that CrowdCloud includes the dashboard/statistics -- I started adding that to our Gearman implementation but haven't had the time to finish it.