Simon Willison’s Weblog

Again, at imified, we're using gearman to manage our queue of feeds to fetch. We have a need for periodic re-polling (since this is a feed reader) so we're potentially vulnerable to slow or dead servers. We have a deamon that continually updates the gearman queue with the list of feeds that need to be fetched. Then the worker processes fetch jobs from gearman. Each worker times out after a period. After too many timeouts or non-200 response the system slows down the fetching of that feed, backing off to only checking the feed once a day. Shoot me an email (first name @ last name .com) if you want more details. Sounds like there's some parallels between what you're doing and what we're doing.

Adam kalsey - 15th December 2007 19:20 - #