Feed Sign in with OpenID OpenID

Simon Willison’s Weblog

Techniques for safely consuming external HTTP on demand? I asked this question on programming.reddit.com yesterday and got some really insightful answers, including Joe Stump from Digg describing how Digg Images uses Danga’s Gearman worker queue.

Tagged , , , , , , , , ,

1 comment

  1. Again, at imified, we're using gearman to manage our queue of feeds to fetch. We have a need for periodic re-polling (since this is a feed reader) so we're potentially vulnerable to slow or dead servers. We have a deamon that continually updates the gearman queue with the list of feeds that need to be fetched. Then the worker processes fetch jobs from gearman. Each worker times out after a period. After too many timeouts or non-200 response the system slows down the fetching of that feed, backing off to only checking the feed once a day. Shoot me an email (first name @ last name .com) if you want more details. Sounds like there's some parallels between what you're doing and what we're doing.

    Adam kalsey - 15th December 2007 19:20 - #

Sign in with OpenID

Auto-HTML: Line breaks are preserved; URLs will be converted in to links.

Manual XHTML: Enter your own, valid XHTML. Allowed tags are a, p, blockquote, ul, ol, li, dl, dt, dd, em, strong, dfn, code, q, samp, kbd, var, cite, abbr, acronym, sub, sup, br, pre

A django site