Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows.
Recent articles
- Weeknotes: datasette-enrichments, datasette-comments, sqlite-chronicle - 8th December 2023
- Datasette Enrichments: a new plugin framework for augmenting your data - 1st December 2023
- llamafile is the new best way to run a LLM on your own computer - 29th November 2023
- Prompt injection explained, November 2023 edition - 27th November 2023
- I'm on the Newsroom Robots podcast, with thoughts on the OpenAI board - 25th November 2023
- Weeknotes: DevDay, GitHub Universe, OpenAI chaos - 22nd November 2023