<?xml version="1.0" encoding="utf-8"?>
<feed xml:lang="en-us" xmlns="http://www.w3.org/2005/Atom"><title>Simon Willison's Weblog: redis</title><link href="http://simonwillison.net/" rel="alternate"/><link href="http://simonwillison.net/tags/redis.atom" rel="self"/><id>http://simonwillison.net/</id><updated>2025-12-23T23:03:00+00:00</updated><author><name>Simon Willison</name></author><entry><title>Quoting Salvatore Sanfilippo</title><link href="https://simonwillison.net/2025/Dec/23/salvatore-sanfilippo/#atom-tag" rel="alternate"/><published>2025-12-23T23:03:00+00:00</published><updated>2025-12-23T23:03:00+00:00</updated><id>https://simonwillison.net/2025/Dec/23/salvatore-sanfilippo/#atom-tag</id><summary type="html">
    &lt;blockquote cite="https://news.ycombinator.com/item?id=46367224#46368706"&gt;&lt;p&gt;If this [MicroQuickJS] had been available in 2010, Redis scripting would have been JavaScript and not Lua. Lua was chosen based on the implementation requirements, not on the language ones... (small, fast, ANSI-C). I appreciate certain ideas in Lua, and people love it, but I was never able to &lt;em&gt;like&lt;/em&gt; Lua, because it departs from a more Algol-like syntax and semantics without good reasons, for my taste. This creates friction for newcomers. I love friction when it opens new useful ideas and abstractions that are worth it, if you learn SmallTalk or FORTH and for some time you are lost, it's part of how the languages are different. But I think for Lua this is not true enough: it feels like it departs from what people know without good reasons.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="https://news.ycombinator.com/item?id=46367224#46368706"&gt;Salvatore Sanfilippo&lt;/a&gt;, Hacker News comment on MicroQuickJS&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/javascript"&gt;javascript&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/lua"&gt;lua&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;&lt;/p&gt;



</summary><category term="javascript"/><category term="lua"/><category term="redis"/><category term="salvatore-sanfilippo"/></entry><entry><title>Scaling HNSWs</title><link href="https://simonwillison.net/2025/Nov/11/scaling-hnsws/#atom-tag" rel="alternate"/><published>2025-11-11T23:38:39+00:00</published><updated>2025-11-11T23:38:39+00:00</updated><id>https://simonwillison.net/2025/Nov/11/scaling-hnsws/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://antirez.com/news/156"&gt;Scaling HNSWs&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Salvatore Sanfilippo spent much of this year working on &lt;a href="https://github.com/redis/redis/blob/8.2.3/modules/vector-sets/README.md"&gt;vector sets for Redis&lt;/a&gt;, which first shipped in &lt;a href="https://redis.io/blog/redis-8-ga/"&gt;Redis 8 in May&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;A big part of that work involved implementing HNSW - Hierarchical Navigable Small World - an indexing technique first introduced in &lt;a href="https://arxiv.org/abs/1603.09320"&gt;this 2016 paper&lt;/a&gt; by Yu. A. Malkov and D. A. Yashunin.&lt;/p&gt;
&lt;p&gt;Salvatore's detailed notes on the Redis implementation here offer an immersive trip through a fascinating modern field of computer science. He describes several new contributions he's made to the HNSW algorithm, mainly around efficient deletion and updating of existing indexes.&lt;/p&gt;
&lt;p&gt;Since embedding vectors are notoriously memory-hungry I particularly appreciated this note about how you can scale a large HNSW vector set across many different nodes and run parallel queries against them for both reads and writes:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;[...] if you have different vectors about the same use case split in different instances / keys, you can ask VSIM for the same query vector into all the instances, and add the WITHSCORES option (that returns the cosine distance) and merge the results client-side, and you have magically scaled your hundred of millions of vectors into multiple instances, splitting your dataset N times [One interesting thing about such a use case is that you can query the N instances in parallel using multiplexing, if your client library is smart enough].&lt;/p&gt;
&lt;p&gt;Another very notable thing about HNSWs exposed in this raw way, is that you can finally scale writes very easily. Just hash your element modulo N, and target the resulting Redis key/instance. Multiple instances can absorb the (slow, but still fast for HNSW standards) writes at the same time, parallelizing an otherwise very slow process.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;It's always exciting to see new implementations of fundamental algorithms and data structures like this make it into Redis because Salvatore's C code is so clearly commented and pleasant to read - here's &lt;a href="https://github.com/redis/redis/blob/8.2.3/modules/vector-sets/hnsw.c"&gt;vector-sets/hnsw.c&lt;/a&gt; and &lt;a href="https://github.com/redis/redis/blob/8.2.3/modules/vector-sets/vset.c"&gt;vector-sets/vset.c&lt;/a&gt;.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=45887466"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/algorithms"&gt;algorithms&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/c"&gt;c&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/computer-science"&gt;computer-science&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/data-structures"&gt;data-structures&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/vector-search"&gt;vector-search&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/embeddings"&gt;embeddings&lt;/a&gt;&lt;/p&gt;



</summary><category term="algorithms"/><category term="c"/><category term="computer-science"/><category term="data-structures"/><category term="redis"/><category term="salvatore-sanfilippo"/><category term="vector-search"/><category term="embeddings"/></entry><entry><title>tidwall/pogocache</title><link href="https://simonwillison.net/2025/Jul/21/pogocache/#atom-tag" rel="alternate"/><published>2025-07-21T23:58:53+00:00</published><updated>2025-07-21T23:58:53+00:00</updated><id>https://simonwillison.net/2025/Jul/21/pogocache/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/tidwall/pogocache"&gt;tidwall/pogocache&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
New project from Josh Baker, author of the excellent &lt;code&gt;tg&lt;/code&gt; C geospatial libarry (&lt;a href="https://simonwillison.net/2023/Sep/23/tg-polygon-indexing/"&gt;covered previously&lt;/a&gt;) and various other &lt;a href="https://github.com/tidwall"&gt;interesting projects&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Pogocache is fast caching software built from scratch with a focus on low latency and cpu efficency.&lt;/p&gt;
&lt;p&gt;Faster: Pogocache is faster than Memcache, Valkey, Redis, Dragonfly, and Garnet. It has the lowest latency per request, providing the quickest response times. It's optimized to scale from one to many cores, giving you the best single-threaded and multithreaded performance.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Faster than Memcache and Redis is a big claim! The README includes a &lt;a href="https://github.com/tidwall/pogocache/blob/main/README.md#design-details"&gt;design details&lt;/a&gt; section that explains how the system achieves that performance, using a sharded hashmap inspired by Josh's &lt;a href="https://github.com/tidwall/shardmap"&gt;shardmap&lt;/a&gt; project and clever application of threads.&lt;/p&gt;
&lt;p&gt;Performance aside, the most interesting thing about Pogocache is the server interface it provides: it emulates the APIs for Redis and Memcached, provides a simple HTTP API &lt;em&gt;and&lt;/em&gt; lets you talk to it over the PostgreSQL wire protocol as well!&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;psql -h localhost -p 9401
=&amp;gt; SET first Tom;
=&amp;gt; SET last Anderson;
=&amp;gt; SET age 37;

$ curl http://localhost:9401/last
Anderson
&lt;/code&gt;&lt;/pre&gt;

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=44638076"&gt;Show HN&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/c"&gt;c&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/caching"&gt;caching&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/http"&gt;http&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/memcached"&gt;memcached&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="c"/><category term="caching"/><category term="http"/><category term="memcached"/><category term="postgresql"/><category term="redis"/></entry><entry><title>Redis is open source again</title><link href="https://simonwillison.net/2025/May/1/redis-is-open-source-again/#atom-tag" rel="alternate"/><published>2025-05-01T17:19:36+00:00</published><updated>2025-05-01T17:19:36+00:00</updated><id>https://simonwillison.net/2025/May/1/redis-is-open-source-again/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://antirez.com/news/151"&gt;Redis is open source again&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Salvatore Sanfilippo:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Five months ago, I rejoined Redis and quickly started to talk with my colleagues about a possible switch to the AGPL license, only to discover that there was already an ongoing discussion, a very old one, too. [...]&lt;/p&gt;
&lt;p&gt;I’ll be honest: I truly wanted the code I wrote for the new Vector Sets data type to be released under an open source license. [...]&lt;/p&gt;
&lt;p&gt;So, honestly, while I can’t take credit for the license switch, I hope I contributed a little bit to it, because today I’m happy. I’m happy that Redis is open source software again, under the terms of the AGPLv3 license.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I'm absolutely &lt;em&gt;thrilled&lt;/em&gt; to hear this. Redis 8.0 is &lt;a href="https://redis.io/blog/redis-8-ga/"&gt;out today under the new license&lt;/a&gt;, including a beta release of &lt;a href="https://redis.io/docs/latest/develop/data-types/vector-sets/"&gt;Vector Sets&lt;/a&gt;. I've been watching Salvatore's work on those with &lt;a href="https://antirez.com/news/149"&gt;fascination&lt;/a&gt;, while sad that I probably wouldn't use it often due to the janky license. That concern is now gone. I'm looking forward to putting them through their paces!&lt;/p&gt;
&lt;p&gt;See also &lt;a href="https://redis.io/blog/agplv3/"&gt;Redis is now available under the AGPLv3 open source license&lt;/a&gt; on the Redis blog. An interesting note from that is that they are also:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Integrating Redis Stack technologies, including JSON, Time Series, probabilistic data types, Redis Query Engine and more into core Redis 8 under AGPL&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;That's a whole bunch of new things that weren't previously part of Redis core.&lt;/p&gt;
&lt;p&gt;I hadn't encountered &lt;a href="https://redis.io/docs/latest/develop/interact/search-and-query/"&gt;Redis Query Engine&lt;/a&gt; before - it looks like that's a whole set of features that turn Redis into more of an Elasticsearch-style document database complete with full-text, vector search operations and geospatial operations and aggregations. It supports search syntax that looks a bit like this:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;FT.SEARCH places "museum @city:(san francisco|oakland) @shape:[CONTAINS $poly]" PARAMS 2 poly 'POLYGON((-122.5 37.7, -122.5 37.8, -122.4 37.8, -122.4 37.7, -122.5 37.7))' DIALECT 3&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;(Noteworthy that Elasticsearch chose the AGPL too when they switched back from the SSPL to an open source license &lt;a href="https://simonwillison.net/2024/Aug/29/elasticsearch-is-open-source-again/"&gt;last year&lt;/a&gt;).

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=43859446"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/open-source"&gt;open-source&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/vector-search"&gt;vector-search&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/janky-licenses"&gt;janky-licenses&lt;/a&gt;&lt;/p&gt;



</summary><category term="open-source"/><category term="redis"/><category term="salvatore-sanfilippo"/><category term="vector-search"/><category term="janky-licenses"/></entry><entry><title>From where I left</title><link href="https://simonwillison.net/2024/Dec/10/from-where-i-left/#atom-tag" rel="alternate"/><published>2024-12-10T18:56:26+00:00</published><updated>2024-12-10T18:56:26+00:00</updated><id>https://simonwillison.net/2024/Dec/10/from-where-i-left/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://antirez.com/news/144"&gt;From where I left&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Four and a half years after he left the project, Redis creator Salvatore Sanfilippo is returning to work on Redis.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Hacking randomly was cool but, in the long run, my feeling was that I was lacking a real purpose, and every day I started to feel a bigger urgency to be part of the tech world again. At the same time, I saw the Redis community fragmenting, something that was a bit concerning to me, even as an outsider.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I'm personally still upset at the license change, but Salvatore sees it as necessary to support the commercial business model for Redis Labs. It feels to me like a betrayal of the volunteer efforts by previous contributors. I &lt;a href="https://news.ycombinator.com/item?id=42378488#42379400"&gt;posted about that&lt;/a&gt; on Hacker News and Salvatore replied:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;I can understand that, but the thing about the BSD license is that such value never gets lost. People are able to fork, and after a fork for the original project to still lead will be require to put something more on the table.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Salvatore's first new project is an exploration of adding vector sets to Redis. The vector similarity API he previews in this post reminds me of why I fell in love with Redis in the first place - it's clean, simple and feels obviously right to me.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;VSIM top_1000_movies_imdb ELE "The Matrix"  WITHSCORES
1) "The Matrix"
2) "0.9999999403953552"
3) "Ex Machina"
4) "0.8680362105369568"
...
&lt;/code&gt;&lt;/pre&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/open-source"&gt;open-source&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/vector-search"&gt;vector-search&lt;/a&gt;&lt;/p&gt;



</summary><category term="open-source"/><category term="redis"/><category term="salvatore-sanfilippo"/><category term="vector-search"/></entry><entry><title>Themes from DjangoCon US 2024</title><link href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2024/#atom-tag" rel="alternate"/><published>2024-09-27T23:36:02+00:00</published><updated>2024-09-27T23:36:02+00:00</updated><id>https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2024/#atom-tag</id><summary type="html">
    &lt;p&gt;I just arrived home from a trip to Durham, North Carolina for DjangoCon US 2024. I’ve already written &lt;a href="https://simonwillison.net/2024/Sep/25/djp-a-plugin-system-for-django/"&gt;about my talk where I announced a new plugin system for Django&lt;/a&gt;; here are my notes on some of the other themes that resonated with me during the conference.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#growing-the-django-software-foundation-dsf-"&gt;Growing the Django Software Foundation (DSF)&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#could-we-fund-a-django-lts-accessibility-audit-"&gt;Could we fund a Django LTS accessibility audit?&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#django-fellows-continue-to-provide-outstanding-value"&gt;Django fellows continue to provide outstanding value&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#django-needs-feature-champions"&gt;Django needs feature champions&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#htmx-fits-django-really-well"&gt;htmx fits Django really well&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#django-ninja-has-positive-buzz"&gt;Django Ninja has positive buzz&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#valkey-as-a-last-minute-sponsor"&gt;Valkey as a last-minute sponsor&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href="https://simonwillison.net/2024/Sep/27/themes-from-djangocon-us-2014/#durham-has-a-world-class-collection-of-tubas"&gt;Durham has a world-class collection of tubas&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id="growing-the-django-software-foundation-dsf-"&gt;Growing the Django Software Foundation (DSF)&lt;/h4&gt;
&lt;p&gt;Jacob Kaplan-Moss gave &lt;a href="https://2024.djangocon.us/talks/if-we-had-1000000-what-could-the-dsf-do-with-4x-its-budget/"&gt;my favorite talk&lt;/a&gt; of the conference, asking what the Django Software Foundation could do if it quadrupled its annual income from $250,000 to $1 million dollars, and then mapping out a convincing path to get there.&lt;/p&gt;
&lt;p&gt;I really liked this diagram Jacob provided summarizing the foundation’s current income and expenditures. It’s pretty cool that $90,000 of annual income comes from individual donors, over a third of the total since corporate donors provide $160,000.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://static.simonwillison.net/static/2024/dsf-diagram.jpg" alt="Financial breakdown diagram with the following numbers:  PLATINUM &amp;amp; GOLD: $125,000 CORPORATE DONORS: $160,000 BUDGET: $255,000 SILVER &amp;amp; BELOW: $35,000 INDIVIDUAL DONORS: $90,000  Spending:  WAGES (FELLOWS): $200,000 GRANTS: $35,000 OTHER: $5,000 FEES/HOSTING: $10,000 SURPLUS: $10,000​​​​​​​​​​​​​​​​" style="max-width: 100%;" /&gt;&lt;/p&gt;
&lt;p&gt;Top priority would be hiring an Executive Director for the foundation, which is currently lead entirely by an elected, volunteer board. I’ve seen how useful a professional ED is from my own experiences &lt;a href="https://simonwillison.net/2024/Sep/18/board-of-the-python-software-foundation/"&gt;on the Python Software Foundation board&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Having someone working full time on the foundation outside of our current fellows - who have more than enough on their plates already - would enable the foundation to both take on more ambitious goals and also raise more money with which to tackle them.&lt;/p&gt;
&lt;p&gt;A line that Jacob used repeatedly in his talk about funding the foundation was this: if you or your organization &lt;em&gt;wouldn’t&lt;/em&gt; want to sponsor Django, he’d love to know why that is - understanding those blockers right now is almost as valuable as receiving actual cash. You can reach out to him at &lt;code&gt;jacob at djangoproject.com&lt;/code&gt;.&lt;/p&gt;
&lt;h4 id="could-we-fund-a-django-lts-accessibility-audit-"&gt;Could we fund a Django LTS accessibility audit?&lt;/h4&gt;
&lt;p&gt;Django fellows and the &lt;a href="https://github.com/django/deps/blob/main/final/0011-accessibility-team.rst"&gt;Django Accessibility Team&lt;/a&gt; have been focusing significant effort on the accessibility of the Django admin. I found this very inspiring, and in combination with the talk of more funding for the foundation it put an idea in my head: what if every Django LTS release (once every two years) was backed by a full, professional accessibility audit, run by an agency staffed with developers who use screen readers?&lt;/p&gt;
&lt;p&gt;Imagine how much impact it would have if the default Django admin interface had excellent, documented accessibility out of the box. It could improve things for hundreds of thousands of users, and set an excellent precedent for projects (and foundations) in the wider open source community.&lt;/p&gt;
&lt;p&gt;This also feels to me like something that should be inherently attractive to sponsors. A lot of agencies use Django for government work, where accessibility is a requirement with teeth. Would one of those agencies like to be the “accessibility sponsor” for a major Django release?&lt;/p&gt;
&lt;h4 id="django-fellows-continue-to-provide-outstanding-value"&gt;Django fellows continue to provide outstanding value&lt;/h4&gt;
&lt;p&gt;The &lt;a href="https://www.djangoproject.com/fundraising/#fellowship-program"&gt;DSF’s fellowship program&lt;/a&gt; remains one of the most impactful initiatives I’ve seen anywhere for ensuring the ongoing sustainability of a community-driven open source project.&lt;/p&gt;
&lt;p&gt;Both of the current fellows, Natalia Bidart and Sarah Boyce, were in attendance and gave talks. It was great getting to meet them in person.&lt;/p&gt;
&lt;p&gt;If you’re not familiar with the program, the fellows are contractors who are paid by the DSF to keep the Django project ticking over - handling many of the somewhat less glamorous tasks of responsible open source maintenance such as ticket triage, release management, security fixes and code review.&lt;/p&gt;
&lt;p&gt;The fellows program is in its tenth year, and is a key reason that Django continues to release new versions &lt;a href="https://www.djangoproject.com/download/#supported-versions"&gt;on a regular schedule&lt;/a&gt; despite having no single corporate parent with paid developers.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://static.simonwillison.net/static/2024/django-roadmap.png" alt="Software release timeline: 4.2 LTS (April 2023), 5.0 (August 2024), 5.1 (2025), 5.2 LTS (2026), 6.0 (2027), 6.1 (2027), 6.2 LTS (2028), 7.0 (2029). LTS versions have extended support periods." style="max-width: 100%;" /&gt;&lt;/p&gt;
&lt;p&gt;Unsurprisingly there is always more work than fellow capacity, hence Jacob’s desire to further expand the existing program.&lt;/p&gt;
&lt;p&gt;The fellows program launched with a policy that fellows should not work on new feature development. I believe this was partly related to interpretation of IRS nonprofit guidelines which have since been reconsidered, and there is a growing consensus now that this policy should be dropped.&lt;/p&gt;
&lt;h4 id="django-needs-feature-champions"&gt;Django needs feature champions&lt;/h4&gt;
&lt;p&gt;Django has a well deserved reputation for stability, reliability and a dependable release process. It has less of a reputation for constantly turning out ground-breaking new features.&lt;/p&gt;
&lt;p&gt;Long-time Django contributors who I talked to all had a similar position on this: the challenge here is that big new features need dedicated champions to both lead design and development on them and to push them through to completion.&lt;/p&gt;
&lt;p&gt;The pool of community members who are both willing and able to take on these larger projects is currently too small.&lt;/p&gt;
&lt;p&gt;There are a number of ways we could address this - most notably through investing financial resources in sponsoring feature development. This has worked well for Django in the past - Django’s migrations work was funded by &lt;a href="https://www.kickstarter.com/projects/andrewgodwin/schema-migrations-for-django"&gt;a Kickstarter campaign&lt;/a&gt; back in 2013.&lt;/p&gt;
&lt;p&gt;The Django Software Foundation will shortly be announcing details of elections for both the DSF board and the Django Steering Council. These are extremely influential positions for people who want to help solve some of these larger problems.&lt;/p&gt;
&lt;h4 id="htmx-fits-django-really-well"&gt;htmx fits Django really well&lt;/h4&gt;
&lt;p&gt;&lt;a href="https://htmx.org/"&gt;htmx&lt;/a&gt; is an incredibly good fit for the uncodified Django community philosophy of building for the web. It came up in multiple talks. It feels like it may be a solution that the Django community has been seeking for years, as a very compelling alternative to writing everything in SPA JavaScript and using Django purely as a backend via something like Django REST Framework.&lt;/p&gt;
&lt;p&gt;I've been slightly resistant to embracing htmx myself purely because it's such a critical dependency and in the past I wasn't convinced of its staying power. It's now mature, stable and widely-enough used that I'm ready to consider it for my own long-term projects.&lt;/p&gt;
&lt;h4 id="django-ninja-has-positive-buzz"&gt;Django Ninja has positive buzz&lt;/h4&gt;
&lt;p&gt;I haven’t paid much attention to &lt;a href="https://django-ninja.dev/"&gt;Django Ninja&lt;/a&gt; but it had a lot of very positive buzz at the conference as well, as a tool for quickly building full-featured, performative API endpoints (thanks to Rust-backed &lt;a href="https://docs.pydantic.dev/"&gt;Pydantic&lt;/a&gt; for serialization) with &lt;a href="https://django-ninja.dev/#interactive-api-docs"&gt;interactive API docs&lt;/a&gt; powered by OpenAPI.&lt;/p&gt;
&lt;p&gt;I respect Django REST Framework a lot, but my personal programming style leans away from Class Based Views, which it uses quite a bit. Django Ninja looks like it might fit my function-view biases better.&lt;/p&gt;
&lt;p&gt;I wrote about Richard Terry’s excellent &lt;a href="https://github.com/radiac/nanodjango"&gt;nanodjango&lt;/a&gt; single-file Django application tool &lt;a href="https://simonwillison.net/2024/Sep/24/nanodjango/"&gt;the other day&lt;/a&gt; - Django Ninja comes baked into that project as well.&lt;/p&gt;
&lt;h4 id="valkey-as-a-last-minute-sponsor"&gt;Valkey as a last-minute sponsor&lt;/h4&gt;
&lt;p&gt;The three platinum sponsors for DjangoCon this year were &lt;a href="https://www.revsys.com/"&gt;REVSYS&lt;/a&gt;, &lt;a href="https://www.caktusgroup.com/"&gt;Caktus Group&lt;/a&gt; and &lt;a href="https://valkey.io/"&gt;Valkey&lt;/a&gt;. Valkey were a late and somewhat surprising addition to the sponsorship lineup.&lt;/p&gt;
&lt;p&gt;Valkey is the &lt;a href="https://www.linuxfoundation.org/press/linux-foundation-launches-open-source-valkey-community"&gt;Linux Foundation backed&lt;/a&gt; fork of Redis, created in response to Redis &lt;a href="https://redis.io/blog/redis-adopts-dual-source-available-licensing/"&gt;ditching their Open Source license&lt;/a&gt; (which I took quite personally, having contributed my own free effort to promoting and improving Redis in the past).&lt;/p&gt;
&lt;p&gt;Aside from expressing thanks to them, I usually don’t pay sponsors that much attention. For some reason this one hit differently - the fact that Valkey were ready to step in as a major sponsor despite being only a few months old has caused me to take that project a whole lot more seriously than I did before. I’ll certainly consider them next time I come across a Redis-shaped problem.&lt;/p&gt;
&lt;h4 id="durham-has-a-world-class-collection-of-tubas"&gt;Durham has a world-class collection of tubas&lt;/h4&gt;
&lt;p&gt;My favorite category of &lt;a href="https://www.niche-museums.com/"&gt;Niche Museum&lt;/a&gt; is one that's available by appointment only where the person who collected everything is available to show you around.&lt;/p&gt;
&lt;p&gt;I always check &lt;a href="https://www.atlasobscura.com/"&gt;Atlas Obscura&lt;/a&gt; any time I visit a new city, and this time I was delighted to learn about The Vincent and Ethel Simonetti Historic Tuba Collection!&lt;/p&gt;
&lt;p&gt;I promoted it in the DjangoCon US #outings Slack channel and got together a group of five conference attendees for a visit on Thursday, shortly before my flight.&lt;/p&gt;
&lt;p&gt;It was &lt;em&gt;peak&lt;/em&gt; Niche Museum. I’ve posted photos and notes over &lt;a href="https://www.niche-museums.com/112"&gt;on my Niche Museums&lt;/a&gt; website, the first new article there in quite a while.&lt;/p&gt;

&lt;p&gt;&lt;img alt="More than a dozen varied and beautiful tubas, each with a neat attached label." src="https://static.simonwillison.net/static/2024/tuba-collection-card.jpeg" /&gt;&lt;/p&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/accessibility"&gt;accessibility&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/conferences"&gt;conferences&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/django"&gt;django&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/djangocon"&gt;djangocon&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/jacob-kaplan-moss"&gt;jacob-kaplan-moss&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/python"&gt;python&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/dsf"&gt;dsf&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/pydantic"&gt;pydantic&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/htmx"&gt;htmx&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="accessibility"/><category term="conferences"/><category term="django"/><category term="djangocon"/><category term="jacob-kaplan-moss"/><category term="python"/><category term="redis"/><category term="dsf"/><category term="pydantic"/><category term="htmx"/></entry><entry><title>redka</title><link href="https://simonwillison.net/2024/Apr/14/redka/#atom-tag" rel="alternate"/><published>2024-04-14T15:21:19+00:00</published><updated>2024-04-14T15:21:19+00:00</updated><id>https://simonwillison.net/2024/Apr/14/redka/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/nalgeon/redka"&gt;redka&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Anton Zhiyanov’s new project to build a subset of Redis (including protocol support) using Go and SQLite. Also works as a Go library.&lt;/p&gt;

&lt;p&gt;The guts of the SQL implementation are in the internal/sqlx folder.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://twitter.com/ohmypy/status/1779489065173467385"&gt;@ohmypy&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/go"&gt;go&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/sqlite"&gt;sqlite&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/anton-zhiyanov"&gt;anton-zhiyanov&lt;/a&gt;&lt;/p&gt;



</summary><category term="go"/><category term="redis"/><category term="sqlite"/><category term="anton-zhiyanov"/></entry><entry><title>Redis Adopts Dual Source-Available Licensing</title><link href="https://simonwillison.net/2024/Mar/21/redis-adopts-dual-source-available-licensing/#atom-tag" rel="alternate"/><published>2024-03-21T02:24:40+00:00</published><updated>2024-03-21T02:24:40+00:00</updated><id>https://simonwillison.net/2024/Mar/21/redis-adopts-dual-source-available-licensing/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://redis.com/blog/redis-adopts-dual-source-available-licensing/"&gt;Redis Adopts Dual Source-Available Licensing&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Well this sucks: after fifteen years (and contributions from more than 700 people), Redis is dropping the 3-clause BSD license going forward, instead being “dual-licensed under the Redis Source Available License (RSALv2) and Server Side Public License (SSPLv1)” from Redis 7.4 onwards.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://mstdn.social/@msw/112130308202090850"&gt;@msw&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/open-source"&gt;open-source&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="open-source"/><category term="redis"/></entry><entry><title>Testcontainers</title><link href="https://simonwillison.net/2024/Feb/28/testcontainers/#atom-tag" rel="alternate"/><published>2024-02-28T02:41:38+00:00</published><updated>2024-02-28T02:41:38+00:00</updated><id>https://simonwillison.net/2024/Feb/28/testcontainers/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://testcontainers.com/"&gt;Testcontainers&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Not sure how I missed this: Testcontainers is a family of testing libraries (for Python, Go, JavaScript, Ruby, Rust and a bunch more) that make it trivial to spin up a service such as PostgreSQL or Redis in a container for the duration of your tests and then spin it back down again.&lt;/p&gt;
&lt;p&gt;The Python example code is delightful:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;redis = DockerContainer("redis:5.0.3-alpine").with_exposed_ports(6379)
redis.start()
wait_for_logs(redis, "Ready to accept connections")
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I much prefer integration-style tests over unit tests, and I like to make sure any of my projects that depend on PostgreSQL or similar can run their tests against a real running instance. I've invested heavily in spinning up Varnish or Elasticsearch ephemeral instances in the past - Testcontainers look like they could save me a lot of time.&lt;/p&gt;
&lt;p&gt;The open source project started in 2015, span off a company called AtomicJar in 2021 and was acquired by Docker in December 2023.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=39531536"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/testing"&gt;testing&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/docker"&gt;docker&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="testing"/><category term="docker"/></entry><entry><title>lmdb.tcl  - the first version of Redis, written in TCL</title><link href="https://simonwillison.net/2023/May/18/lmdb/#atom-tag" rel="alternate"/><published>2023-05-18T16:57:52+00:00</published><updated>2023-05-18T16:57:52+00:00</updated><id>https://simonwillison.net/2023/May/18/lmdb/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e88a8"&gt;lmdb.tcl  - the first version of Redis, written in TCL&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Really neat piece of computing history here—the very first version of what later became Redis, written as a 319 line TCL stript for LLOOGG, Salvatore Sanfilippo’s old analytics startup.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=35989909"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="salvatore-sanfilippo"/></entry><entry><title>Scaling Mastodon: The Compendium</title><link href="https://simonwillison.net/2022/Nov/29/scaling-mastodon-the-compendium/#atom-tag" rel="alternate"/><published>2022-11-29T05:46:03+00:00</published><updated>2022-11-29T05:46:03+00:00</updated><id>https://simonwillison.net/2022/Nov/29/scaling-mastodon-the-compendium/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://hazelweakly.me/blog/scaling-mastodon/"&gt;Scaling Mastodon: The Compendium&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Hazel Weakly’s collection of notes on scaling Mastodon, covering PostgreSQL, Sidekiq, Redis, object storage and more.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://hachyderm.io/@nova/109422755533605556"&gt;hachyderm.io/@nova&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/scaling"&gt;scaling&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/mastodon"&gt;mastodon&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/sidekiq"&gt;sidekiq&lt;/a&gt;&lt;/p&gt;



</summary><category term="postgresql"/><category term="redis"/><category term="scaling"/><category term="mastodon"/><category term="sidekiq"/></entry><entry><title>Dragonfly: A modern replacement for Redis and Memcached</title><link href="https://simonwillison.net/2022/May/30/dragonfly/#atom-tag" rel="alternate"/><published>2022-05-30T22:02:32+00:00</published><updated>2022-05-30T22:02:32+00:00</updated><id>https://simonwillison.net/2022/May/30/dragonfly/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/dragonflydb/dragonfly"&gt;Dragonfly: A modern replacement for Redis and Memcached&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
I was initially pretty skeptical of the tagline: does Redis really need a “modern” replacement? But the Background section of the README makes this look like a genuinely interesting project. It re-imagines Redis to have its keyspace partitioned across multiple threads, and uses the VLL lock manager described in a 2014 paper to “compose atomic multi-key operations without using mutexes or spinlocks”. The initial benchmarks show up to a 25x increase in throughput compared to Redis. It’s written in C++.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=31560547"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/c-plus-plus"&gt;c-plus-plus&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="c-plus-plus"/><category term="redis"/></entry><entry><title>A tiny CI system</title><link href="https://simonwillison.net/2022/Apr/26/a-tiny-ci-system/#atom-tag" rel="alternate"/><published>2022-04-26T15:39:27+00:00</published><updated>2022-04-26T15:39:27+00:00</updated><id>https://simonwillison.net/2022/Apr/26/a-tiny-ci-system/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://www.0chris.com/tiny-ci-system.html"&gt;A tiny CI system&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Christian Ştefănescu shares a recipe for building a tiny self-hosted CI system using Git and Redis. A post-receive hook runs when a commit is pushed to the repo and uses redis-cli to push jobs to a list. Then a separate bash script runs a loop with a blocking “redis-cli blpop jobs” operation which waits for new jobs and then executes the CI job as a shell script.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://twitter.com/stchris_/status/1518977088723861505"&gt;@stchris_&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/bash"&gt;bash&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/continuous-integration"&gt;continuous-integration&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/git"&gt;git&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="bash"/><category term="continuous-integration"/><category term="git"/><category term="redis"/></entry><entry><title>Last Mile Redis</title><link href="https://simonwillison.net/2021/Jul/17/last-mile-redis/#atom-tag" rel="alternate"/><published>2021-07-17T02:44:55+00:00</published><updated>2021-07-17T02:44:55+00:00</updated><id>https://simonwillison.net/2021/Jul/17/last-mile-redis/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://fly.io/blog/last-mile-redis/"&gt;Last Mile Redis&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Fly.io article about running a local redis cache in each of their geographic regions—“Cache data overlaps a lot less than you assume it will. For the most part, people in Singapore will rely on a different subset of data than people in Amsterdam or São Paulo or New Jersey.” But then they note that Redis has the ability to act as both a replica of a primary AND a writable server at the same time (“replica-read-only no”), which actually makes sense for a cache—it lets you cache local data but send out cluster-wide cache purges if necessary.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=27861510"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/caching"&gt;caching&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/fly"&gt;fly&lt;/a&gt;&lt;/p&gt;



</summary><category term="caching"/><category term="redis"/><category term="fly"/></entry><entry><title>RabbitMQ Streams Overview</title><link href="https://simonwillison.net/2021/Jul/13/rabbitmq-streams-overview/#atom-tag" rel="alternate"/><published>2021-07-13T23:29:18+00:00</published><updated>2021-07-13T23:29:18+00:00</updated><id>https://simonwillison.net/2021/Jul/13/rabbitmq-streams-overview/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/"&gt;RabbitMQ Streams Overview&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
New in RabbitMQ 3.9: streams are a persisted, replicated append-only log with non-destructive consuming semantics. Sounds like it fits the same hole as Kafka and Redis Streams, an extremely useful pattern.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/message-queues"&gt;message-queues&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/rabbitmq"&gt;rabbitmq&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/kafka"&gt;kafka&lt;/a&gt;&lt;/p&gt;



</summary><category term="message-queues"/><category term="rabbitmq"/><category term="redis"/><category term="kafka"/></entry><entry><title>huey</title><link href="https://simonwillison.net/2019/Feb/25/huey/#atom-tag" rel="alternate"/><published>2019-02-25T19:49:06+00:00</published><updated>2019-02-25T19:49:06+00:00</updated><id>https://simonwillison.net/2019/Feb/25/huey/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/coleifer/huey"&gt;huey&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Charles Leifer’s “little task queue for Python”. Similar to Celery, but it’s designed to work with Redis, SQLite or in the parent process using background greenlets. Worth checking out for the really neat design. The project is new to me, but it’s been under active development since 2011 and has a very healthy looking rate of releases.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/python"&gt;python&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/queues"&gt;queues&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/sqlite"&gt;sqlite&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/charles-leifer"&gt;charles-leifer&lt;/a&gt;&lt;/p&gt;



</summary><category term="python"/><category term="queues"/><category term="redis"/><category term="sqlite"/><category term="charles-leifer"/></entry><entry><title>Quoting Mesh: Compacting Memory Management for C/C++ Applications</title><link href="https://simonwillison.net/2019/Feb/18/mesh-compacting-memory-management-cc-applications/#atom-tag" rel="alternate"/><published>2019-02-18T15:26:59+00:00</published><updated>2019-02-18T15:26:59+00:00</updated><id>https://simonwillison.net/2019/Feb/18/mesh-compacting-memory-management-cc-applications/#atom-tag</id><summary type="html">
    &lt;blockquote cite="https://arxiv.org/abs/1902.04738"&gt;&lt;p&gt;This paper introduces Mesh, a plug-in replacement for malloc that, for the first time, eliminates fragmentation in unmodified C/C++ applications. Mesh combines novel randomized algorithms with widely-supported virtual memory operations to provably reduce fragmentation, breaking the classical Robson bounds with high probability. Mesh generally matches the runtime performance of state-of-the-art memory allocators while reducing memory consumption; in particular, it reduces the memory of consumption of Firefox by 16% and Redis by 39%.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="https://arxiv.org/abs/1902.04738"&gt;Mesh: Compacting Memory Management for C/C++ Applications&lt;/a&gt;&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/firefox"&gt;firefox&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/memory"&gt;memory&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="firefox"/><category term="memory"/><category term="redis"/></entry><entry><title>Introduction to Redis Streams</title><link href="https://simonwillison.net/2018/Oct/18/introduction-to-redis-streams/#atom-tag" rel="alternate"/><published>2018-10-18T08:35:03+00:00</published><updated>2018-10-18T08:35:03+00:00</updated><id>https://simonwillison.net/2018/Oct/18/introduction-to-redis-streams/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://redis.io/topics/streams-intro"&gt;Introduction to Redis Streams&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Redis 5.0 is out, introducing the first new Redis data type in several years: streams, a Kafka-like mechanism for implementing a replayable event stream that can be read by many different subscribers.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/kafka"&gt;kafka&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="kafka"/></entry><entry><title>Scaling a High-traffic Rate Limiting Stack With Redis Cluster</title><link href="https://simonwillison.net/2018/Apr/26/redis-cluster/#atom-tag" rel="alternate"/><published>2018-04-26T18:34:25+00:00</published><updated>2018-04-26T18:34:25+00:00</updated><id>https://simonwillison.net/2018/Apr/26/redis-cluster/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://brandur.org/redis-cluster"&gt;Scaling a High-traffic Rate Limiting Stack With Redis Cluster&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Brandur Leach describes the simple, elegant and performant design of Redis Cluster, and talks about how Stripe used it to scaled their rate-limiting from one to ten nodes.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/rate-limiting"&gt;rate-limiting&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/scaling"&gt;scaling&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/brandur-leach"&gt;brandur-leach&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/stripe"&gt;stripe&lt;/a&gt;&lt;/p&gt;



</summary><category term="rate-limiting"/><category term="redis"/><category term="scaling"/><category term="brandur-leach"/><category term="stripe"/></entry><entry><title>Quoting The Redis Manifesto</title><link href="https://simonwillison.net/2018/Mar/2/redis-manifesto/#atom-tag" rel="alternate"/><published>2018-03-02T19:11:31+00:00</published><updated>2018-03-02T19:11:31+00:00</updated><id>https://simonwillison.net/2018/Mar/2/redis-manifesto/#atom-tag</id><summary type="html">
    &lt;blockquote cite="https://github.com/antirez/redis/blob/unstable/MANIFESTO"&gt;&lt;p&gt;Code is like a poem; it's not just something we write to reach some practical result. Sometimes people that are far from the Redis philosophy suggest using other code written by other authors (frequently in other languages) in order to implement something Redis currently lacks. But to us this is like if Shakespeare decided to end Enrico IV using the Paradiso from the Divina Commedia. Is using any external code a bad idea? Not at all. Like in "One Thousand and One Nights" smaller self contained stories are embedded in a bigger story, we'll be happy to use beautiful self contained libraries when needed. At the same time, when writing the Redis story we're trying to write smaller stories that will fit in to other code.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="https://github.com/antirez/redis/blob/unstable/MANIFESTO"&gt;The Redis Manifesto&lt;/a&gt;&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/programming"&gt;programming&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="programming"/><category term="redis"/></entry><entry><title>Quoting Salvatore Sanfilippo</title><link href="https://simonwillison.net/2017/Nov/8/disque/#atom-tag" rel="alternate"/><published>2017-11-08T22:51:14+00:00</published><updated>2017-11-08T22:51:14+00:00</updated><id>https://simonwillison.net/2017/Nov/8/disque/#atom-tag</id><summary type="html">
    &lt;blockquote cite="https://news.ycombinator.com/item?id=15656938"&gt;&lt;p&gt;For Redis 4.2 I'm moving Disque as a Redis module. To do this, Redis modules are getting a fully featured Cluster API. This means that it will be possible, for instance, to write a Redis module that orchestrates N Redis masters, using Raft or any other consensus algorithm, as a single distributed system. This will allow to also model strong guarantees easily.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="https://news.ycombinator.com/item?id=15656938"&gt;Salvatore Sanfilippo&lt;/a&gt;&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="salvatore-sanfilippo"/></entry><entry><title>Redis Streams and the Unified Log</title><link href="https://simonwillison.net/2017/Nov/8/redis-streams-and-the-unified-log/#atom-tag" rel="alternate"/><published>2017-11-08T16:37:52+00:00</published><updated>2017-11-08T16:37:52+00:00</updated><id>https://simonwillison.net/2017/Nov/8/redis-streams-and-the-unified-log/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://brandur.org/redis-streams#rocket-rides-unified"&gt;Redis Streams and the Unified Log&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
In which Brandur Leach explores the new Kafka-style streams functionality coming to Redis 4.0, and shows an example of a robust at-least once processing architecture built on a combination of Redis streams and PostgreSQL transactions. I really like the pattern of writing log records to a staging table in PostgreSQL first in order to bundle them up in the same transaction as the originating state change, then have a separate process read them from that table and publish them to Redis.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/brandur-leach"&gt;brandur-leach&lt;/a&gt;&lt;/p&gt;



</summary><category term="postgresql"/><category term="redis"/><category term="brandur-leach"/></entry><entry><title>Quoting Brandur Leach</title><link href="https://simonwillison.net/2017/Nov/8/redis-streams/#atom-tag" rel="alternate"/><published>2017-11-08T16:23:55+00:00</published><updated>2017-11-08T16:23:55+00:00</updated><id>https://simonwillison.net/2017/Nov/8/redis-streams/#atom-tag</id><summary type="html">
    &lt;blockquote cite="https://brandur.org/redis-streams"&gt;&lt;p&gt;Redis streams aren’t exciting for their innovativeness, but rather than they bring building a unified log architecture within reach of a small and/or inexpensive app. Kafka is infamously difficult to configure and get running, and is expensive to operate once you do.  [...] Redis on the other hand is probably already in your stack.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="https://brandur.org/redis-streams"&gt;Brandur Leach&lt;/a&gt;&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/kafka"&gt;kafka&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/brandur-leach"&gt;brandur-leach&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="kafka"/><category term="brandur-leach"/></entry><entry><title>Secondary indexing with Redis</title><link href="https://simonwillison.net/2017/Nov/7/secondary-indexing-with-redis/#atom-tag" rel="alternate"/><published>2017-11-07T02:00:13+00:00</published><updated>2017-11-07T02:00:13+00:00</updated><id>https://simonwillison.net/2017/Nov/7/secondary-indexing-with-redis/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://redis.io/topics/indexes"&gt;Secondary indexing with Redis&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
I haven’t seen this section of the official Redis documentation before, and it’s absolutely fantastic—well worth reading the whole thing. It talks through various ways in which you can set up indexes in Redis, mainly by leaning on sorted sets—which it turns out will binary lexicographically sort items with the same score. This makes it easy to implement autocomplete with Redis—but if you use them creatively you can implement subject/predicate/object graph searches or even N-dimensional range queries as well.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/></entry><entry><title>walrus</title><link href="https://simonwillison.net/2017/Nov/6/walrus/#atom-tag" rel="alternate"/><published>2017-11-06T01:14:58+00:00</published><updated>2017-11-06T01:14:58+00:00</updated><id>https://simonwillison.net/2017/Nov/6/walrus/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://walrus.readthedocs.io/en/latest/"&gt;walrus&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Fascinating collection of Python utilities for working with Redis, by Charles Leifer. There are a ton of interesting ideas in here. It starts with Python object wrappers for Redis so you can interact with lists, sets, sorted sets and Redis hashes using Python-like objects. Then it gets really interesting: walrus ships with implementations of autocomplete, rate limiting, a graph engine (using a sorted set hexastore) and an ORM-style models mechanism which manages secondary indexes and even implements basic full-text search.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/python"&gt;python&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/charles-leifer"&gt;charles-leifer&lt;/a&gt;&lt;/p&gt;



</summary><category term="python"/><category term="redis"/><category term="charles-leifer"/></entry><entry><title>Scaling the GitLab database</title><link href="https://simonwillison.net/2017/Oct/30/scaling-the-gitlab-database/#atom-tag" rel="alternate"/><published>2017-10-30T20:53:24+00:00</published><updated>2017-10-30T20:53:24+00:00</updated><id>https://simonwillison.net/2017/Oct/30/scaling-the-gitlab-database/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://about.gitlab.com/2017/10/02/scaling-the-gitlab-database/"&gt;Scaling the GitLab database&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Lots of interesting details on how GitLab have worked to scale their PostgreSQL setup. They’ve avoided sharding so far, instead opting for database pooling with pgbouncer and read-only replicas using hot standbys. I like the way they deal with replica lag—they store the current WAL position in a redis key for the user every time there’s a write, then use pg_last_xlog_replay_location() on the various replicas to check and see if they have caught up next time the user makes a request that needs to read some data.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/replication"&gt;replication&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/scaling"&gt;scaling&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/gitlab"&gt;gitlab&lt;/a&gt;&lt;/p&gt;



</summary><category term="postgresql"/><category term="redis"/><category term="replication"/><category term="scaling"/><category term="gitlab"/></entry><entry><title>Streams: a new general purpose data structure in Redis</title><link href="https://simonwillison.net/2017/Oct/3/redis-streams/#atom-tag" rel="alternate"/><published>2017-10-03T15:25:41+00:00</published><updated>2017-10-03T15:25:41+00:00</updated><id>https://simonwillison.net/2017/Oct/3/redis-streams/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="http://antirez.com/news/114"&gt;Streams: a new general purpose data structure in Redis&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Exciting new Redis feature inspired by Kafka: redis streams, which allow you to construct an efficient, in-memory list of messages (similar to a Kafka log) which clients can read sections of or block against and await real-time delivery of new messages. As expected from Salvatore the API design is clean, obvious and covers a wide range of exciting use-cases. Planned for release with Redis 4 by the end of the year!


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/salvatore-sanfilippo"&gt;salvatore-sanfilippo&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/kafka"&gt;kafka&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="salvatore-sanfilippo"/><category term="kafka"/></entry><entry><title>Can you recommend a few video lectures on web application design with Redis?</title><link href="https://simonwillison.net/2012/Nov/25/can-you-recommend-a/#atom-tag" rel="alternate"/><published>2012-11-25T11:23:00+00:00</published><updated>2012-11-25T11:23:00+00:00</updated><id>https://simonwillison.net/2012/Nov/25/can-you-recommend-a/#atom-tag</id><summary type="html">
    &lt;p&gt;&lt;em&gt;My answer to &lt;a href="https://www.quora.com/Can-you-recommend-a-few-video-lectures-on-web-application-design-with-Redis/answer/Simon-Willison"&gt;Can you recommend a few video lectures on web application design with Redis?&lt;/a&gt; on Quora&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We have a collection of 27 videos of conference talks that cover Redis on Lanyrd, gathered by our community: &lt;span&gt;&lt;a href="http://lanyrd.com/topics/redis/video/"&gt;Redis Videos | Lanyrd&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This talk by &lt;span&gt;&lt;a href="https://www.quora.com/profile/Obie-Fernandez"&gt;Obie Fernandez&lt;/a&gt;&lt;/span&gt; on &lt;span&gt;&lt;a href="http://lanyrd.com/2012/railsconf/srhtx/"&gt;Redis Application Patterns in Rails&lt;/a&gt;&lt;/span&gt; sounds like it might cover what you're looking for.&lt;/p&gt;

&lt;p&gt;
&lt;iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/dH6VYRMRQFw" frameborder="0" allowfullscreen="allowfullscreen"&gt;
&lt;/iframe&gt;
&lt;/p&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/web-development"&gt;web-development&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/quora"&gt;quora&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="redis"/><category term="web-development"/><category term="quora"/></entry><entry><title>How we use Redis at Bump</title><link href="https://simonwillison.net/2011/Jul/16/bump/#atom-tag" rel="alternate"/><published>2011-07-16T16:37:00+00:00</published><updated>2011-07-16T16:37:00+00:00</updated><id>https://simonwillison.net/2011/Jul/16/bump/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="http://devblog.bu.mp/how-we-use-redis-at-bump"&gt;How we use Redis at Bump&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
A couple of neat tricks I hadn’t seen before: using Redis to aggregate log files from multiple servers (they all push in to a Redis queue, then one process pulls from the queue and writes to disk), and using Redis blocking queues for RPC by specifying a different temporary queue to return the result.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/recovered"&gt;recovered&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/bump"&gt;bump&lt;/a&gt;&lt;/p&gt;



</summary><category term="redis"/><category term="recovered"/><category term="bump"/></entry><entry><title>Quoting Dan Manges</title><link href="https://simonwillison.net/2011/Jun/30/braintree/#atom-tag" rel="alternate"/><published>2011-06-30T21:27:00+00:00</published><updated>2011-06-30T21:27:00+00:00</updated><id>https://simonwillison.net/2011/Jun/30/braintree/#atom-tag</id><summary type="html">
    &lt;blockquote cite="http://www.braintreepayments.com/inside-braintree/how-we-built-the-software-that-processes-billions-in-payments"&gt;&lt;p&gt;We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic, which gives us a window of a few seconds to make some changes before letting the requests through. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python’s Tornado and Redis.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p class="cite"&gt;&amp;mdash; &lt;a href="http://www.braintreepayments.com/inside-braintree/how-we-built-the-software-that-processes-billions-in-payments"&gt;Dan Manges&lt;/a&gt;, Braintree&lt;/p&gt;

    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/deployment"&gt;deployment&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/http"&gt;http&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/redis"&gt;redis&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/tornado"&gt;tornado&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/recovered"&gt;recovered&lt;/a&gt;&lt;/p&gt;



</summary><category term="deployment"/><category term="http"/><category term="redis"/><category term="tornado"/><category term="recovered"/></entry></feed>