Simon Willison's Weblog: brandur-leach

Soft Deletion Probably Isn't Worth It

2022-07-19T20:40:06+00:00

Soft Deletion Probably Isn't Worth It

Brandur argues that soft deletion—where you delete records by populating a “is_deleted” or “deleted_at” column in your table—isn’t worth the additional complexity and risk it adds to other database queries. Instead, he suggests having a separate deleted records table which records the deleted data in a JSON blob—allowing you to review and recover it manually if necessary, and giving you an easy way to expire deleted records that have exceeded your retention policy.

Tags: databases, brandur-leach

Heroku: Core Impact

2022-05-16T04:24:50+00:00

Heroku: Core Impact

Ex-Heroku engineer Brandur Leach pulls together some of the background information circulating concerning the now more than a month long Heroku security incident and provides some ex-insider commentary on what went right and what went wrong with a platform that left a huge, if somewhat underappreciated impact on the technology industry at large.

Via Hacker News

Tags: heroku, brandur-leach

Single dependency stacks

2022-02-09T18:43:55+00:00

Single dependency stacks

Brandur Leach notes that the core services at Crunchy (admittedly a PostgreSQL hosting and consultancy company) have only one stateful dependency – Postgres. No Redis, ElasticSearch or anything else. This means that problems like rate limiting and search, which are often farmed out to external services, are all handled using either PostgreSQL or in-memory mechanisms on their servers.

Via Hacker News

Tags: postgresql, brandur-leach

Transactionally Staged Job Drains in Postgres

2021-12-18T01:34:13+00:00

Transactionally Staged Job Drains in Postgres

Any time I see people argue that relational databases shouldn’t be used to implement job queues I think of this post by Brandur from 2017. If you write to a queue before committing a transaction you run the risk of a queue consumer trying to read from the database before the new row becomes visible. If you write to the queue after the transaction there’s a risk an error might result in your message never being written. So: write to a relational staging table as part of the transaction, then have a separate process read from that table and write to the queue.

Tags: postgresql, queues, scaling, brandur-leach

Quoting Hyrum's Law

2018-08-11T00:33:29+00:00

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

— Hyrum's Law

Tags: api-design, brandur-leach

Scaling a High-traffic Rate Limiting Stack With Redis Cluster

2018-04-26T18:34:25+00:00

Scaling a High-traffic Rate Limiting Stack With Redis Cluster

Brandur Leach describes the simple, elegant and performant design of Redis Cluster, and talks about how Stripe used it to scaled their rate-limiting from one to ten nodes.

Tags: rate-limiting, redis, scaling, brandur-leach, stripe

Touring a Fast, Safe, and Complete(ish) Web Service in Rust

2018-03-28T15:47:26+00:00

Touring a Fast, Safe, and Complete(ish) Web Service in Rust

Brandur’s notes from building a high performance web service in Rust, using PostgreSQL via the Diesel ORM and the Rust actix-web framework which provides Erlang-style actors and promise-based async concurrency.

Tags: async, postgresql, rust, brandur-leach

Scaling Postgres with Read Replicas & Using WAL to Counter Stale Reads

2017-11-18T18:42:46+00:00

Scaling Postgres with Read Replicas & Using WAL to Counter Stale Reads

The problem with sending writes to the primary and balancing reads across replicas is dealing with replica lag—what if you write to the primary and then read from a replica that hasn’t had the new state applied to it yet? Brandur Leach dives deep into an elegant solution using PostgreSQL’s LSN (log sequence numbers) accesesed using pg_last_wal_replay_lsn(). An observer process continuously polls the replicas for their most recently applied LSN and stores them in a table. A column in the Users table then records the min_lsn valid for that user, updating it to the pg_current_wal_lsn() of the primary whenever that user makes a write. Combining the two allows the application to randomly select a replica that is up-to-date for the purposes of a specific user any time it needs to make a read.

Via Hacker News

Tags: postgresql, replication, scaling, brandur-leach

Redis Streams and the Unified Log

2017-11-08T16:37:52+00:00

Redis Streams and the Unified Log

In which Brandur Leach explores the new Kafka-style streams functionality coming to Redis 4.0, and shows an example of a robust at-least once processing architecture built on a combination of Redis streams and PostgreSQL transactions. I really like the pattern of writing log records to a staging table in PostgreSQL first in order to bundle them up in the same transaction as the originating state change, then have a separate process read them from that table and publish them to Redis.

Tags: postgresql, redis, brandur-leach

Quoting Brandur Leach

2017-11-08T16:23:55+00:00

Redis streams aren’t exciting for their innovativeness, but rather than they bring building a unified log architecture within reach of a small and/or inexpensive app. Kafka is infamously difficult to configure and get running, and is expensive to operate once you do. [...] Redis on the other hand is probably already in your stack.

— Brandur Leach

Tags: redis, kafka, brandur-leach

Benefit of TEXT with CHECK over VARCHAR(X) in PostgreSQL

2017-10-28T00:59:34+00:00

Benefit of TEXT with CHECK over VARCHAR(X) in PostgreSQL

Brandur suggests using email TEXT CHECK (char_length(email) <= 255) to define a column with a length limit in PostgreSQL over VARCHAR(255) because TEXT and VARCHAR are equally performant but a CHECK length can be changed later on without locking the table, whereas a VARCHAR requires an ALTER TABLE with an exclusive lock.

Tags: postgresql, brandur-leach

Implementing Stripe-like Idempotency Keys in Postgres

2017-10-27T17:51:52+00:00

Implementing Stripe-like Idempotency Keys in Postgres

Having clients send “idempotency keys” with API requests in order to be able to safely retry them if something’s goes wrong is a really neat trick for making transactional APIs more robust. Here Brandur Leach talks implementation strategies.

Via Hacker News

Tags: api-design, idempotency, postgresql, brandur-leach, stripe