Simon Willison’s Weblog


6 items tagged “abtesting”


Optimizing for Taste. David Cramer’s detailed explanation as to why his company Sentry mostly avoids A/B testing. David wrote this as an internal blog post originally, but is now sharing it with the world. I found myself nodding along vigorously as I read this—lots of astute observations here.

I particularly appreciated his closing note: “The strength of making a decision is making it. You can always make a new one later. Choose the obvious path forward, and if you don’t see one, find someone who does.” # 27th September 2023, 4:34 am


Experiments, growth engineering, and exposing company secrets through your API (via) This is fun: Jon Luca observes that many companies that run A/B tests have private JSON APIs that list all of their ongoing experiments, and uses them to explore tests from Lyft, Airbnb, Pinterest, Amazon and more. Facebook and Instagram use SSL Stapling which makes it harder to spy on their mobile app traffic. # 26th February 2019, 4:49 am


now-ab. Intriguing example of a Zeit Now microservice: now-ab is a Node.js HTTP proxy which proxies through to one of two or more other Now-deployed applications based on a cookie. If you don’t have the cookie, it picks a backend at random and sets the cookie. Admittedly this is the easiest part of implementing A/B testing (the hard part is the analytics: tracking exposures and conversions) but as an example of a microservice architectural pattern this is fascinating. # 16th November 2017, 11:03 pm

2009 The rise and fall of MySpace (via) Lots of stuff about the internal politics at News Corporation. Of particular interest: MySpace have to take feature proposals to News Corp for approval. Meanwhile, Facebook are leading the industry in their use of A/B testing to figure out exactly what features their users will respond well to. # 5th December 2009, 5:09 pm

Effective A/B Testing. Impressively comprehensive presentation on A/B testing, from theory to practice to statistical analysis of the results. # 13th September 2009, 11:49 pm

You should follow me on Twitter. Dustin Curtis did a simple A/B testing experiment on his blog and found that the text “you should follow me on Twitter” had the highest click-through rate—173% more effective than “I’m on Twitter”. # 15th July 2009, 10:43 am