7 items tagged “ab-testing”
2024
I think the mistake the industry has made is (and I had to learn this as well), that "we observed ab tests work really well" is really a statement that should read "the majority of the changes we make are characterized as hill-climbing growth of a post-PMF b2c product and ab tests work really well for that".
2023
Optimizing for Taste. David Cramer’s detailed explanation as to why his company Sentry mostly avoids A/B testing. David wrote this as an internal blog post originally, but is now sharing it with the world. I found myself nodding along vigorously as I read this—lots of astute observations here.
I particularly appreciated his closing note: “The strength of making a decision is making it. You can always make a new one later. Choose the obvious path forward, and if you don’t see one, find someone who does.”
2019
Experiments, growth engineering, and exposing company secrets through your API (via) This is fun: Jon Luca observes that many companies that run A/B tests have private JSON APIs that list all of their ongoing experiments, and uses them to explore tests from Lyft, Airbnb, Pinterest, Amazon and more. Facebook and Instagram use SSL Stapling which makes it harder to spy on their mobile app traffic.
2017
now-ab. Intriguing example of a Zeit Now microservice: now-ab is a Node.js HTTP proxy which proxies through to one of two or more other Now-deployed applications based on a cookie. If you don’t have the cookie, it picks a backend at random and sets the cookie. Admittedly this is the easiest part of implementing A/B testing (the hard part is the analytics: tracking exposures and conversions) but as an example of a microservice architectural pattern this is fascinating.
2009
FT.com: The rise and fall of MySpace (via) Lots of stuff about the internal politics at News Corporation. Of particular interest: MySpace have to take feature proposals to News Corp for approval. Meanwhile, Facebook are leading the industry in their use of A/B testing to figure out exactly what features their users will respond well to.
Effective A/B Testing. Impressively comprehensive presentation on A/B testing, from theory to practice to statistical analysis of the results.
You should follow me on Twitter. Dustin Curtis did a simple A/B testing experiment on his blog and found that the text “you should follow me on Twitter” had the highest click-through rate—173% more effective than “I’m on Twitter”.