Simon Willison’s Weblog

Log Management: What is the complexity behind Loggly and similar services?

My answer to Log Management: What is the complexity behind Loggly and similar services? on Quora

Pricing isn’t about how hard it is to build something (and building a reliable, highly-scalable centralized log search and archiving system isn’t trivial). It’s about how much value it provides to the customer.

If you are a company that generates 7GB to 1.5TB of logging data per day, making those logs centralized and searchable is a big win for you, and is something that requires a decent amount of engineering effort. $350/month probably isn’t much money for you—if you have more than a dozen or so employees it ends up as more or less a rounding error on your monthly expenses.

Compare that to building it yourself. Firstly, if you build it in-house, you have to maintain it. Can you do that on less than one engineer-day per month? Even if you can, what’s the opportunity cost of them working on logging instead of things that actually help you solve the unique problems that characterise your business? And will the thing you build in-house be as good as the thing that Loggly (a company that only focuses on building excellent log processing software) can provide?