18 items tagged “logging”
Queryable Logging with Blacklite (via) Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a “live” database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.
Lots of interesting notes here on using SQLite for high performance logging.
“SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs.” # 21st August 2023, 6:13 pm
22 days ago n1mmy pushed a change to
help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.
I had some fun playing around with ASGI middleware and logging during our flight back to England for the holidays.[... 2535 words]
Logs vs. metrics: a false dichotomy (via) Nick Stenning discusses the differences between logs and metrics: most notably that metrics can be derived from logs but logs cannot be reconstituted starting with time-series metrics. # 3rd August 2019, 4:46 pm
Targeted diagnostic logging in production (via) Will Sargent defines diagnostic logging as “debug logging statements with an audience”, and proposes controlling this style if logging via a feature flat system to allow detailed logging to be turned on in production against a selected subset if users in order to help debug difficult problems. Lots of great background material in the topic of observability here too. # 24th July 2019, 5:44 am
Heroku’s default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).[... 400 words]
System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?
We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.[... 113 words]
I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.[... 1674 words]
“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB. # 26th August 2009, 7:09 pm
Python logging from multiple processes. Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs. # 13th August 2009, 11:55 pm
LoggerFS. Clever use of FUSE: a virtual filesystem which looks out for lines appended to a log file (matched with a regular expression) and stores them in a database instead. # 29th October 2007, 10:40 am