18 items tagged “logging”
2023
Queryable Logging with Blacklite (via) Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a “live” database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.
Lots of interesting notes here on using SQLite for high performance logging.
“SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs.”
2021
Replaying logs to exercise the new API
22 days ago n1mmy pushed a change to help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.
2019
Logging to SQLite using ASGI middleware
I had some fun playing around with ASGI middleware and logging during our flight back to England for the holidays.
[... 2,535 words]Logs vs. metrics: a false dichotomy (via) Nick Stenning discusses the differences between logs and metrics: most notably that metrics can be derived from logs but logs cannot be reconstituted starting with time-series metrics.
Targeted diagnostic logging in production (via) Will Sargent defines diagnostic logging as “debug logging statements with an audience”, and proposes controlling this style if logging via a feature flat system to allow detailed logging to be turned on in production against a selected subset if users in order to help debug difficult problems. Lots of great background material in the topic of observability here too.
2017
Running gunicorn behind nginx on Heroku for buffering and logging
Heroku’s default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).
[... 400 words]2012
System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?
We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.
[... 113 words]2009
clarity. A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine.
Python Logging 101. A really useful introduction to Python’s logging module by that module’s author, Vinay Sajip.
Django ponies: Proposals for Django 1.2
I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.
[... 1,674 words]Justniffer. Packet sniffing tool that can output sniffed HTTP traffic formatted the same way as an Apache access_log file.
“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB.
Python logging from multiple processes. Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs.
2008
django-db-log. Middleware that logs Django exceptions to the database, using a clever scheme based on an MD5 of the traceback text to group duplicate errors in to batches.
daemon.py (via) Neat little Python module for daemonizing a process; handles logging and pid files out of the box.
2007
LoggerFS. Clever use of FUSE: a virtual filesystem which looks out for lines appended to a log file (matched with a regular expression) and stores them in a database instead.
jQuery Logging (via) Brilliant four line jQuery plugin that lets you insert Firebug console.log() calls directly in to chains.
django-logging. Looks neat—includes the ability to use Python’s standard logging module to log messages to a footer appended to your site’s HTML output.