Simon Willison’s Weblog

Subscribe

18 items tagged “logging”

2023

Queryable Logging with Blacklite (via) Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a “live” database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.

Lots of interesting notes here on using SQLite for high performance logging.

“SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs.” # 21st August 2023, 6:13 pm

2021

Replaying logs to exercise the new API

22 days ago n1mmy pushed a change to help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.

[... 542 words]

2019

Logging to SQLite using ASGI middleware

I had some fun playing around with ASGI middleware and logging during our flight back to England for the holidays.

[... 2535 words]

Logs vs. metrics: a false dichotomy (via) Nick Stenning discusses the differences between logs and metrics: most notably that metrics can be derived from logs but logs cannot be reconstituted starting with time-series metrics. # 3rd August 2019, 4:46 pm

Targeted diagnostic logging in production (via) Will Sargent defines diagnostic logging as “debug logging statements with an audience”, and proposes controlling this style if logging via a feature flat system to allow detailed logging to be turned on in production against a selected subset if users in order to help debug difficult problems. Lots of great background material in the topic of observability here too. # 24th July 2019, 5:44 am

2017

Running gunicorn behind nginx on Heroku for buffering and logging

Heroku’s default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).

[... 400 words]

2012

System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?

We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.

[... 113 words]

2009

clarity. A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine. # 4th November 2009, 10:36 pm

Python Logging 101. A really useful introduction to Python’s logging module by that module’s author, Vinay Sajip. # 29th September 2009, 6:40 pm

Django ponies: Proposals for Django 1.2

I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.

[... 1674 words]

Justniffer. Packet sniffing tool that can output sniffed HTTP traffic formatted the same way as an Apache access_log file. # 25th September 2009, 10:12 pm

“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB. # 26th August 2009, 7:09 pm

Python logging from multiple processes. Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs. # 13th August 2009, 11:55 pm

2008

django-db-log. Middleware that logs Django exceptions to the database, using a clever scheme based on an MD5 of the traceback text to group duplicate errors in to batches. # 13th May 2008, 8:07 am

daemon.py (via) Neat little Python module for daemonizing a process; handles logging and pid files out of the box. # 8th January 2008, 9:58 pm

2007

LoggerFS. Clever use of FUSE: a virtual filesystem which looks out for lines appended to a log file (matched with a regular expression) and stores them in a database instead. # 29th October 2007, 10:40 am

jQuery Logging (via) Brilliant four line jQuery plugin that lets you insert Firebug console.log() calls directly in to chains. # 19th October 2007, 12:52 pm

django-logging. Looks neat—includes the ability to use Python’s standard logging module to log messages to a footer appended to your site’s HTML output. # 24th April 2007, 6:50 am