Simon Willison’s Weblog

13 items tagged “logging”

Running gunicorn behind nginx on Heroku for buffering and logging

Heroku’s default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).

[... 400 words]

System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?

We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.

[... 113 words]

clarity. A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine. # 4th November 2009, 10:36 pm

Python Logging 101. A really useful introduction to Python’s logging module by that module’s author, Vinay Sajip. # 29th September 2009, 6:40 pm

Django ponies: Proposals for Django 1.2

I’ve decided to step up my involvement in Django development in the run-up to Django 1.2, so I’m currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I’m also ensuring I have the code to back them up—my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don’t think he’d appreciate a repeat performance.

[... 1674 words]

Justniffer. Packet sniffing tool that can output sniffed HTTP traffic formatted the same way as an Apache access_log file. # 25th September 2009, 10:12 pm

“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB. # 26th August 2009, 7:09 pm

Python logging from multiple processes. Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs. # 13th August 2009, 11:55 pm

django-db-log. Middleware that logs Django exceptions to the database, using a clever scheme based on an MD5 of the traceback text to group duplicate errors in to batches. # 13th May 2008, 8:07 am

daemon.py (via) Neat little Python module for daemonizing a process; handles logging and pid files out of the box. # 8th January 2008, 9:58 pm

LoggerFS. Clever use of FUSE: a virtual filesystem which looks out for lines appended to a log file (matched with a regular expression) and stores them in a database instead. # 29th October 2007, 10:40 am

jQuery Logging (via) Brilliant four line jQuery plugin that lets you insert Firebug console.log() calls directly in to chains. # 19th October 2007, 12:52 pm

django-logging. Looks neat—includes the ability to use Python’s standard logging module to log messages to a footer appended to your site’s HTML output. # 24th April 2007, 6:50 am