26 items tagged “sysadmin”
2021
Why all my servers have an 8GB empty file (via) This is such a good idea: keep a 8GB empty spacer.img file on any servers you operate, purely so that if the server runs out of space you can delete the file and get some breathing room for getting everything else working again. I’ve had servers run out of space in the past and it’s an absolute pain to sort out—this trick would have really helped.
2019
Setting up Datasette, step by step (via) Tobias describes how he runs Datasette on his own server/VPS, using nginx and systemd. I’m doing something similar for some projects and systemd really does feel like the solution to the “ensure a Python process keeps running” problem I’ve been fighting for over a decade. I really like how Tobias creates a dedicated Linux user for each of his deployed Python projects.
2013
Would you test your web app against simulated infrastructure failure?
I think this is a pretty interesting idea—simulating these kind of conditions isn’t easy so I imagine many teams don’t bother. If it was good (really easy to get started with, great control and reporting tools, maybe helped set up the actual tests to replay) and I trusted the service I would definitely consider paying for it.
[... 125 words]2012
I would like to setup a web-server which will be used solely by myself. What would be the safest way to do so in terms of confidentiality of the contents?
I haven’t configured them myself, but it might be worth looking in to client SSL certificates for this. That way your server won’t communicate with any browser that hasn’t installed a certificate which you generate. I believe the BBC used to use this for a lot of their important servers which they wanted to be accessible only by their own developers from across the internet (I don’t know if they still do).
[... 108 words]What are some system administration best practices? If someone is running a production web server, what are the basic things they should be doing?
Graph everything. I’m not a good sysadmin, but one thing I’ve learned from working with with good sysadmins is that they spend a bunch of time looking at graphs.
[... 107 words]System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?
We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.
[... 113 words]2011
The Virtues of Monitoring. Fantastic guide to the various levels of monitoring required for a modern web application.
2010
Setting up Munin on Ubuntu. Useful guide to setting up my favourite graphing/monitoring tool for personal projects.
Undelete! How to undelete a file accidentally removed using rm on Linux, by grepping through the raw bytes on the hard drive searching for a unique string that was contained in the file. “grep -a -B 25 -A 100 ’some string in the file’ /dev/sda1 > results.txt”
Automated deployments with Fabric—tips and tricks. “If it’s not in a Fabric fabfile, it’s not deployable”—I’m slowly applying this philosophy to my personal projects.
Automate EC2 Instance Setup with user-data Scripts (via) I knew about EC2’s user-data feature—what I didn’t know is that the Alestic and Canonical images are configured so that if the user-data starts with #! the instance will automatically execute it as a shell script as soon as it boots up (after networking has been configured).
Notes from a production MongoDB deployment. Notes from running MongoDB for 8 months in production, with 664 million documents spread across 72 GB master and slave servers located in two different data centers.
jacobian’s django-deployment-workshop. Notes and resources from Jacob’s 3 hour Django deployment workshop at PyCon, including example configuration files for Apache2 + mod_wsgi, nginx, PostgreSQL and pgpool.
Linux performance basics. This kind of Linux knowledge is rapidly becoming a key skill for server-side web development.
2009
Round-robin Django setup with nginx. An nginx trick I didn’t know: a low proxy_connect_timeout value (e.g. 2 seconds) combined with the proxy_next_upstream setting means that if one of your backends breaks a user won’t even see an error, they’ll just have a short delay before getting a response from a working server.
Using Graphics Card Memory as Swap (via) Interesting idea: “Graphic cards contain a lot of very fast RAM, typically between 64 and 512 MB. With Linux, it’s possible to use it as swap space, or even as RAM disk.”
I loathe [hardware load balancers]. They’re expensive, restrictive, slow, and generally cause you a lot more pain and suffering than they’re worth. At my last job, one of my projects was to convert most of one of our existing clusters from a load-balancing appliance to use keepalived. Why would we do this? Because the $100k worth of appliance wasn’t capable of doing the job that $15k worth of commodity hardware and an installation of keepalived were handling with ease.
apache.org incident report for 8/28/2009. Various apache.org sites were down for a while last week—here the Apache Infrastructure Team provide a detailed description of what happened (a security breach on a minor server, which provided non-priveleged SSH access to mirror servers via an SSH key used for backups) and how they are responding. Useful for neophyte sysadmins like myself.
Installing Django, Solr, Varnish and Supervisord with Buildout. Useful, detailed instructions... but I still think this stuff is Way Too Difficult at the moment. I’m a big fan of the idea of sites that are assembled from multiple smaller web services talking HTTP to each other, but ensuring all the moving parts stay running is massively more painful than just running Apache and MySQL.
The Django and Ubuntu Intrepid Almanac. Will Larson’s impressively comprehensive guide to configuring and securing an Ubuntu VPS from scratch to run Django, using PostgreSQL and Apache/mod_wsgi behind nginx.
Infrastructure for Modern Web Sites. Leonard’s thoughts on what the next generation of web frameworks should aim to provide.
2008
backup_to_s3.py. I wrote Yet Another S3 backup script today. It’s a thin wrapper about boto that doesn’t do anything particularly impressive, but it fits my brain.
We’re all ops people now. Edd’s experience reflects my own: the kind of systems I’m building these days involve way more than just development, they often involve significant sysadmin type skills as well. Desperately need to get better at that stuff.
god—process and task monitoring done right. I have a long running animosity towards every process monitoring tool currently in existence; I’ll have to put this one through its paces and see if it sucks less.
2007
System Administrator Appreciation Day. The last Friday in July is Annual System Administrator Appreciation day. Thank (or hug) your sysadmin today!
Understanding your (Red Hat Enterprise Linux) daemons. A handy bestiary.