Simon Willison’s Weblog

Python Developers Survey 2023 Results (via) The seventh annual Python survey is out. Here are the things that caught my eye or that I found surprising:

25% of survey respondents had been programming in Python for less than a year, and 33% had less than a year of professional experience.

37% of Python developers reported contributing to open-source projects last year - a new question for the survey. This is delightfully high!

6% of users are still using Python 2. The survey notes:

Almost half of Python 2 holdouts are under 21 years old and a third are students. Perhaps courses are still using Python 2?

In web frameworks, Flask and Django neck and neck at 33% each, but FastAPI is a close third at 29%! Starlette is at 6%, but that's an under-count because it's the basis for FastAPI.

The most popular library in "other framework and libraries" was BeautifulSoup with 31%, then Pillow 28%, then OpenCV-Python at 22% (wow!) and Pydantic at 22%. Tkinter had 17%. These numbers are all a surprise to me.

pytest scores 52% for unit testing, unittest from the standard library just 25%. I'm glad to see pytest so widely used, it's my favourite testing tool across any programming language.

The top cloud providers are AWS, then Google Cloud Platform, then Azure... but PythonAnywhere (11%) took fourth place just ahead of DigitalOcean (10%). And Alibaba Cloud is a new entrant in sixth place (after Heroku) with 4%. Heroku's ending of its free plan dropped them from 14% in 2021 to 7% now.

Linux and Windows equal at 55%, macOS is at 29%. This was one of many multiple-choice questions that could add up to more than 100%.

In databases, SQLite usage was trending down - 38% in 2021 to 34% for 2023, but still in second place behind PostgreSQL, stable at 43%.

The survey incorporates quotes from different Python experts responding to the numbers, it's worth reading through the whole thing.

# 3rd September 2024, 2:47 am / open-source, postgresql, python, sqlite, surveys, pytest, psf, pydantic

Glue code to quickly copy data from one Postgres table to another (via) The Python script that Retool used to migrate 4TB of data between two PostgreSQL databases. I find the structure of this script really interesting—it uses Python to spin up a queue full of ID ranges to be transferred and then starts some threads, but then each thread shells out to a command that runs “psql COPY (SELECT ...) TO STDOUT” and pipes the result to “psql COPY xxx FROM STDIN”. Clearly this works really well (“saturate the database’s hardware capacity” according to a comment on HN), and neatly sidesteps any issues with Python’s GIL.

# 19th April 2022, 4:57 pm / postgresql, python

mycli. Really neat auto-complete enabled MySQL terminal client, built using the excellent python-prompt-toolkit. Has a sister-project for PostgreSQL called pgcli.

# 11th June 2018, 7:08 pm / mysql, postgresql, python

MapOSMatic. Clever service built on top of OpenStreetMap, which renders double sided city maps with a map and grid on one size and an A-Z street name index on the other. Runs on top of Mapnik, PostGIS and Cairo, with a few thousand additional lines of Python and Django.

# 11th July 2010, 12:15 pm / cairo, django, mapping, openstreetmap, postgis, postgresql, python, recovered, maposmatic

Appending the request URL to SQL statements in Django. A clever frame-walking monkey-patch which pulls the most recent HttpRequest object out of the Python stack and adds the current request.path to each SQL query as an SQL comment, so you can see it in debugging tools such as slow query logs and the PostgreSQL “select * from pg_stat_activity” query.

# 2nd June 2010, 9:09 am / chris-lamb, debugging, django, orm, postgresql, python, sql, recovered, monkeypatch

jacobian’s django-deployment-workshop. Notes and resources from Jacob’s 3 hour Django deployment workshop at PyCon, including example configuration files for Apache2 + mod_wsgi, nginx, PostgreSQL and pgpool.

# 19th February 2010, 2:28 pm / apache, deployment, django, jacob-kaplan-moss, modwsgi, nginx, pgpool, postgresql, pycon, python, sysadmin

Django-Jython 1.0.0 released! Now with database backends for PostgreSQL, Oracle and MySQL. The next release (planned for next month) should provide full compatibility with Django 1.1—the current release has 1.1 support for PostgreSQL but only 1.0 support for the other two databases.

# 9th November 2009, 1:53 pm / django, jython, leosoto, mysql, oracle, postgresql, python

Install Django, GeoDjango, PostgreSQL and PostGIS on OSX Leopard. This tutorial worked perfectly for me.

# 24th July 2009, 11:47 am / django, geodjango, osx, postgis, postgresql, python

EveryBlock source code released. EveryBlock’s Knight Foundation grant required them to release the source code after two years, under the GPL. Lots of neat Django / PostgreSQL / GIS tricks to be found within.

# 1st July 2009, 8:01 pm / django, everyblock, gis, gpl, open-source, postgresql, python

Represent. Andrei Scheinkman and Derek Willis describe how they built the NYTimes Represent feature using GeoDjango and PostGIS.

# 29th December 2008, 10:10 pm / andrei-scheinkman, derek-willis, django, geodjango, gis, new-york-times, postgis, postgresql, python

bpgsql. Barry Pederson’s pure Python PostgreSQL client library now ships with a Django backend.

# 23rd September 2008, 11:42 am / barry-pederson, bpgsql, django, postgresql, python

Persistent Django on Amazon EC2 and EBS—the easy way. Useful tutorial on getting Django up and running on EC2 with EBS for a persistent PostgreSQL database.

# 21st August 2008, 9:32 pm / amazon, aws, django, ebs, ec2, postgresql, python

Django Unit Tests and Transactions. If you’re using a transactional database engine (MySQL with InnoDB, Postgres or SQLite) you can speed things up by running each of your unit tests inside a transaction and rolling back in tearDown().

# 7th July 2008, 2:14 pm / django, innodb, mysql, postgresql, python, sqlite, transactions, unittest, unittesting

ThingDB. Another extensible key/value pair data store, constructed for the Open Library based on Aaron Swartz’s Infogami technology.

# 17th July 2007, 10:21 am / aaron-swartz, infogami, openlibrary, postgresql, python, thingdb

pg8000 v1.02. The pure Python PostgreSQL library now supports DB-API 2.0 (and SSL too). That didn’t take long!

# 13th March 2007, 9:18 pm / dbapi, pg8000, postgresql, python

pg8000 (via) A pure-Python interface to PostgreSQL, using the PostgreSQL network protocol directly. Doesn’t (yet) support DB-API 2.0, but that’s promised in a future release.

# 9th March 2007, 7:35 pm / pg8000, postgresql, python