Simon Willison’s Weblog


Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions

16th January 2024

I use cookiecutter to start almost all of my Python projects. It helps me quickly generate a skeleton of a project with my preferred directory structure and configured tools.

I made some major upgrades to my python-lib cookiecutter template today. Here’s what it can now do to help you get started with a new Python library:

  • Create a pyproject.toml file configured for use with setuptools. In my opinion this is the pattern with the current lowest learning curve—I wrote about that in detail in this TIL.
  • Add a skeleton README and an Apache 2.0 LICENSE file.
  • Create your_package/ for your code to go in.
  • Create tests/ with a skeleton test.
  • Include pytest as a test dependency.
  • Configure GitHub Actions with two workflows in .github/workflows—one for running the tests against Python 3.8 through 3.12, and one for publishing releases of your package to PyPI.

The changes I made today are that I switched from to pyproject.toml, and I made a big improvement to how the publishing workflow authenticates with PyPI.

Publishing to PyPI with Trusted Publishing

My previous version of this template required you to jump through quite a few hoops to get PyPI publishing to work. You needed to create a PyPI token that could publish a new package, then paste that token into a GitHub Actions secret, then publish the package, and then disable that token and create a new one dedicated to just updating this package in the future.

The new version is much simpler, thanks to PyPI’s relatively new Trusted Publishers mechanism.

To publish a new package, you need to sign into PyPI and create a new “pending publisher”. Effectively you tell PyPI "My GitHub repository myname/name-of-repo should be allowed to publish packages with the name name-of-package".

Here’s that form for my brand new datasette-test library, the first library I published using this updated template:

Screenshot of the create pending publisher form on PyPI. PyPI Project Name is set to datasette-test. Owner is set to datasette. Repository name is datasette-test. Workflow name is publish.yml. Environment name is release.

Then create a release on GitHub, with a name that matches the version number from your pyproject.toml. Everything else should Just Work.

I wrote more about Trusted Publishing in this TIL.

Creating a package using a GitHub repository template

The most time consuming part of this project was getting my GitHub repository template to work properly.

There are two ways to use my cookiecutter template. You can use the cookiecutter command-line tool like this:

pipx install cookiecutter
cookiecutter gh:simonw/python-lib
# Answer a few questions here

But a more fun and convenient option is to use my GitHub repository template, simonw/python-lib-template-repository.

This lets you fill in a form on GitHub to create a new repository which will then execute the cookiecutter template for you and update itself with the result.

Create a new repository form. I'm using the python-lib-template-repository template, and it asks for my repository name (my-new-python-library) and description.

You can see an example of a repository created using this template at datasette/datasette-test.

Adding it all together

There are quite a lot of moving parts under the scenes here, but the end result is that anyone can now create a Python library with test coverage, GitHub CI and release automation by filling in a couple of forms and clicking some buttons.

For more details on how this all works, and how it’s evolved over time:

This is Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions by Simon Willison, posted on 16th January 2024.

Part of series My open source process

  1. The Perfect Commit - Oct. 29, 2022, 8:41 p.m.
  2. Coping strategies for the serial project hoarder - Nov. 26, 2022, 3:47 p.m.
  3. Things I've learned about building CLI tools in Python - Sept. 30, 2023, 12:12 a.m.
  4. Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions - Jan. 16, 2024, 9:59 p.m.

Next: Talking about Open Source LLMs on Oxide and Friends

Previous: What I should have said about the term Artificial Intelligence