Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions
16th January 2024
I use cookiecutter to start almost all of my Python projects. It helps me quickly generate a skeleton of a project with my preferred directory structure and configured tools.
I made some major upgrades to my python-lib cookiecutter template today. Here’s what it can now do to help you get started with a new Python library:
- Create a
pyproject.tomlfile configured for use with
setuptools. In my opinion this is the pattern with the current lowest learning curve—I wrote about that in detail in this TIL.
- Add a skeleton
READMEand an Apache 2.0
your_package/__init__.pyfor your code to go in.
tests/test_your_package.pywith a skeleton test.
pytestas a test dependency.
- Configure GitHub Actions with two workflows in
.github/workflows—one for running the tests against Python 3.8 through 3.12, and one for publishing releases of your package to PyPI.
The changes I made today are that I switched from
pyproject.toml, and I made a big improvement to how the publishing workflow authenticates with PyPI.
Publishing to PyPI with Trusted Publishing
My previous version of this template required you to jump through quite a few hoops to get PyPI publishing to work. You needed to create a PyPI token that could publish a new package, then paste that token into a GitHub Actions secret, then publish the package, and then disable that token and create a new one dedicated to just updating this package in the future.
The new version is much simpler, thanks to PyPI’s relatively new Trusted Publishers mechanism.
To publish a new package, you need to sign into PyPI and create a new “pending publisher”. Effectively you tell PyPI "My GitHub repository
myname/name-of-repo should be allowed to publish packages with the name
Here’s that form for my brand new datasette-test library, the first library I published using this updated template:
Then create a release on GitHub, with a name that matches the version number from your
pyproject.toml. Everything else should Just Work.
Creating a package using a GitHub repository template
The most time consuming part of this project was getting my GitHub repository template to work properly.
There are two ways to use my cookiecutter template. You can use the cookiecutter command-line tool like this:
pipx install cookiecutter
# Answer a few questions here
But a more fun and convenient option is to use my GitHub repository template, simonw/python-lib-template-repository.
This lets you fill in a form on GitHub to create a new repository which will then execute the cookiecutter template for you and update itself with the result.
You can see an example of a repository created using this template at datasette/datasette-test.
Adding it all together
There are quite a lot of moving parts under the scenes here, but the end result is that anyone can now create a Python library with test coverage, GitHub CI and release automation by filling in a couple of forms and clicking some buttons.
For more details on how this all works, and how it’s evolved over time:
- A cookiecutter template for writing Datasette plugins from June 2020 describes my first experiments with cookiecutter
- Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions from August 2021 describes my earliest attempts at using GitHub repository templates for this
How to build, test and publish an open source Python library is a ten minute talk I gave at PyGotham in November 2021. It describes
setup.pyin detail, which is no longer my preferred approach.
More recent articles
- Interesting ideas in Observable Framework - 3rd March 2024
- Weeknotes: Getting ready for NICAR - 27th February 2024
- The killer app of Gemini Pro 1.5 is video - 21st February 2024
- Weeknotes: a Datasette release, an LLM release and a bunch of new plugins - 9th February 2024
- LLM 0.13: The annotated release notes - 26th January 2024
- Weeknotes: datasette-test, datasette-build, PSF board retreat - 21st January 2024
- Talking about Open Source LLMs on Oxide and Friends - 17th January 2024
- What I should have said about the term Artificial Intelligence - 9th January 2024