Simon Willison's Weblog: psf

Anthropic invests $1.5 million in the Python Software Foundation and open source security

2026-01-13T23:58:17+00:00

Anthropic invests $1.5 million in the Python Software Foundation and open source security

This is outstanding news, especially given our decision to withdraw from that NSF grant application back in October.

We are thrilled to announce that Anthropic has entered into a two-year partnership with the Python Software Foundation (PSF) to contribute a landmark total of $1.5 million to support the foundation’s work, with an emphasis on Python ecosystem security. This investment will enable the PSF to make crucial security advances to CPython and the Python Package Index (PyPI) benefiting all users, and it will also sustain the foundation’s core work supporting the Python language, ecosystem, and global community.

Note that while security is a focus these funds will also support other aspects of the PSF's work:

Anthropic’s support will also go towards the PSF’s core work, including the Developer in Residence program driving contributions to CPython, community support through grants and other programs, running core infrastructure such as PyPI, and more.

Tags: open-source, python, ai, psf, anthropic

PyCon US 2026 call for proposals is now open

2025-11-02T19:22:46+00:00

PyCon US 2026 call for proposals is now open

PyCon US is coming to the US west coast! 2026 and 2027 will both be held in Long Beach, California - the 2026 conference is set for May 13th-19th next year.

The call for proposals just opened. Since we'll be in LA County I'd love to see talks about Python in the entertainment industry - if you know someone who could present on that topic please make sure they know about the CFP!

The deadline for submissions is December 19th 2025. There are two new tracks this year:

PyCon US is introducing two dedicated Talk tracks to the schedule this year, "The Future of AI with Python" and "Trailblazing Python Security". For more information and how to submit your proposal, visit this page.

Now is also a great time to consider sponsoring PyCon - here's the sponsorship prospectus.

Via @pycon.us

Tags: call-for-proposals, conferences, pycon, python, psf

The PSF has withdrawn a $1.5 million proposal to US government grant program

2025-10-27T20:32:07+00:00

The PSF has withdrawn a $1.5 million proposal to US government grant program

The Python Software Foundation was recently "recommended for funding" (NSF terminology) for a $1.5m grant from the US government National Science Foundation to help improve the security of the Python software ecosystem, after an grant application process lead by Seth Larson and Loren Crary.

The PSF's annual budget is less than $6m so this is a meaningful amount of money for the organization!

We were forced to withdraw our application and turn down the funding, thanks to new language that was added to the agreement requiring us to affirm that we "do not, and will not during the term of this financial assistance award, operate any programs that advance or promote DEI, or discriminatory equity ideology in violation of Federal anti-discrimination laws."

Our legal advisors confirmed that this would not just apply to security work covered by the grant - this would apply to all of the PSF's activities.

This was not an option for us. Here's the mission of the PSF:

The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers.

If we accepted and spent the money despite this term, there was a very real risk that the money could be clawed back later. That represents an existential risk for the foundation since we would have already spent the money!

I was one of the board members who voted to reject this funding - a unanimous but tough decision. I’m proud to serve on a board that can make difficult decisions like this.

If you'd like to sponsor the PSF you can find out more on our site. I'd love to see a few more of the large AI labs show up on our top-tier visionary sponsors list.

Tags: open-source, python, psf

Python 3.14

2025-10-08T04:10:06+00:00

Python 3.14

This year's major Python version, Python 3.14, just made its first stable release!

As usual the what's new in Python 3.14 document is the best place to get familiar with the new release:

The biggest changes include template string literals, deferred evaluation of annotations, and support for subinterpreters in the standard library.

The library changes include significantly improved capabilities for introspection in asyncio, support for Zstandard via a new compression.zstd module, syntax highlighting in the REPL, as well as the usual deprecations and removals, and improvements in user-friendliness and correctness.

Subinterpreters look particularly interesting as a way to use multiple CPU cores to run Python code despite the continued existence of the GIL. If you're feeling brave and your dependencies cooperate you can also use the free-threaded build of Python 3.14 - now officially supported - to skip the GIL entirely.

A new major Python release means an older release hits the end of its support lifecycle - in this case that's Python 3.9. If you maintain open source libraries that target every supported Python versions (as I do) this means features introduced in Python 3.10 can now be depended on! What's new in Python 3.10 lists those - I'm most excited by structured pattern matching (the match/case statement) and the union type operator, allowing int | float | None as a type annotation in place of Optional[Union[int, float]].

If you use uv you can grab a copy of 3.14 using:

uv self update
uv python upgrade 3.14
uvx python@3.14

Or for free-threaded Python 3.1;:

uvx python@3.14t

The uv team wrote about their Python 3.14 highlights in their announcement of Python 3.14's availability via uv.

The GitHub Actions setup-python action includes Python 3.14 now too, so the following YAML snippet in will run tests on all currently supported versions:

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/setup-python@v6
  with:
    python-version: ${{ matrix.python-version }}

Full example here for one of my many Datasette plugin repos.

Tags: gil, open-source, python, github-actions, psf, uv

Announcing the 2025 PSF Board Election Results!

2025-09-16T20:39:41+00:00

Announcing the 2025 PSF Board Election Results!

I'm happy to share that I've been re-elected for second term on the board of directors of the Python Software Foundation.

Jannis Leidel was also re-elected and Abigail Dogbe and Sheena O’Connell will be joining the board for the first time.

Tags: python, psf

The 2025 PSF Board Election is Open!

2025-09-09T10:13:33+00:00

The 2025 PSF Board Election is Open!

The Python Software Foundation's annual board member election is taking place right now, with votes (from previously affirmed voting members) accepted from September 2nd, 2:00 pm UTC through Tuesday, September 16th, 2:00 pm UTC.

I've served on the board since 2022 and I'm running for a second term. Here's the opening section of my nomination statement.

Hi, I'm Simon Willison. I've been a board member of the Python Software Foundation since 2022 and I'm running for re-election in 2025.

Last year I wrote a detailed article about Things I’ve learned serving on the board of the Python Software Foundation. I hope to continue learning and sharing what I've learned for a second three-year term.

One of my goals for a second term is to help deepen the relationship between the AI research world and the Python Software Foundation. There is an enormous amount of value being created in the AI space using Python and I would like to see more of that value flow back into the rest of the Python ecosystem.

I see the Python Package Index (PyPI) as one of the most impactful projects of the Python Software Foundation and plan to continue to advocate for further investment in the PyPI team and infrastructure.

As a California resident I'm excited to see PyCon return to the West Coast, and I'm looking forward to getting involved in helping make PyCon 2026 and 2027 in Long Beach, California as successful as possible.

I'm delighted to have been endorsed this year by Al Sweigart, Loren Crary and Christopher Neugebauer. If you are a voting member I hope I have earned your vote this year.

You can watch video introductions from several of the other nominees in this six minute YouTube video and this playlist.

Tags: python, psf

2025 Python Packaging Ecosystem Survey

2025-05-18T11:50:06+00:00

2025 Python Packaging Ecosystem Survey

If you make use of Python packaging tools (pip, Anaconda, uv, dozens of others) and have opinions please spend a few minutes with this year's packaging survey. This one was "Co-authored by 30+ of your favorite Python Ecosystem projects, organizations and companies."

Tags: packaging, pip, python, surveys, psf

PyPI now supports project archival

2025-01-30T16:46:34+00:00

PyPI now supports project archival

Neat new PyPI feature, similar to GitHub's archiving repositories feature. You can now mark a PyPI project as "archived", making it clear that no new releases are planned (though you can switch back out of that mode later if you need to).

I like the sound of these future plans around this topic:

Project archival is the first step in a larger project, aimed at improving the lifecycle of projects on PyPI. That project includes evaluating additional project statuses (things like "deprecated" and "unmaintained"), as well as changes to PyPI's public APIs that will enable clients to retrieve and act on project status information. You can track our progress on these fronts by following along with warehouse#16844!

Tags: pypi, python, psf

PyPI now supports digital attestations

2024-11-14T19:56:49+00:00

PyPI now supports digital attestations

Dustin Ingram:

PyPI package maintainers can now publish signed digital attestations when publishing, in order to further increase trust in the supply-chain security of their projects. Additionally, a new API is available for consumers and installers to verify published attestations.

This has been in the works for a while, and is another component of PyPI's approach to supply chain security for Python packaging - see PEP 740 – Index support for digital attestations for all of the underlying details.

A key problem this solves is cryptographically linking packages published on PyPI to the exact source code that was used to build those packages. In the absence of this feature there are no guarantees that the .tar.gz or .whl file you download from PyPI hasn't been tampered with (to add malware, for example) in a way that's not visible in the published source code.

These new attestations provide a mechanism for proving that a known, trustworthy build system was used to generate and publish the package, starting with its source code on GitHub.

The good news is that if you're using the PyPI Trusted Publishers mechanism in GitHub Actions to publish packages, you're already using this new system. I wrote about that system in January: Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions - and hundreds of my own PyPI packages are already using that system, thanks to my various cookiecutter templates.

Trail of Bits helped build this feature, and provide extra background about it on their own blog in Attestations: A new generation of signatures on PyPI:

As of October 29, attestations are the default for anyone using Trusted Publishing via the PyPA publishing action for GitHub. That means roughly 20,000 packages can now attest to their provenance by default, with no changes needed.

They also built Are we PEP 740 yet? (key implementation here) to track the rollout of attestations across the 360 most downloaded packages from PyPI. It works by hitting URLs such as https://pypi.org/simple/pydantic/ with a Accept: application/vnd.pypi.simple.v1+json header - here's the JSON that returns.

I published an alpha package using Trusted Publishers last night and the files for that release are showing the new provenance information already:

Which links to this Sigstore log entry with more details, including the Git hash that was used to build the package:

Sigstore is a transparency log maintained by Open Source Security Foundation (OpenSSF), a sub-project of the Linux Foundation.

Via Hacker News

Tags: github, packaging, pypi, python, github-actions, psf, supply-chain, dustin-ingram

Perks of Being a Python Core Developer

2024-10-12T16:34:16+00:00

Perks of Being a Python Core Developer

Mariatta Wijaya provides a detailed breakdown of the exact capabilities and privileges that are granted to Python core developers - including commit access to the Python main, the ability to write or sponsor PEPs, the ability to vote on new core developers and for the steering council election and financial support from the PSF for travel expenses related to PyCon and core development sprints.

Not to be under-estimated is that you also gain respect:

Everyone’s always looking for ways to stand out in resumes, right? So do I. I’ve been an engineer for longer than I’ve been a core developer, and I do notice that having the extra title like open source maintainer and public speaker really make a difference. As a woman, as someone with foreign last name that nobody knows how to pronounce, as someone who looks foreign, and speaks in a foreign accent, having these extra “credentials” helped me be seen as more or less equal compared to other people.

Via lobste.rs

Tags: open-source, python, psf

Weeknotes: Three podcasts, two trips and a new plugin system

2024-09-30T17:43:22+00:00

I fell behind a bit on my weeknotes. Here's most of what I've been doing in September.

Lisbon, Portugal and Durham, North Carolina

I had two trips this month. The first was a short visit to Lisbon, Portugal for the Python Software Foundation's annual board retreat. This inspired me to write about Things I've learned serving on the board of the Python Software Foundation.

The second was to Durham, North Carolina for DjangoCon US 2024. I wrote about that one in Themes from DjangoCon US 2024.

My talk at DjangoCon was about plugin systems, and in a classic example of conference-driven development I ended up writing and releasing a new plugin system for Django in preparation for that talk. I introduced that in DJP: A plugin system for Django.

Podcasts

I haven't been a podcast guest since January, and then three came along at once! All three appearences involved LLMs in some way but I don't think there was a huge amount of overlap in terms of what I actually said.

I went on The Software Misadventures Podcast to talk about my career to-date.
My appearance on TWIML dug into ways in which I use Claude and ChatGPT to help me write code.
I was the guest for the inaugral episode of Gergely Orosz's Pragmatic Engineer Podcast, which ended up touching on a whole array of different topics relevant to modern software engineering, from the importance of open source to the impact AI tools are likely to have on our industry.

Gergely has been sharing neat edited snippets from our conversation on Twitter. Here's one on RAG and another about how open source has been the the biggest productivity boost of my career.

On the blog

NotebookLM's automatically generated podcasts are surprisingly effective - Sept. 29, 2024
Themes from DjangoCon US 2024 - Sept. 27, 2024
DJP: A plugin system for Django - Sept. 25, 2024
Notes on using LLMs for code - Sept. 20, 2024
Things I've learned serving on the board of the Python Software Foundation - Sept. 18, 2024
Notes on OpenAI's new o1 chain-of-thought models - Sept. 12, 2024
Notes from my appearance on the Software Misadventures Podcast - Sept. 10, 2024
Teresa T is name of the whale in Pillar Point Harbor near Half Moon Bay - Sept. 8, 2024

Museums

The Vincent and Ethel Simonetti Historic Tuba Collection

Releases

shot-scraper 1.5 - 2024-09-27
A command-line utility for taking automated screenshots of websites
django-plugin-datasette 0.2 - 2024-09-26
Django plugin to run Datasette inside of Django
djp 0.3.1 - 2024-09-26
A plugin system for Django
llm-gemini 0.1a5 - 2024-09-24
LLM plugin to access Google's Gemini family of models
django-plugin-blog 0.1.1 - 2024-09-24
A blog for Django as a DJP plugin.
django-plugin-database-url 0.1 - 2024-09-24
Django plugin for reading the DATABASE_URL environment variable
django-plugin-django-header 0.1.1 - 2024-09-23
Add a Django-Compositions HTTP header to a Django app
llm-jina-api 0.1a0 - 2024-09-20
Access Jina AI embeddings via their API
llm 0.16 - 2024-09-12
Access large language models from the command-line
datasette-acl 0.4a4 - 2024-09-10
Advanced permission management for Datasette
llm-cmd 0.2a0 - 2024-09-09
Use LLM to generate and execute commands in your shell
files-to-prompt 0.3 - 2024-09-09
Concatenate a directory full of files into a single prompt for use with LLMs
json-flatten 0.3.1 - 2024-09-07
Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object
csv-diff 1.2 - 2024-09-06
Python CLI tool and library for diffing CSV and JSON files
datasette 1.0a16 - 2024-09-06
An open source multi-tool for exploring and publishing data
datasette-search-all 1.1.4 - 2024-09-06
Datasette plugin for searching all searchable tables at once

TILs

How streaming LLM APIs work - 2024-09-21

Tags: django, podcasts, weeknotes, psf, llms, djp

Things I've Learned Serving on the Board of The Perl Foundation

2024-09-24T01:42:14+00:00

Things I've Learned Serving on the Board of The Perl Foundation

My post about the PSF board inspired Perl Foundation secretary Makoto Nozaki to publish similar notes about how TPF (also known since 2019 as TPRF, for The Perl and Raku Foundation) operates.

Seeing this level of explanation about other open source foundations is fascinating. I’d love to see more of these.

Along those lines, I found the 2024 Financial Report from the Zig foundation really interesting too.

Via @ heismakoto

Tags: open-source, perl, zig, psf

Things I've learned serving on the board of the Python Software Foundation

2024-09-18T14:15:37+00:00

Two years ago I was elected to the board of directors for the Python Software Foundation - the PSF. I recently returned from the annual PSF board retreat (this one was in Lisbon, Portugal) and this feels like a good opportunity to write up some of the things I've learned along the way.

What is the PSF?
The PSF employs staff
A lot of this is about money
The PSF does not directly develop Python itself
PyPI - the Python Package Index
PyCon is a key commitment
Other PSF activities
Work Groups
Acting as a fiscal sponsor
Life as a board member
The kinds of things the board talks about
Want to know more?

What is the PSF?

The PSF is a US 501(c)(3) non-profit organization with the following mission:

The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers.

That mission definition is really important. Board members and paid staff come and go, but the mission remains constant - it's the single most critical resource to help make decisions about whether the PSF should be investing time, money and effort into an activity or not.

The board's 501(c)(3) status is predicated on following the full expanded mission statement. When our finances get audited (we conduct an annual "friendly audit", which is considered best practice for organizations at our size), the auditors need to be able to confirm that we've been supporting that mission through our management of the tax-exempt funds that have been entrusted to us.

This auditability is an interesting aspect of how 501(c)(3) organizations work, because it means you can donate funds to them and know that the IRS will ostensibly be ensuring that the money is spent in a way that supports their stated mission.

Board members have fiduciary responsibility for the PSF. A good explanation of this can be found here on BoardSource, which also has other useful resources for understanding the roles and responsibilities of non-profit board members.

(Developing at least a loose intuition for US tax law around non-profits is one of the many surprising things that are necessary to be an effective board member.)

The PSF employs staff

The PSF currently employs 12 full-time staff members. Members of the board do not directly manage the activities of the staff - in fact board members telling staff what to do is highly inappropriate.

Instead, the board is responsible for hiring an Executive Director - currently Deb Nicholson - who is then responsible for hiring and managing (directly or indirectly) those other staff members. The board is responsible for evaluating the Executive Director's performance.

I joined the board shortly after Deb was hired, so I have not personally been part of a board hiring committee for a new Executive Director.

While paid staff support and enact many of the activities of the PSF, the foundation is fundamentally a volunteer-based organization. Many PSF activities are carried out by these volunteers, in particular via Work Groups.

A lot of this is about money

A grossly simplified way to think about the PSF is that it's a bucket of money that is raised from sponsors and the Python community (via donations and membership fees), and then spent to support the community and the language in different ways.

The PSF spends money on staff, on grants to Python-related causes and on infrastructure and activities that support Python development and the Python community itself.

You can see how that money has been spent in the 2023 Annual Impact Report. The PSF had $4,356,000 revenue for that year and spent $4,508,000 - running a small loss, but not a concerning one given our assets from previous years.

The most significant categories of expenditure in 2023 were PyCon US ($1,800,000), our Grants program ($677,000), Infrastructure (including PyPI) ($286,000) and our Fiscal Sponsorees ($204,000) - I'll describe these in more detail below.

The PSF does not directly develop Python itself

This is an important detail to understand. The PSF is responsible for protecting and supporting the Python language and community, but development of CPython itself is not directly managed by the PSF.

Python development is handled by the Python core team, who are governed by the 5-person Python Steering Council. The Steering Council is elected by the core team. The process for becoming a core developer is described here.

How this all works is defined by PEP 13: Python Language Governance (and several subsequent PEPs). This structure was created - with much discussion - after Guido van Rossum stepped down from his role as Python BDFL in 2018.

The PSF's executive director maintains close ties with the steering council, meeting with them 2-3 times a month. The PSF provides financial support for some Python core activities, such as infrastructure used for Python development and sponsoring travel to and logistics for core Python sprints.

More recently, the PSF has started employing Developers in Residence to directly support the work of both the core Python team and initiatives such as the Python Package Index.

PyPI - the Python Package Index

One of the most consequential projects directly managed by the PSF is PyPI, the Python Package Index. This is the system that enables pip install name-of-package to do its thing.

Having PyPI managed by a non-profit that answers directly to the community it serves is a very good thing.

PyPI's numbers are staggering. Today there are 570,000 projects consisting of 12,035,133 files, serving 1.9 billion downloads a day (that number from PyPI Stats). Bandwidth for these downloads is donated by Fastly, a PSF Visionary Sponsor who recently signed a five year agreement to continue this service.

(This was a big deal - prior to that agreement there was concern over what would happen if Fastly ever decided to end that sponsorship.)

PyCon is a key commitment

The annual US Python Conference - PyCon US - is a big part of the PSF's annual activities and operations. With over 3,000 attendees each year (and a $1.8m budget for 2023) running that conference represents a full-time job for several PSF staff members.

In the past PyCon US has also been responsible for the majority of the PSF's operating income. This is no longer true today - in fact it ran at a slight loss this year. This is not a big problem: the PSF's funding has diversified, and the importance of PyCon US to the Python community is such that the PSF is happy to lose money running the event if necessary.

Other PSF activities

Many of these are detailed in the full mission statement.

Operating python.org and making Python available to download. It's interesting to note that Python is distributed through many alternative routes that are not managed by the PSF - through Linux packaging systems like Ubuntu, Debian and Red Hat, via tools like Docker or Homebrew, by companies such as Anaconda or through newer channels such as uv.
Owning and protecting the Python trademarks and the Python intellectual property rights under the (OSI compliant) Python license. This is one of the fundamental reasons for the organization to exist, but thankfully is one of the smaller commitments in terms of cost and staff time.
Running the annual PyCon US conference.
Operating the Python Packaging Index. Fastly provide the CDN, but the PSF still takes on the task of developing and operating the core PyPI web application and the large amounts of moderation and user support that entails.
Supporting infrastructure used for core Python development, and logistics for core Python sprints.
Issuing grants to Python community efforts.
Caring for fiscal sponsorees.
Supporting the work of PSF Work Groups.

Work Groups

A number of PSF initiatives take place in the form of Work Groups, listed here. Work Groups are teams of volunteers from the community who collaborate on projects relevant to the PSF's mission.

Each Work Group sets its own cadence and ways of working. Some groups have decisions delegated to them by the board - for example the Grants Work Group for reviewing grant proposals and the Code of Conduct Work Group for enforcing Code of Conduct activity. Others coordinate technical projects such as the Infrastructure Working Group, who manage and make decisions on various pieces of technical infrastructure relevant to Python and the PSF.

Work Groups are formed by a board vote, with a designated charter. Most recently the board approved a charter for a new User Success Work Group, focusing on things like improving the new Python user onboarding experience.

This is another term I was unfamiliar with before joining the board: the idea of a fiscal sponsor, which is a key role played by the PSF.

Running a non-profit organization is decidedly non-trivial: you need a legal structure, a bank account, accounting, governance, the ability to handle audits - there's a whole lot of complexity behind the scenes.

Looking to run an annual community conference? You'll need a bank account, and an entity that can sign agreements with venues and vendors.

Want to accept donations to support work you are doing? Again, you need an entity, and a bank account, and some form of legal structure that ensures your donors can confidently trust you with their money.

Instead of forming a whole new non-profit for this, you can instead find an existing non-profit that is willing to be your "fiscal sponsor". They'll handle the accounting and various other legal aspects, which allows you to invest your efforts in the distinctive work that you are trying to do.

The PSF acts as a fiscal sponsor for a number of different organizations - 20 as-of the 2023 report - including PyLadies, Twisted, Pallets, Jazzband, PyCascades and North Bay Python. The PSF's accounting team invest a great deal of effort in making all of this work.

The PSF generally takes a 10% cut of donations to its fiscal sponsorees. This doesn't actually cover the full staffing cost of servicing these organizations, but this all still makes financial sense in terms of the PSF's mission to support the global Python community.

Life as a board member

There are 12 board members. Elections are held every year after PyCon US, voted on by the PSF membership - by both paid members and members who have earned voting rights through being acknowledged as PSF fellows.

Board members are elected for three year terms. Since 1-3 new board members are likely to join annually, these terms ensure there is overlap which helps maintain institutional knowledge about how the board operates.

The board's activities are governed by the PSF Bylaws, and there is a documented process for modifying them (see ARTICLE XI).

We have board members from all over the world. This is extremely important, because the PSF is responsible for the health and growth of the global Python community. A perennial concern is how to ensure that board candidates are nominated from around the world, in order to maintain that critical global focus.

The board meets once a month over Zoom, has ongoing conversations via Slack and meets in-person twice a year: once at PyCon US and once at a "retreat" in a different global city, selected to try and minimize the total amount of travel needed to get all of our global board members together in the same place.

Our most recent retreat was in Lisbon, Portugal. The retreat before that was in Malmö in Sweden.

I considered using an analogy that describes each board member as 1/12th of the "brain" of the PSF, but that doesn't hold up: the paid, full-time staff of the PSF make an enormous number of decisions that impact how the PSF works.

Instead, the board acts to set strategy, represent the global community and help ensure that the PSF's activities are staying true to that mission. Like I said earlier, the mission definition really is critical. I admit that in the past I've been a bit cynical about the importance of mission statements: being a board member of a 501(c)(3) non-profit has entirely cured me of that skepticism!

Board members can also sit on board committees, of which there are currently four: the Executive Committee, Finance Committee, PyCon US Committee and Membership Committee. These mainly exist so that relevant decisions can be delegated to them, helping reduce the topics that must be considered by the full board in our monthly meetings.

The kinds of things the board talks about

Our Lisbon retreat involved two full 9-hour days of discussion, plus social breakfasts, lunches and dinners. It was an intense workload.

I won't even attempt to do it justice here, but I'll use a couple of topics to illustrate the kind of things we think about on the board.

The first is our grants strategy. The PSF financially sponsors Python community events around the world. In the past this grants program has suffered from low visibility and, to simplify, we've felt that we weren't giving away enough money.

Over the past year we've fixed that: board outreach around the grants program and initiatives such as grants office hours have put our grants program in a much healthier position... but potentially too healthy.

We took steps to improve that visibily and streamline that process, and they worked! This gives us a new problem: we now have no shortage of applicants, so we need to figure out how to stick within a budget that won't harm the financial sustainability of the PSF itself.

Does this mean we say no to more events? Should we instead reduce the size of our grants? Can we take other initiatives, like more actively helping events find alternative forms of sponsorship?

Grants shouldn't just be about events - but if we're making grants to other initiatives that support the Python community how can we fairly select those, manage the budget allocated to supporting them and maximize the value the Python community gets from the money managed by the PSF?

A much larger topic for the retreat was strategic planning. What should our goals be for the PSF that can't be achieved over a short period of time? Projects and initiatives that might require a one-year, three-year or five-year margin of planning.

Director terms only last three years (though board members can and frequently do stand for re-election), so having these long-term goals planned and documented in detail is crucial.

A five-year plan is not something that can be put together over two days of work, but the in-person meeting is a fantastic opportunity to kick things off and ensure each board member gets to participate in shaping that process.

Want to know more?

The above is by no means a comprehensive manual to the PSF, but it's a good representation of the things I would have found most valuable to understand when I first got involved with the organization.

For a broader set of perspectives on how the board works and what it does, I recommend the FAQs about the PSF Board video on YouTube.

If you're interested in helping the PSF achieve its mission, we would love to have you involved:

Encourage your company to sponsor the PSF directly, or to sponsor Python events worldwide
Volunteer at PyCon US or help with other suitable PSF initiatives
Join a Work Group that's addressing problems you want to help solve
Run your own event and apply for a grant
Join the PSF as a voting member and vote in our elections
Run for the board elections yourself!

We're always interested in hearing from our community. We host public office hours on the PSF Discord every month, at different times of day to to cater for people in different timezones - here's the full calendar of upcoming office hours.

Tags: pypi, python, psf

Python Developers Survey 2023 Results

2024-09-03T02:47:45+00:00

Python Developers Survey 2023 Results

The seventh annual Python survey is out. Here are the things that caught my eye or that I found surprising:

25% of survey respondents had been programming in Python for less than a year, and 33% had less than a year of professional experience.

37% of Python developers reported contributing to open-source projects last year - a new question for the survey. This is delightfully high!

6% of users are still using Python 2. The survey notes:

Almost half of Python 2 holdouts are under 21 years old and a third are students. Perhaps courses are still using Python 2?

In web frameworks, Flask and Django neck and neck at 33% each, but FastAPI is a close third at 29%! Starlette is at 6%, but that's an under-count because it's the basis for FastAPI.

The most popular library in "other framework and libraries" was BeautifulSoup with 31%, then Pillow 28%, then OpenCV-Python at 22% (wow!) and Pydantic at 22%. Tkinter had 17%. These numbers are all a surprise to me.

pytest scores 52% for unit testing, unittest from the standard library just 25%. I'm glad to see pytest so widely used, it's my favourite testing tool across any programming language.

The top cloud providers are AWS, then Google Cloud Platform, then Azure... but PythonAnywhere (11%) took fourth place just ahead of DigitalOcean (10%). And Alibaba Cloud is a new entrant in sixth place (after Heroku) with 4%. Heroku's ending of its free plan dropped them from 14% in 2021 to 7% now.

Linux and Windows equal at 55%, macOS is at 29%. This was one of many multiple-choice questions that could add up to more than 100%.

In databases, SQLite usage was trending down - 38% in 2021 to 34% for 2023, but still in second place behind PostgreSQL, stable at 43%.

The survey incorporates quotes from different Python experts responding to the numbers, it's worth reading through the whole thing.

Via PSF news

Tags: open-source, postgresql, python, sqlite, surveys, pytest, psf, pydantic

PSF announces a new five year commitment from Fastly

2024-05-17T13:52:16+00:00

PSF announces a new five year commitment from Fastly

Fastly have been donating CDN resources to Python—most notably to the PyPI package index—for ten years now.

The PSF just announced at PyCon US that Fastly have agreed to a new five year commitment. This is a really big deal, because it addresses the strategic risk of having a key sponsor like this who might change their support policy based on unexpected future conditions.

Thanks, Fastly. Very much appreciated!

Tags: pypi, python, psf, fastly

Django Chat: Datasette, LLMs, and Django

2024-01-24T20:41:21+00:00

Django Chat: Datasette, LLMs, and Django

I’m the guest on the latest episode of the Django Chat podcast. We talked about Datasette, LLMs, the New York Times OpenAI lawsuit, the Python Software Foundation and all sorts of other topics.

Tags: django, podcasts, ai, psf, generative-ai, llms, podcast-appearances

Weeknotes: datasette-test, datasette-build, PSF board retreat

2024-01-21T11:34:43+00:00

I wrote about Page caching and custom templates in my last weeknotes. This week I wrapped up that work, modifying datasette-edit-templates to be compatible with the jinja2_environment_from_request() plugin hook. This means you can edit templates directly in Datasette itself and have those served either for the full instance or just for the instance when served from a specific domain (the Datasette Cloud case).

Testing plugins with Playwright

As Datasette 1.0 draws closer, I've started thinking about plugin compatibility. This is heavily inspired by my work on Datasette Cloud, which has been running the latest Datasette alphas for several months.

I spotted that datasette-cluster-map wasn't working correctly on Datasette Cloud, as it hadn't been upgraded to account for JSON API changes in Datasette 1.0.

datasette-cluster-map 0.18 fixed that, while continuing to work with previous versions of Datasette. More importantly, it introduced Playwright tests to exercise the plugin in a real Chromium browser running in GitHub Actions.

I've been wanting to establish a good pattern for this for a while, since a lot of Datasette plugins include JavaScript behaviour that warrants browser automation testing.

Alex Garcia figured this out for datasette-comments - inspired by his code I wrote up a TIL on Writing Playwright tests for a Datasette Plugin which I've now also used in datasette-search-all.

datasette-test

datasette-test is a new library that provides testing utilities for Datasette plugins. So far it offers two:

from datasette_test import Datasette
import pytest

@pytest.mark.asyncio
async def test_datasette():
    ds = Datasette(plugin_config={"my-plugin": {"config": "goes here"})

This datasette_test.Datasette class is a subclass of Datasette which helps write tests that work against both Datasette <1.0 and Datasette >=1.0a8 (releasing shortly). The way plugin configuration works is changing, and this plugin_config= parameter papers over that difference for plugin tests.

The other utility is a wait_until_responds("http://localhost:8001") function. Thes can be used to wait until a server has started, useful for testing with Playwright. I extracted this from Alex's datasette-comments tests.

datasette-build

So far this is just the skeleton of a new tool. I plan for datasette-build to offer comprehensive support for converting a directory full of static data files - JSON, TSV, CSV and more - into a SQLite database, and eventually to other database backends as well.

So far it's pretty minimal, but my goal is to use plugins to provide optional support for further formats, such as GeoJSON or Parquet or even .xlsx.

I really like using GitHub to keep smaller (less than 1GB) datasets under version control. My plan is for datasette-build to support that pattern, making it easy to load version-controlled data files into a SQLite database you can then query directly.

PSF board in-person meeting

I spent the last two days of this week at the annual Python Software Foundation in-person board meeting. It's been fantastic catching up with the other board members over more than just a Zoom connection, and we had a very thorough two days figuring out strategy for the next year and beyond.

Blog entries

Releases

datasette-edit-templates 0.4.3 - 2024-01-17
Plugin allowing Datasette templates to be edited within Datasette
datasette-test 0.2 - 2024-01-16
Utilities to help write tests for Datasette plugins and applications
datasette-cluster-map 0.18.1 - 2024-01-16
Datasette plugin that shows a map for any data with latitude/longitude columns
datasette-build 0.1a0 - 2024-01-15
Build a directory full of files into a SQLite database
datasette-auth-tokens 0.4a7 - 2024-01-13
Datasette plugin for authenticating access using API tokens
datasette-search-all 1.1.2 - 2024-01-08
Datasette plugin for searching all searchable tables at once

TILs

Publish releases to PyPI from GitHub Actions without a password or token - 2024-01-15
Using pprint() to print dictionaries while preserving their key order - 2024-01-15
Using expect() to wait for a selector to match multiple items - 2024-01-13
literalinclude with markers for showing code in documentation - 2024-01-10
Writing Playwright tests for a Datasette Plugin - 2024-01-09
How to get Cloudflare to cache HTML - 2024-01-09
Running Varnish on Fly - 2024-01-08

Tags: projects, datasette, weeknotes, datasette-cloud, playwright, psf

Introducing PyPI Organizations

2023-04-23T20:29:39+00:00

Introducing PyPI Organizations

Launched at PyCon US today: Organizations allow packages on the Python Package Index to be owned by a group, not an individual user account. “We’re making organizations available to community projects for free, forever, and to corporate projects for a small fee.”—this is the first revenue generating PyPI feature.

Tags: open-source, packaging, pypi, python, psf

Weeknotes: Joining the board of the Python Software Foundation

2022-07-30T19:02:04+00:00

A few weeks ago I was elected to the board of directors for the Python Software Foundation.

I put myself up for election partly because I've found myself saying "I wish the PSF would help with ..." a few times over the years, and realized that joining the board could be a more useful way to actively participate, rather than shouting from the sidelines.

I was quite surprised to win. I wrote up a short manifesto - you can see that here - but the voting system lets voters select as many candidates as they like, so it's possible I got in more on broad name recognition among the voters than based on what I wrote. I don't think there's a way to tell one way or the other.

I had my first board meeting on Wednesday, where I formally joined the board and got to vote on my first resolutions. This is my first time as a board member for a non-profit and I have learned a bunch already, with a lot more to go!

Board terms last three years. I expect it will take me at least a few months to get fully up to speed on how everything works.

As a board member, my primary responsibilities are to show up to the meetings, vote on resolutions, act as an ambassador for the PSF to the Python community and beyond and help both set the direction for the PSF and ensure that the PSF meets its goals and holds true to its values.

I'm embarassed to admit that I wrote my election manifesto without a deep understanding of how the PSF operates and how much is possible for it to get done. Here's the section I wrote about my goals should I be elected:

I believe there are problems facing the Python community that require dedicated resources beyond volunteer labour. I'd like the PSF to invest funding in the following areas in particular:

Improve Python onboarding. In coaching new developers I've found that the initial steps to getting started with a Python development environment can be a difficult hurdle to cross. I'd like to help direct PSF resources to tackling this problem, with a goal of making the experience of starting to learn Python as smooth as possible, no matter what platform the learner is using.

Make Python a great platform for distributing software. In building my own application, Datasette, in Python I've seen how difficult it can be to package up a Python application so that it can be installed by end-users, who aren't ready to install Python and learn pip in order to try out a new piece of software. I've researched solutions for this for my own software using Homebrew, Docker, an Electron app and WASM/Pyodide. I'd like the PSF to invest in initiatives and documentation to make this as easy as possible, so that one of the reasons to build with Python is that distributing an application to end-users is already a solved problem.

I still think these are good ideas, and I hope to make progress on them during my term as a director - but I'm not going to start arguing for new initiatives until I've learned the ropes and fully understood the PSF's abilities, current challenges and existing goals.

In figuring out how the board works, one of the most useful pages I stumbled across was this list of resolutions voted on by the board, dating back to 2001. There are over 1,600 of them! Browsing through them gave me a much better idea of the kind of things the board has the authority to do.

Scraping data into Datasette Lite

Because everything looks like a nail when you have a good hammer, I explored the board resolutions by loading them into Datasette. I tried a new trick this time: I scraped data from that page into a CSV file, then loaded up that CSV file in Datasette Lite via a GitHub Gist.

My scraper isn't perfect - it misses about 150 resolutions because they don't exactly fit the format it expects, but it was good enough for a proof of concept. I wrote that in a Jupyter Notebook which you can see here.

Here's the CSV in a Gist. The great thing about Gists is that GitHub serve those files with the access-control-allow-origin: * HTTP header, which means you can load them cross-domain.

Here's what you get if you paste the URL to that CSV into Datasette Lite (using this new feature I added last month):

And here's a SQL query that shows the sum total dollar amount from every resolution that mentions "Nigeria":

with filtered as (
  select * from
    [psf-resolutions]
  where
    "dollars" is not null
    and "text" like '%' || :search || '%'
)
select
  'Total: $' || printf('%,d', sum(dollars)) as text,
  null as date
from filtered
union all
select
  text, date
from filtered;

I'm using a new-to-me trick here: I use a CTE to filter down to just the rows I am interested in, then I create a first row that sums the dollar amounts as the text column and leaves the date column null, then unions that against the rows from the query.

Important note: These numbers aren't actually particularly useful. Just because the PSF board voted on a resolution does not mean that the money made it to the grantee - there are apparently situations where the approved grant may not be properly claimed and transferred. Also, my scraper logic isn't perfect. Plus the PSF spends a whole lot of money in ways that don't show up in these resolutions.

So this is a fun hack, and a neat way to start getting a general idea of how the PSF works, but any numbers it produces should not be taken as the absolute truth.

As a general pattern though, I really like this workflow of generating CSV files, saving them to a Gist and then opening them directly in Datasette Lite. It provides a way to use Datasette to share and explore data without needing to involve any server-side systems (other than GitHub Gists) at all!

Big-endian bugs in sqlite-fts4

sqlite-fts4 is a small Python library I wrote that adds SQLite functions for calculating relevance scoring for full-text search using the FTS4 module that comes bundled with SQLite. I described that project in detail in Exploring search relevance algorithms with SQLite.

It's a dependency of sqlite-utils so it has a pretty big install base, despite being relatively obscure.

This week I had a fascinating bug report from Sarah Julia Kriesch: Test test_underlying_decode_matchinfo fails on PPC64 and s390x on openSUSE.

The s390x is an IBM mainframe architecture and it uses a big-endian byte order, unlike all of the machines I use which are little-endian.

This is the first time I've encountered a big-endian v.s. little-endian bug in my entire career! I was excited to dig in.

Here's the relevant code:

 def decode_matchinfo(buf): 
     # buf is a bytestring of unsigned integers, each 4 bytes long 
     return struct.unpack("I" * (len(buf) // 4), buf)

SQLite FTS4 provides a matchinfo binary string which you need to decode in order to calculate the relevance score. This code uses the struct standard library module to unpack that binary string into a list of integers.

My initial attempt at fixing this turned out to be entirely incorrect.

I didn't have a big-endian machine available for testing, and I assumed that the problem was caused by Python interpreting the bytes as the current architecture's byte order. So I applied this fix:

    return struct.unpack(">" + ("I" * (len(buf) // 4)), buf)

The > prefix there ensures that struct will always interpret the bytes as little-endian. I wrote up a TIL and shipped 1.0.2 with the fix.

Sarah promptly got back to me and reported some new failing tests.

It turns out my fix was entirely incorrect - in fact, I'd broken something that previously was working just fine.

The clue is in the SQLite documentation for matchinfo (which I really should have checked):

The matchinfo function returns a blob value. If it is used within a query that does not use the full-text index (a "query by rowid" or "linear scan"), then the blob is zero bytes in size. Otherwise, the blob consists of zero or more 32-bit unsigned integers in machine byte-order (emphasis mine).

Looking more closely at the original bug report, the test that failed was this one:

@pytest.mark.parametrize(
    "buf,expected",
    [
        (
            b"\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00",
            (1, 2, 2, 2),
        )
    ],
)
def test_underlying_decode_matchinfo(buf, expected):
    assert expected == decode_matchinfo(buf)

That test hard-codes a little-endian binary string and checks the output of my decode_matchinfo function. This is obviously going to fail on a big-endian system.

So my original behaviour was actually correct: I was parsing the string using the byte order of the architecture, and SQLite was providing the string in the byte order of the architecture. The only bug was in my test.

I reverted my previous fix and fixed the test instead:

@pytest.mark.parametrize(
    "buf,expected",
    [
        (
            b"\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00"
            if sys.byteorder == "little"
            else b"\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02",
            (1, 2, 2, 2),
        )
    ],
)
def test_underlying_decode_matchinfo(buf, expected):
    assert expected == decode_matchinfo(buf)

sys.byteorder reports the byte order of the host system, so this test now passes on both little-endian and big-endian systems.

There was one remaining challenge: how to test this? I wasn't going to make the same mistake of shipping a fix that hadn't actually been exercised on the target architecture a second time.

After quite a bit of research (mainly throwing the terms docker and s390x into the GitHub code search engine and seeing what I could find) I figured out a fix. It turns out you can use Docker and QEMU to run an emulated s390x system - both on a Mac laptop and in GitHub Actions.

Short version:

docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker run -it multiarch/ubuntu-core:s390x-focal /bin/bash

For the longer version, check my TIL: Emulating a big-endian s390x with QEMU.

Releases this week

sqlite-fts4: 1.0.3 - (5 releases total) - 2022-07-30
Custom Python functions for working with SQLite FTS4
shot-scraper: 0.14.2 - (17 releases total) - 2022-07-28
A comand-line utility for taking automated screenshots of websites
datasette-sqlite-fts4: 0.3.1 - 2022-07-28
Datasette plugin that adds custom SQL functions for working with SQLite FTS4
datasette-publish-vercel: 0.14.1 - (22 releases total) - 2022-07-23
Datasette plugin for publishing data using Vercel
datasette-insert: 0.8 - (8 releases total) - 2022-07-22

TIL this week

Tags: python, sqlite, datasette, weeknotes, sqlite-utils, datasette-lite, psf

Donate to the PSF!

2003-12-20T22:02:07+00:00

Donate to the PSF!

Support Python as well.

Tags: python, psf

Simon Willison's Weblog: psf

Anthropic invests $1.5 million in the Python Software Foundation and open source security

PyCon US 2026 call for proposals is now open

The PSF has withdrawn a $1.5 million proposal to US government grant program

Python 3.14

Announcing the 2025 PSF Board Election Results!

The 2025 PSF Board Election is Open!

2025 Python Packaging Ecosystem Survey

PyPI now supports project archival

PyPI now supports digital attestations

Perks of Being a Python Core Developer

Weeknotes: Three podcasts, two trips and a new plugin system

Lisbon, Portugal and Durham, North Carolina

Podcasts

On the blog

Museums

Releases

TILs

Things I've Learned Serving on the Board of The Perl Foundation

Things I've learned serving on the board of the Python Software Foundation

What is the PSF?

The PSF employs staff

A lot of this is about money

The PSF does not directly develop Python itself

PyPI - the Python Package Index

PyCon is a key commitment

Other PSF activities

Work Groups

Acting as a fiscal sponsor

Life as a board member

The kinds of things the board talks about

Want to know more?

Python Developers Survey 2023 Results

PSF announces a new five year commitment from Fastly

Django Chat: Datasette, LLMs, and Django

Weeknotes: datasette-test, datasette-build, PSF board retreat

Testing plugins with Playwright

datasette-test

datasette-build

PSF board in-person meeting

Blog entries

Releases

TILs

Introducing PyPI Organizations

Weeknotes: Joining the board of the Python Software Foundation

Scraping data into Datasette Lite

Big-endian bugs in sqlite-fts4

Releases this week

TIL this week

Donate to the PSF!