Simon Willison’s Weblog

Rye lets you get from no Python on a computer to a fully functioning Python project in under a minute with linting, formatting and everything in place.

[...] Because it was demonstrably designed to avoid interference with any pre-existing Python configurations, Rye allows for a smooth and gradual integration and the emotional barrier of picking it up even for people who use other tools was shown to be low.

— Armin Ronacher # 4th February 2024, 3:12 pm

Find a level of abstraction that works for what you need to do. When you have trouble there, look beneath that abstraction. You won’t be seeing how things really work, you’ll be seeing a lower-level abstraction that could be helpful. Sometimes what you need will be an abstraction one level up. Is your Python loop too slow? Perhaps you need a C loop. Or perhaps you need numpy array operations.

You (probably) don’t need to learn C.

— Ned Batchelder # 24th January 2024, 6:25 pm

[On Python 3.12 subinterpreters] there’s massive advantages for mixed C(++) and Python: I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library.

Previously this required rewriting the whole C++ library to support either pickling (multiplying the total memory consumption by the number of cores), or support allocating everything in shared memory (which means normal C++ types like `std::string` are unusable, need to switch e.g. to boost::interprocess).

Now is sufficient to pickle a pointer to a C++ object as an integer, and it’ll still be a valid pointer in the other subinterpreter.

— ynik # 2nd October 2023, 6:13 pm

My strong hunch is that the GIL does not need removing, if a) subinterpreters have their own GILs and b) an efficient way is provided to pass (some) data between subinterpreters lock free and c) we find good patterns to make working with subinterpreters work.

— Armin Ronacher # 11th April 2023, 4:47 pm

Several libraries let you declare objects with type-hinted members and automatically derive validation rules and serialization/deserialization from the type hints – Pydantic is the most popular, but alternatives like msgspec are out there too. There’s also a whole new generation of web frameworks like FastAPI and Starlite which use type hints at runtime to do not just input validation and serialization/deserialization but also things like dependency injection.

Personally, I’ve seen more significant gains in productivity from those runtime usages of Python’s type hints than from any static ahead-of-time type checking, which mostly is only useful to me as documentation.

— James Bennett # 7th April 2023, 2:19 am

I’ve always believed that a book, even a technical book, should try to tell a cohesive story. The challenge is that as Python has grown in popularity, it has really turned into three different languages--each with their own story. There is a whimsical Python for scripting and tinkering, a quirky Python for asynchronous programming and networking, and a serious Python for enterprise applications. Sometimes these stories intersect. Sometimes not.

— David Beazley # 18th July 2021, 2:53 pm

Async functions require an event loop to run. Flask, as a WSGI application, uses one worker to handle one request/response cycle. When a request comes in to an async view, Flask will start an event loop in a thread, run the view function there, then return the result.

Each request still ties up one worker, even for async views. The upside is that you can run async code within a view, for example to make multiple concurrent database queries, HTTP requests to an external API, etc. However, the number of requests your application can handle at one time will remain the same.

— Using async and await in Flask 2.0 # 12th May 2021, 5:59 pm

When you have to mock a collaborator, avoid using the Mock object directly. Either use mock.create_autospec() or mock.patch(autospec=True) if at all possible. Autospeccing from the real collaborator means that if the collaborator’s interface changes, your tests will fail. Manually speccing or not speccing at all means that changes in the collaborator’s interface will not break your tests that use the collaborator: you could have 100% test coverage and your library would fall over when used!

— Thea Flowers # 17th March 2021, 4:44 pm

Inevitably we got round to talking about async.

As much of an unneeded complication as it is for so many day-to-day use-cases, it’s important for Python because, if and when you do need the high throughput handling of these io-bound use-cases, you don’t want to have to switch language.

The same for Django: most of what you’re doing has no need of async but you don’t want to have to change web framework just because you need a sprinkling of non-blocking IO.

— Carlton Gibson # 27th September 2020, 3:09 pm

The ASGI specification provides an opportunity for Python to hit a productivity/performance sweet-spot for a wide range of use-cases, from writing high-volume proxy servers through to bringing large-scale web applications to market at speed.

— Tom Christie # 8th October 2018, 2:43 pm

At Harvard we’ve built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their JupyterHub environment. Our most ambitious so far has been setting up each student in the course with a p2.xlarge machine with cuda and TensorFlow so they could do deep learning work for their final projects. We supported 15 courses last year, and got deployment time for an implementation down to only 2-3 hours.

— Chris Rogers # 5th June 2018, 7:37 pm

I think that “bad technology” can kill a startup, but slightly different variations of good technology don’t have much effect. Choose what you know/like best. And Ruby and Python are both in this latter category.

— enko on Hacker News # 2nd October 2010, 11:19 am

Right now, pypy compiled with JIT can run the whole CPython test suite without crashing, which means we’re done with obvious bugs and the only ones waiting for us are really horrible.

— Maciej Fijalkowski # 1st May 2009, 3:04 pm

Apparently [unladen-swallow] is already 30% faster than CPython, and this version is being used to run some of the Python code on YouTube.

— Ted Leung # 30th March 2009, 10:10 am

I don’t think that Python 3.0 is a bad thing. But that it’s displayed so prominently on the Python web site, without any kind of warning that it’s not going to work with 99% of the Python code out there, scares the hell out of me. People are going to download and install 3.0 by default, and nothing’s going to work. They’re going to complain, and many are going to simply walk away.

— Christopher Lenz # 6th December 2008, 10 am

My Universal Feed Parser was conceived as a weapon against what I considered the gravest error of XML: draconian error handling. Recently, someone asked me to implement a switch that makes it not fall back on lax parsing in the case of an XML wellformedness error. I said no, not because it would be difficult to implement, but because that defeats its entire reason for being.

— Mark Pilgrim # 5th August 2008, 10:52 pm

DjangoCon 2008. Venue: Gooleplex, San Francisco Bay Area. Dates: 6th and 7th Sept. Official post will be on djangoproject.com soon.

— Robert Lofthouse # 13th July 2008, 4:50 pm

What amazes me is how close Ruby 1.9 bytecode and Python 2.5 bytecode are. Some things translate almost directly. [...] And, really, if that’s true (and I vouch that it is truly, truly true,) then how are Python and Ruby still on separate runtimes?

— Why the lucky stiff # 5th May 2008, 10:14 pm

The Google App Engine model class, db.Model, is not the same as the model class used by Django. As a result, you cannot directly use the Django forms framework with Google App Engine. However, Google App Engine includes a module, db.djangoforms, which casts between the datastore models used with Google App Engine and the Django models specification. In most cases, you can use db.djangoforms.ModelForm in the same manner as the Django framework.

— Google App Engine docs # 8th April 2008, 1:48 pm

I definitely like Python 3K’s Unicode support better [...] In fact, I think I prefer Ruby 1.8’s non-support for Unicode over Ruby 1.9’s “support”. The problem is one that is all to familiar to Python programmers. You can have a fully unit tested library and have somebody pass you a bad string, and you will fall over.

— Sam Ruby # 28th December 2007, 7:05 pm

If you only remember one thing about handling non-HTML output via Django: know that you can use the HttpResponse object as if it were a file. Writing to such an object and returning it will give you the output you wrote. It’s a very simple concept, but one that translates well to third-party libraries.

— Alex de Landgraaf # 3rd December 2007, 8:44 pm

Django may be built for the Web, but CouchDB is built of the Web. I’ve never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated.

— Jacob Kaplan-Moss # 20th October 2007, 1:46 pm

The use of double underscores creates a separate namespace for names that are part of the Python language definition, so that programmers are free to create variables, attributes, and methods that start with letters, without fear of silently colliding with names that have a language-defined purpose.

— Ka-Ping Yee # 1st September 2007, 1:30 am

The recent announcement that Mozilla’s next JavaScript engine, Tamarin, will also be a container for functionality written in Python and Ruby (and, one assumes, beyond) is proof that JavaScript is the new Parrot.

— Aaron Straup Cope # 29th July 2007, 9:17 pm

Just because Java was once aimed at a set-top box OS that didn’t support multiple address spaces, and just because process creation in Windows used to be slow as a dog, doesn’t mean that multiple processes (with judicious use of IPC) aren’t a much better approach to writing apps for multi-CPU boxes than threads.

— Guido van Rossum # 8th May 2007, 9:21 pm

Dependency injection is the enterprisey name for trampling over namespaces with reckless abandon.

— Ted Dziuba # 16th December 2006, 1:05 am