Simon Willison’s Weblog

Items tagged python in Aug

Filters: Month: Aug × python ×


Announcing the Consortium for Python Data API Standards (via) Interesting effort to unify the fragmented DataFrame API ecosystem, where increasing numbers of libraries offer APIs inspired by Pandas that imitate each other but aren’t 100% compatible. The announcement includes some very clever code to support the effort: custom tooling to compare the existing APIs, and an ingenious GitHub Actions setup to run traces (via sys.settrace), derive type signatures and commit those generated signatures back to a repository. # 19th August 2020, 5:48 am

Pysa: An open source static analysis tool to detect and prevent security issues in Python code (via) Interesting new static analysis tool for auditing Python for security vulnerabilities—things like SQL injection and os.execute() calls. Built by Facebook and tested extensively on Instagram, a multi-million line Django application. # 7th August 2020, 8:50 pm

PyPI now supports uploading via API token (via) All of my open source Python libraries are set up to automatically deploy new tagged releases as PyPI packages using Circle CI or Travis, but I’ve always get a bit uncomfortable about sharing my PyPI password with those CI platforms to get this to work. PyPI just added scopes authentication tokens, which means I can issue a token that’s only allowed to upload a specific project and see an audit log of when that token was last used. # 1st August 2019, 4:03 pm

The subset of reStructuredText worth committing to memory

reStructuredText is the standard for documentation in the Python world.

[... 1183 words]

Compiling SQLite for use with Python Applications (via) Charles Leifer’s recent tutorial on how to compile and build the latest SQLite (with window function support) for use from Python via his pysqlite3 library. # 15th August 2018, 3:51 pm

coleifer/pysqlite3. Now that the pysqlite package is bundled as part of the Python standard library the original open source project is no longer actively maintained, and has not been upgraded for Python 3. Charles Leifer has been working on pysqlite3, a stand-alone package of the module. Crucially, this should enable compiling the latest version of SQLite (via the amalgamation package) without needing to upgrade the version that ships with the operating system. # 15th August 2018, 3:15 pm

Faust: Python Stream Processing (via) A new open source stream processing system released by Robinhood, created by Vineet Goel and Celery creator Ask Solem. The API looks delightful, making very smart use of Python decorators and async/await. The initial release requires Kafka but they plan to support multiple backends, hopefully including Redis Streams. # 6th August 2018, 10:51 pm

Datasette unit tests: monkeytype_call_traces (via) Faceted browse against every function call that occurs during the execution of Datasette’s test suite. I used Instagram’s MonkeyType tool to generate this, which can run Python code and generates a SQLite database of all of the traced calls. It’s intended to be used to automatically add mypy annotations to your code, but since it produces a SQLite database as a by-product I’ve started exploring the intermediary format using Datasette. Generating this was as easy as running “monkeytype run `which pytest`” in the Datasette root directory. # 2nd August 2018, 9:03 pm

Web app: what programming knowledge do I need to create a goings on app?

For this kind of application a much more important question than "how can I build it? is "where will I get the data from?"—If you don’t have a good answer for that building the app is a waste of your time. The world is littered with local events listings apps that no one uses because they don’t have good data.

[... 82 words]

Does SQLAlchemy depend on MySQLdb?

No. SQLAlchemy can talk to all sorts of different DB-API compliant backends, including MySQL Connector/J (Jython only), MySQL Connector/Python, mysql-python (the MySQLdb module) and OurSQL—plus backends for many other databases. See the full list here:

[... 50 words]

What is the best way to learn about setting up server software for Python based web apps?

I’m a big fan of Fabric for automated deployment scripts. Start by reading this tutorial: http://morethanseven.net/2009/07...

[... 40 words]

What is the highest traffic website built on top of Django?

My best guess would be Disqus. Instagram are pretty enormous these days as well.

[... 31 words]

Writing your own traceroute in Python. How to implement traceroute in Python, using the low-level socket module. # 9th August 2010, 12:58 pm

Exploring OAuth-Protected APIs. One of the downsides of OAuth is that it makes debugging APIs in your browser much harder. Seth Fitzsimmons’ oauth-proxy solves this by running a Twisted-powered proxy on your local machine which OAuth-signs every request going through it using your consumer key, secret and tokens for that API. Using it with a browsers risks exposing your key and token (but not secret) to sites you accidentally browse to—it would be useful if you could pass a whitelist of API domains as a command line option to the proxy. # 23rd August 2009, 11:06 am

easy_install no longer working with SourceForge-hosted projects? Unsurprising, since installation software (which is often run as root) that crawls the web and scrapes HTML pages for download links is a horrible, horrible idea. # 19th August 2009, 11:38 am

Kung Fu People (via) The first site to launch based on the open source Django code from djangopeople.net! # 19th August 2009, 11:37 am

How do you install lxml on OS X Leopard without using MacPorts or Fink? I’ve asked on Stack Overflow... hope I get a good answer. # 14th August 2009, 1:04 pm

Python logging from multiple processes. Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs. # 13th August 2009, 11:55 pm

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub. # 13th August 2009, 1:51 pm

minixsv (via) As far as I can tell, this is the only library that can validate XML using pure Python (no C extension required). I’d be extremely happy if someone would write a pure Python library (or one that only depends on ElementTree, which is included in the standard library) for validating XML against a Relax NG Compact syntax schema. Even DTD validation would be better than nothing! # 12th August 2009, 4:59 pm

topia.termextract. Impressive Python term extraction library (similar to the various term extraction web APIs but you can run it on your own hardware), incorporating a Parts-Of-Speech tagging algorithm. # 10th August 2009, 9:26 pm

Richard Jones: Something I’m working on... Python’s with statement appears to provide just enough syntactic sugar to create some really interesting DSL-style APIs—here’s a very promising example for laying out GUI applications. # 7th August 2009, 3:47 pm

Using Akismet with Django’s new comments framework. A nice example that demonstrates two features that were recently rolled in to the Django 1.0 betas: the new signals library and the new comments framework. # 28th August 2008, 10:12 am

URLsafe base64 encoding/decoding in two lines. A much better solution than my base65 hack—if you understand how base64 padding works (I didn’t) you can use it to generate URL-safe compressed hashes. Performance should be significantly better than my version. # 28th August 2008, 9:57 am

Django snippets: Sign a string using SHA1, then shrink it using url-safe base65. I needed a way to create tamper-proof URLs and cookies by signing them, but didn’t want the overhead of a full 40 character SHA1 hash. After some experimentation, it turns out you can knock a 40 char hash down to 27 characters by encoding it using a custom base65 encoding which only uses URL-safe characters. # 27th August 2008, 10:18 pm

Django 1.0 beta 2 released! 1.0 draws ever closer. Important new features include major documentation and comment system refactorings, plus the removal of a bunch of deprecated code (including oldforms). Feature and string freezes are now in place, so it’s just bugs and documentation improvements between now and the final release. # 27th August 2008, 2:41 pm

Django documentation (for 1.0). The documentation refactor is in: the docs for the upcoming 1.0 release have been tidied up, rearranged and ported to a new documentation system based on Sphinx (the Python documentation toolkit, NOT the full-text search engine). The URL has also changed to docs.djangoproject.com. # 24th August 2008, 10:49 am

The Python Property Builtin. The always-educational Adam Gomaa explains the Python property built-in and shows how it can be used to improve Django’s model-based URL generation. # 23rd August 2008, 1:08 pm

django-timezones. Models, form fields and a template filter for dealing with timezones in Django. # 21st August 2008, 11:18 pm

Persistent Django on Amazon EC2 and EBS—the easy way. Useful tutorial on getting Django up and running on EC2 with EBS for a persistent PostgreSQL database. # 21st August 2008, 9:32 pm