August 2018
Aug. 2, 2018
Datasette unit tests: monkeytype_call_traces (via) Faceted browse against every function call that occurs during the execution of Datasette’s test suite. I used Instagram’s MonkeyType tool to generate this, which can run Python code and generates a SQLite database of all of the traced calls. It’s intended to be used to automatically add mypy annotations to your code, but since it produces a SQLite database as a by-product I’ve started exploring the intermediary format using Datasette. Generating this was as easy as running “monkeytype run `which pytest`” in the Datasette root directory.
Aug. 6, 2018
Analyzing US Election troll tweets with Datasette
FiveThirtyEight published nearly 3 million tweets from accounts associated with the Russian “Internet Research Agency”, based on research by Darren Linvill and Patrick Warren at at Clemson University.
[... 110 words]Analyzing US Election Russian Facebook Ads
Two interesting data sources have emerged in the past few weeks concerning the Russian impact on the 2016 US elections.
[... 922 words]OWASP Top 10 2007-2017: The Fall of CSRF. I was surprised to learn recently that CSRF didn’t make it into the 2017 OWASP Top 10 security vulnerabilities (after featuring almost every year since the list started). The credited reason is that web frameworks do a good enough job protecting against CSRF by default that it’s no longer a top-ten problem. Defaults really do matter.
How to Read an RFC. An extremely useful guide to reading RFCs by Mark Nottingham. I didn’t know most of the stuff in here.
Faust: Python Stream Processing (via) A new open source stream processing system released by Robinhood, created by Vineet Goel and Celery creator Ask Solem. The API looks delightful, making very smart use of Python decorators and async/await. The initial release requires Kafka but they plan to support multiple backends, hopefully including Redis Streams.
Aug. 7, 2018
Securing Web Sites Made Them Less Accessible (via) This is fascinating: the move to HTTP everywhere breaks local HTTP caching servers (like Squid) which are still used in remote areas that get their internet by a high latency satellite connection.
Aug. 9, 2018
Easy explainer: a "blockchain" is a linked list with an append-only restriction, and appending is made incredibly expensive but super parallelizable, so when things work well a big group of people can work together and it's too expensive for a small evil group to compete. [...] Does your problem benefit from storing information in an append-only list, and relying on a central authority to manage it is so bad that it's worth paying the enormous append costs to have a bunch of Chinese servers manage it for you? Then maybe look at a blockchain.
Using achievement stats to estimate sales on steam (via) Really interesting data leak exploit here: Valve’s Steam API was showing the percentage of users that gained a specific achievement up to 16 decimal places—which inadvertently leaked their exact usage statistics, since if 0.012782207690179348 percent of players get an achievement the only possible input is 8 players out of 62,587.
Aug. 11, 2018
With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
Aug. 13, 2018
Experiences with running PostgreSQL on Kubernetes (via) Fascinating interview that makes a solid argument for the idea that running stateful data stores like PostgreSQL or Cassandra is made harder, not easier when you add an orchestration tool like Kubernetes into the mix.
Aug. 15, 2018
Window Functions in SQLite 3.25.0. The next release of SQLite (apparently due for release in September) will add window functions, as specified in various SQL standards and already available in PostgreSQL. This is going to dramatically improve SQLite as an engine for performing analytical queries, especially across time series data. It’s also going to further emphasize the need for people to be able to upgrade their SQLite versions beyond those provided by the operating system—the default Ubuntu run by Travis CI still only ships with SQLite 3.8 for example.
coleifer/pysqlite3. Now that the pysqlite package is bundled as part of the Python standard library the original open source project is no longer actively maintained, and has not been upgraded for Python 3. Charles Leifer has been working on pysqlite3, a stand-alone package of the module. Crucially, this should enable compiling the latest version of SQLite (via the amalgamation package) without needing to upgrade the version that ships with the operating system.
Compiling SQLite for use with Python Applications (via) Charles Leifer’s recent tutorial on how to compile and build the latest SQLite (with window function support) for use from Python via his pysqlite3 library.
Aug. 18, 2018
Text to Image (via) Ridiculously entertaining demo by Cris Valenzuela that feeds any text you type to a neural network that then attempts to generate an image for your text.
Every day more than 1 trillion events are written into a streaming ingestion pipeline, which is processed and written to a 100PB cloud-native data warehouse. And every day, our users run more than 150,000 jobs against this data, spanning everything from reporting and analysis to machine learning and recommendation algorithms.
Beyond Interactive: Notebook Innovation at Netflix. Netflix have been investing heavily in their internal Jupyter notebooks infrastructure: it’s now the most popular tool for working with data at Netflix. They also use parameterized notebooks to make it easy to create templates for reusable operations, and scheduled notebooks for recurring tasks. “When a Spark or Presto job executes from the scheduler, the source code is injected into a newly-created notebook and executed. That notebook then becomes an immutable historical record, containing all related artifacts — including source code, parameters, runtime config, execution logs, error messages, and so on.”
Redux vs. The React Context API. Nice explanation of the new Context API in React 16.3, which provides an easy way for passing props down through a tree of components without needing to explicitly pass the prop at every level of the tree. The comparison with Redux doubles as a useful explanation of the value that Redux provides.
Observable Tutorial 2: Dog pictures (via) Observable have a neat new set of tutorials on how to get started with their reactive notebooks. You don’t even need to sign up for the service: they have a “Scratchpad” link in their navigation bar now which lets you spin up a test notebook with one click.
How about if, instead of ditching Twitter for Mastodon, we all start blogging and subscribing to each other's Atom feeds again instead? The original distributed social network could still work pretty well if we actually start using it
— @simonw
Aug. 19, 2018
How to Instantly Publish Data to the Internet with Datasette
I presented a session about Datasette at the PyBay 2018 conference in San Francisco. I talked about the project itself and demonstrated ways of creating and publishing databases using csvs-to-sqlite, Datasette Publish and my new sqlite-utils library.
[... 2,043 words]Slides, notes and links from my Datasette talk at PyBay (via) I presented a session about Datasette at the PyBay conference in San Francisco this morning. I talked about the project itself and demonstrated ways of creating and publishing databases using csvs-to-sqlite, Datasette Publish and my new sqlite-utils library.
Aug. 22, 2018
6 Great Uses of the Spread Operator. As I’ve been getting more comfortable with 2018-era JavaScript the spread operator and object restructuring are two of the features I have found most interesting.
jq recipes. Remy Sharp’s handy collection of jq recipes, each one linking to an interactive demo on jqterm.com. I thought jq was just for extracting values from a JSON document—I hadn’t realized how powerful it was for modifying and extending those documents as well.
Aug. 25, 2018
I don’t like Jupyter Notebooks—a presentation by Joel Grus (via) Fascinating talk by Joel Grus at the Jupyter conference in New York. He highlights some of the drawbacks of he Jupyter way of working, including the huge confusion that can come from the ability to execute cells out of order (something Observable notebooks solve brilliantly using spreadsheet-style reactive cell associations). He also makes strong arguments that notebooks encourage a way of working that discourages people from producing stable, repeatable and well tested code.
Honeycomb changelog (via) Too few hosted services have detailed user-facing changelogs. This one from Honeycomb (a metrics, tracing and observavility platform) is a particularly great example. I especially like the use of animated screenshots, something I’ve been evangelizing pretty heavily recently for internal communication at work.
In case you missed it: @GoogleColab can open any @ProjectJupyter notebook directly from @github!
To run the notebook, just replace "github.com" with "colab.research.google.com/github/" in the notebook URL, and it will be loaded into Colab.
The subset of reStructuredText worth committing to memory
reStructuredText is the standard for documentation in the Python world.
[... 1,186 words]Most administrators will force users to change their password at regular intervals, typically every 30, 60 or 90 days. This imposes burdens on the user (who is likely to choose new passwords that are only minor variations of the old) and carries no real benefits as stolen passwords are generally exploited immediately. [...] Regular password changing harms rather than improves security, so avoid placing this burden on users. However, users must change their passwords on indication or suspicion of compromise.
The Future of Notebooks: Lessons from JupyterCon (via) It sounds like reactive notebooks (where cells keep track of their dependencies on other cells and re-evaluate when those update) were a hot topic at JupyterCon this year.