Simon Willison’s Weblog

40 items tagged “testing”

2021

I’m pretty convinced that the biggest single contributor to improved software in my lifetime wasn’t object-orientation or higher-level languages or functional programming or strong typing or MVC or anything else: It was the rise of testing culture.

Tim Bray # 1st June 2021, 2:35 pm

When you have to mock a collaborator, avoid using the Mock object directly. Either use mock.create_autospec() or mock.patch(autospec=True) if at all possible. Autospeccing from the real collaborator means that if the collaborator’s interface changes, your tests will fail. Manually speccing or not speccing at all means that changes in the collaborator’s interface will not break your tests that use the collaborator: you could have 100% test coverage and your library would fall over when used!

Thea Flowers # 17th March 2021, 4:44 pm

Blazing fast CI with pytest-split and GitHub Actions (via) pytest-split is a neat looking variant on the pattern of splitting up a test suite to run different parts of it in parallel on different machines. It involves maintaining a periodically updated JSON file in the repo recording the average runtime of different tests, to enable them to be more fairly divided among test runners. Includes a recipe for running as a matrix in GitHub Actions. # 22nd February 2021, 7:06 pm

Litestream runs continuously on a test server with generated load and streams backups to S3. It uses physical replication so it’ll actually restore the data from S3 periodically and compare the checksum byte-for-byte with the current database.

Ben Johnson # 11th February 2021, 8:50 pm

2020

How to cheat at unit tests with pytest and Black

I’ve been making a lot of progress on Datasette Cloud this week. As an application that provides private hosted Datasette instances (initially targeted at data journalists and newsrooms) the majority of the code I’ve written deals with permissions: allowing people to form teams, invite team members, promote and demote team administrators and suchlike.

[... 885 words]

2019

parameterized. I love the @parametrize decorator in pytest, which lets you run the same test multiple times against multiple parameters. The only catch is that the decorator in pytest doesn’t work for old-style unittest TestCase tests, which means you can’t easily add it to test suites that were built using the older model. I just found out about parameterized which works with unittest tests whether or not you are running them using the pytest test runner. # 19th February 2019, 9:05 pm

2018

Documentation unit tests

Keeping documentation synchronized with an evolving codebase is difficult. Without extreme discipline, it’s easy for documentation to get out-of-date as new features are added.

[... 1518 words]

Hynek Schlawack: Testing & Packaging (via) “How to ensure that your tests run code that you think they are running, and how to measure your coverage over multiple tox runs (in parallel!)”—Hynek makes a convincing argument for putting your packaged Python code in a src/ directory for ease of testing and coverage. # 22nd May 2018, 10:12 pm

2017

I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea. The problem is that you get diminishing returns on our tests as the coverage increases much beyond 70% (I made that number up… no science there). Why is that? Well, when you strive for 100% all the time, you find yourself spending time testing things that really don’t need to be tested. Things that really have no logic in them at all (so any bugs could be caught by ESLint and Flow). Maintaining tests like this actually really slow you and your team down.

Kent C. Dodds # 27th October 2017, 6:20 am

Cypress (via) Promising looking new open source testing framework for full-blown web integration testing—a modern alternative to Selenium. I spent five minutes playing with the demo and was really impressed by it—especially their “time travel” feature which lets you hover over a passed test and see the state of the browser when each of those assertions was executed. # 11th October 2017, 4:14 pm

2011

One interesting quirk of Pinboard is a complete absence of unit tests. I used to be a die-hard believer in testing, but in Pinboard tried a different approach, as an experiment. Instead of writng tests I try to be extremely careful in coding, and keep the code size small so I continue to understand it. I’ve found my defect rate to be pretty comparable to earlier projects that included extensive test suites and fixtures, but I am much more productive on Pinboard.

Maciej Ceglowski # 11th February 2011, 2:57 am

2010

Unit Testing Achievements. A plugin for Python’s nose test runner that adds achievements—“Night Shift: Make a failing suite pass between 12am and 5am.” # 28th February 2010, 3:56 pm

twitter-text-conformance (via) This is a neat idea: Twitter have released open source libraries for parsing standard tweet syntax in Ruby and Java, but they’ve also released a set of YAML unit tests aimed at anyone who wants to implement the same parsing logic in other languages. # 6th February 2010, 3:39 pm

rlisagor’s freshen. A Python clone of Ruby’s innovative Cucumber testing framework. Tests are defined as a set of plain-text scenarios, which are then executed by being matched against test functions decorated with regular expressions. Has anyone used this or Cucumber? I’m intrigued but unconvinced—are the plain text scenarios really a useful way of defining tests? # 5th January 2010, 7:30 pm

2009

There is no WebKit on Mobile. PPK ran 27 tests against 19 different WebKit-on-mobile implementations and found enormous disparities between the levels of support in currently available mobile phones. # 7th October 2009, 12:23 pm

shunit2 (via) xUnit style testing for shell scripts. # 27th September 2009, 7:34 pm

Fabric factory. Promising looking continuous integration server written in Django, which uses Fabric scripts to define actions. # 21st September 2009, 6:35 pm

Test-Driven Heresy. Tim Bray advocates TDD for maintenance development, but argues that it may not be as useful during the exploratory, greenfield development phase of a project. # 24th June 2009, 11:03 am

Testing Django Views for Concurrency Issues. Neat decorator for executing a Django view under high concurrency in your unit tests, to help spot errors caused by database race conditions that should be executed inside a transaction. # 27th May 2009, 10:01 am

Nose 0.11 released. My favourite Python testing tool just got some really neat new features, including the ability to parallelize tests across multiple processes (hence CPUs) using the multiprocess module, Xunit XML output for integration with continuous integration tools and a --failed switch to re-run only the last batch of failed tests. # 8th May 2009, 11:24 am

Continuous deployment in 5 easy steps. A classic case of a number in a title making the article look less interesting than it actually is. Lots of interesting information here from IMVU’s Eric Ries. # 1st April 2009, 12:25 am

It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

Timothy Fitz # 10th February 2009, 3:06 pm

Changeset 9793: SMTP testing documentation. I didn’t know this trick: running “python -m smtpd -n -c DebuggingServer localhost:1025” will start up a simple SMTP server which dumps received e-mails to the terminal instead of forwarding them on. # 29th January 2009, 1:35 pm

Django now has fast tests. Changeset 9756 switched Django’s TestCase class to running tests inside a transaction and rolling back at the end (instead of doing a full dump and reload). “Ellington’s test suite, which was taking around 1.5-2 hours to run on Postgres, has been reduced to 10 minutes.” # 16th January 2009, 11:40 am

Localbuilder. Gareth Rushgrove’s neat little Python continuous integration tool—it watches a directory for changes, then runs a command when it spots any. # 14th January 2009, 10:57 pm

2008

Django snippets: RequestFactory. I’ve been wanting this for ages; when I finally got around to writing it it turned out to only be a dozen or so lines of code. Makes it easy to create mock request objects in Django, which you can then use for testing view functions directly (bypassing the current test client mechanism which requires views to be assigned to a URLconf before they can be tested). # 12th August 2008, 12:40 pm

ratproxy. “A semi-automated, largely passive web application security audit tool”—watches you browse and highlights potential XSS, CSRF and other vulnerabilities in your application. Created by Michal Zalewski at Google. # 3rd July 2008, 2:35 pm

QUnit. The jQuery unit testing framework is now documented and supported as a separate project. # 26th May 2008, 5:31 pm

Python one-liner of the day. I love the idea of publishing one-liners accompanied by one-line test suites. # 26th April 2008, 10:24 am

xPyUnit: Uniting in Python with XML reporting. Should be just the ticket for integrating Django’s testing framework with Cruise Control. # 27th March 2008, 12:35 pm