Simon Willison’s Weblog

How can I download a web server’s directory and all subdirectories with one command?

15th January 2012

My answer to How can I download a web server’s directory and all subdirectories with one command? on Quora

Use wget (you can install it with apt-get install wget)

$ wget --recursive http://example.com

That will create a directory called example.com and put the mirrored downloaded files in the right sub-directories inside it.

If you just want to download a subdirectory, do this:

$ wget --recursive http://example.com/subdirectory --no-parent

The --no-parent option ensures wget won’t follow links up to parent directories of the one you want to download.