Top Menu

How do I download an entire raw data directory?

In order to recursively download an entire raw data directory from, you can use the following script:

wget -r -e robots=off -nd URL


  • The -r is recursive
  • The -nd does not create directories, but will append an extension if two files have identical names.
  • The -e robots=off tells wget to ignore the robots.txt file. If this command is left out, the robots.txt file tells wget that it does not like web crawlers which makes wget stop.


wget  -r -e robots=off -nd

— Last revised on September 28, 2016 —