wget

WGET with UserAgent, random-wait and depth level

Submitted by Gast (nicht überprüft) on So, 12/06/2020 - 14:04

WGET do:

  1. Download example.com
  2. Download all level-1 and level-2 links
  3. Download everything to : /root/t3mp/wget
  4. Identify as 'Mozilla/MyUserAgent'
  5. Ignore robots.txt
  6. Random wait between requests (average : 3 seconds)



wget -U 'Mozilla/MyUserAgent' -P /root/t3mp/wget --wait=3 --random-wait -nd -r -l 2 -e robots=off https://example.com