wget

WGET do:

1. Download example.com
2. Download all level-1 and level-2 links
3. Download everything to : /root/t3mp/wget
4. Identify as 'Mozilla/MyUserAgent'
5. Ignore robots.txt
6. Random wait between requests (average : 3 seconds)

wget -U 'Mozilla/MyUserAgent' -P /root/t3mp/wget --wait=3 --random-wait -nd -r -l 2 -e robots=off https://example.com

1 * * * * wget -O /dev/null -o /dev/null my-example-website.com

Beispiel-Code:

wget -nd -r -l 1 -e robots=off https://example.com

-l steht für Level/Depth/Tiefe.