Request

Request

The Request section of the Settings dialog provides the following controls:

Identity

Use this control to customize the way SiteSucker identifies itself when making an HTTP request. Some sites are very particular about which browsers they will allow. You can use this feature to "fool" the site into thinking that SiteSucker is an approved browser.

To change SiteSucker's identity, simply click on this control and select one of the web browsers listed. (If you choose None, SiteSucker will not include any identifying information when making requests.)

You can add identities to the standard list of web browsers by clicking the Identities button in the Preferences.

Attempts

Use this control to specify the number of times SiteSucker should attempt to download a file. SiteSucker will only retry downloading a file if a timeout, network connection, or too many requests (429) error occurs.

Timeout

Use this control to select the length of time that SiteSucker should wait for a response from the server.

Delay

Use this control to specify the minimum length of time that SiteSucker should delay between HTTP requests to the same host. This feature allows SiteSucker to use less bandwidth and avoid anti-mining safeguards employed by some sites.

If a Crawl-delay is already being imposed by the site's robots.txt file, the longer of the two delays will be used. See Robot Exclusions for more information about the robots.txt file.