Request
The Request settings allow you to customize the HTTP requests that SiteSucker sends to the server.
The Request screen provides the following settings:
Identity
Use this control to customize the user agent string that SiteSucker provides to identify itself when making HTTP requests. Some sites are very particular about which browsers they will allow. You can use this feature to “fool” the site into thinking that SiteSucker is an approved browser.
To change SiteSucker’s identity, simply tap on this control and select one of the web browsers listed. If you choose Web View, the web view’s default user agent string is included in all requests. This is the same web view that’s used in the Download Using Web Views option under the Webpage settings.
Attempts
Use this control to specify the number of times SiteSucker should attempt to download a file. SiteSucker will only retry downloading a file if a timeout, network connection, too many requests (429), bad gateway (502), or gateway timeout (504) error occurs.
Timeout
Use this control to select the length of time that SiteSucker should wait for a response from the server.
Delay
Use this control to specify the minimum length of time that SiteSucker should delay between HTTP requests to the same host. This feature allows SiteSucker to use less bandwidth and avoid anti-mining safeguards employed by some sites.
If a Crawl-delay is already being imposed by the site’s robots.txt file, the longer of the two delays will be used. See Robot Exclusions for more information about the robots.txt file.