The URL settings offer options that affect URLs and provide a way to specify which URLs should be included in or excluded from the download.
The URL pane of the Settings dialog provides the following controls:
Check All Links
Check this box to have SiteSucker check all links in all downloaded HTML files — including links to files that the app is not downloading — and log any errors that occur. With this option turned on, SiteSucker may report many errors that you normally would not see. This setting is intended as a debugging tool for web designers who want to see if their own sites have any bad links.
To minimize the time it takes to check all links, set the Filter setting under the File Type settings to Allow Specified File Types with nothing checked so that only HTML and CSS are downloaded, and set the File Modification setting under the General settings to Delete After Analysis, which will delete HTML and CSS files after they are downloaded and analyzed.
Scan Comments for URLs
Treat Ambiguous URLs as Folders
Check this box to have SiteSucker treat ambiguous URLs as folders. If a URL does not end with a ‘/’ or a file extension, SiteSucker considers it to be ambiguous. For example, if this option is on and SiteSucker downloads a webpage from
https://www.example.com/directory, the webpage will be saved at
www.example.com/directory/index.html in the destination folder. If this option is off, the webpage will be saved at
www.example.com/directory.html in the destination folder. See File Names for more information about how SiteSucker names downloaded files.
Download Links in PDFs
Check this box to have SiteSucker download files linked to PDF documents if allowed by other settings. Since PDF links cannot be localized, the PDF documents themselves are never modified.
Use this control to limit downloaded files to those at a specific site, those within a specific directory, or those having a specific URL. This option works in conjunction with the Include and Exclude URL settings and the General settings. SiteSucker provides the following path constraints:
- None - SiteSucker downloads the file specified in the URL text box and every file that it links to and every site that these files link to, etc. Be aware that this option could result in a HUGE download if allowed to continue forever.
Host - SiteSucker limits the download to those files on the host of the original file being downloaded. For example, if the URL is
https://www.example.com/directory/home.html, this setting limits the download to those URLs beginning with
- Host + 1 - SiteSucker limits the download to those files on the host of the original file being downloaded (just like the Host option), plus one level of files from other domains linked to the original host.
Subdomains - SiteSucker limits the download to those files within the second-level domain and all subdomains of the original file being downloaded. Extending the previous example, this setting will download URLs beginning with
Directory - SiteSucker only downloads those files that are within the directory of the original file being downloaded. For example, if you are downloading
https://www.example.com/directory/using this setting, SiteSucker will only download files in the
directorydirectory. But if you are downloading
https://www.example.com/directory, SiteSucker will download all files from
www.example.comunless the Treat Ambiguous URLs as Folders setting is on, in which case SiteSucker will only download files in the
- URL Settings - SiteSucker only downloads the file specified in the URL text box and any files that have URLs allowed by the Include and Exclude URL settings.
Include and Exclude URL Settings
The Include and Exclude URL settings work in conjunction with the URL Constraint setting and the General settings according to the following rules:
- If this is the original URL (that is, the URL specified in the URL text field), then the file is downloaded.
- Otherwise, if the URL begins with one of the strings or matches one of the regular expressions in the Exclude table, then the file is not downloaded.
- Otherwise, if the URL meets the requirements of the current URL Constraint setting, then the file is allowed to download.
- Otherwise, if the URL begins with one of the strings or matches one of the regular expressions in the Include table, then the file is allowed to download.
- Otherwise, if the Always Download HTML and CSS option in the General settings is on and the URL references an HTML or CSS file type, then the file is allowed to download.
- Otherwise, if the Include Supporting Files option in the General settings is on and the URL references a non-HTML file type, then the file is allowed to download.
- Otherwise, the file is not downloaded.
In the tables under the Include and Exclude tabs, enter absolute URLs (that is, URLs beginning with
https://) or regular expression patterns. URLs should be entered as they appear in the Safari address and search field, i.e., without encoding except for characters from the ISO-8859-1 extended character set and spaces (which are encoded as
If you are adding a regular expression, check the corresponding Regex box. When using regular expressions, the pattern must match the entire URL. For example, to match any URL that contains “
logout”, enter the “
.*logout.*” regular expression. The pattern syntax currently supported is that specified by ICU, which is described at Regular Expressions - ICU Documentation.
To add a row to the table, click the button, enter the URL or pattern, and press ↩.
To remove rows from the table, select them in the table and click the button.
To modify a row, double-click on a string in the table, enter a new string, and press ↩.