The Webpage settings let you customize how webpages are downloaded, analyzed, and modified.
The Webpage pane of the Settings dialog provides the following controls:
Use this control to specify the text encoding for webpages. SiteSucker will read and save all webpages using the specified text encoding. If it is set to Default, SiteSucker will try to detect the webpage’s text encoding. This setting is ignored when reading or saving webpages that were previously downloaded.
Download Using Web Views
Check this box to have SiteSucker download a site as a PDF document. Each webpage is rendered as a single page in the document, and the width and minimum height of each PDF page is determined by the Web View Size setting. The order of pages in the document will be somewhat random due to the nonlinear nature of websites. Internet links are preserved in the document, and links to downloaded pages are modified so that the user is sent to the appropriate location within the document. Only webpages and other files that can be displayed in a web browser are included in the document. This control is only enabled if the Download Using Web Views setting is on. When this setting is on, other settings that do not apply are ignored.
Web View Size
Use this control to specify the size of hidden web views. The web view width can effect the appearance of downloaded webpages and PDF documents created by SiteSucker. The web view height determines the minimum page height when creating PDF documents. This control is only enabled if the Download Using Web Views setting is on.
data- and do not contain uppercase characters.
To add a custom data attribute, click the button, enter the name of the attribute, and press ↩.
To remove custom data attributes, select them in the table and click the button.
To modify a custom data attribute, double-click on its name in the table, enter a new name, and press ↩. All names in the table must be unique.
Settings under this tab allow you to use regular expressions to replace text in HTML files or extract URLs from HTML text.
To replace text in HTML files, set the Template Type to Early Substitution or Late Substitution and enter a search pattern and a substitution template for the text you would like to replace. If text is found that matches the search pattern, it is altered in accordance with the substitution template. The template specifies what should be used to replace each match, with the back-reference
$0 representing the matched text,
$1 representing the contents of the first capture group, and so on. To delete the matched text, leave the template blank.
To extract URLs from HTML text, set the Template Type to URL Extraction and enter a search pattern and a URL template that specifies a URL that SiteSucker should download. The URL template is ignored if it produces a blank URL or a URL that is identical to the template.
The URL Extraction and Early Substitution search patterns are applied after any pre-analysis script is run, but before SiteSucker scans HTML files for URLs. The Late Substitution search pattern is applied after HTML files are localized. Search patterns are applied in the order in which they appear in the table, and the order of search patterns can be rearranged by dragging them in the table. The pattern syntax currently supported is that specified by ICU, which is described at Regular Expressions - ICU Documentation.
As an example, in the image shown above, SiteSucker is instructed to do the following:
extract a URL from the first argument of the
- replace URLs that have a certain query string with the same URL without the query string.
To add a row to the table, click the button, set the Template Type, enter the Search Pattern and Template, and press ↩.
To remove rows from the table, select them in the table and click the button.
To modify a row, double-click on a string in the table, enter a new string, and press ↩.
This feature can be used to perform any number of tasks before SiteSucker saves a webpage. For example, it can be used to click on buttons that modify a webpage before SiteSucker saves it; click on links that download attachments; extract obscure URLs from a webpage; or rename files.
In the image shown above, the script calls the
moreImagesFunction() after the window has loaded and calls it again whenever webpage changes occur. When the
.postMessage(<argument>) in your code. The message names recognized by SiteSucker, their use, and their arguments are discussed in the table below.
|delay||Allows you to add a delay before saving a webpage. The function argument should be an integer or a floating-point number that specifies the delay in seconds. Essentially, this function resets the Save Delay setting to the value passed into the function without modifying the setting in the SiteSucker document.|
|rename||Allows you to rename files. The function argument should be an array containing two strings: a search pattern followed by a substitution template. These strings are added temporarily to the Replace table under the Path settings and are removed after the document stops downloading. You could use this feature to rename files with the text content of a button or anchor.|
After the last page is saved, SiteSucker will analyze all the saved pages.