After activating the site auditor, and after you run an audit, you will be glad to know that the site auditor has additional settings that you can modify for future audits.

You will find "Crawl Settings" by clicking the settings button at the top right of the dashboard and switching to the Settings tab.

Finally, click "edit" next to "Crawl Settings".


A popup window will appear with the following crawl settings options available.

  • Start url - This is where the crawl will begin. This is the campaign URL, and you have the option to update that here. Note: Customizing the campaign URL will also affect the rank tracker.

  • Include Sub-Domains - With this option enabled, the auditor will also crawl subdomains based on your URL. For example: if the campaign URL is www.agencyanalytics.com, and 'include subdomains" is enabled, the subdomain help.agencyanalytics.com will also be crawled. Disable this option if you only want to crawl www.agencyanalytics.com and its subdirectories (e.g. www.agencyanalytics.com/blog).

  • Useragent - This is always set to Mozilla/5.0 (compatible; aa/1.0), which is our crawler. If your website is blocking our crawler, you can white list our useragent by name. Note that you cannot white list our crawler by IP, as there is no fixed IP or range. Lastly, you can also update your robots.txt to allow our crawler.

  • Page Crawl Limit - The number of pages our auditor will crawl. Use this setting to limit the number of webpages we'll scan during the audit.

  • Excluded URLs/Paths - Specify the pages you want our auditor to exclude from being crawled. Here's a short guide to help you with this.

  • Monthly Page Crawls - An indicator that tells you how many pages you have audited, and how close you are to your account's limit. This limit is account wide.


What's Next

Did this answer your question?