This article covers excluding pages from a crawl, ignoring tests or pages, showing or hiding ignored items, viewing issues and errors, viewing and comparing past audits, downloading and exporting results, and allowing our crawler through robots.txt and firewalls.
Before you start: Site Auditor is available as a paid add-on for accounts created before November 2023. To learn more, reach out to our support team. New to Site Auditor? Start with the Connect and Manage Site Auditor article to connect it and run your first audit.
Excluding vs. Ignoring: What's the Difference?
These two sound similar but do different things:
Excluding a page stops the auditor from crawling it at all. The page never appears in your results. Use this to keep the crawler off sections like a blog or staging pages
Ignoring a test or page still causes it to be crawled, but hides it from your results and removes it from your audit score. Use this to clean up reporting without changing what gets crawled
How do I exclude specific pages from being crawled?
Excluding a path or URL stops the auditor from scanning that part of the site. You can also use robots.txt to allow or block our crawler (see the "Why Isn't the Auditor Crawling My Site?" section below).
Step 1: Click the settings button at the top right of your Site Auditor dashboard, click Settings, then select Manage to open the crawl settings.
Step 2: Click Edit.
Step 3: Enter the path or URL you want to skip in the Excluded URLs / Paths box, then click the + button to the right. Add as many as you need, then click Save at the bottom.
Paths are excluded using a contains rule, so anything matching the path you enter is skipped.
Example 1: If the Client URL is www.agencyanalytics.com and the blog lives at www.agencyanalytics.com/blog (with posts like /blog/myblogpost1, /blog/myblogpost2, and so on), enter "agencyanalytics.com/blog" to exclude the entire blog and all its posts.
Example 2: If the site hosts listings like www.agencyanalytics.com/usa/listing/entry1 and www.agencyanalytics.com/canada/listing/entry2, enter "/listing/" to exclude all listing pages. To exclude only Canadian listings, enter "/canada/listing".
Other examples:
Exclude a subdomain by entering its full URL, like
toronto.agencyanalytics.com, to skip all pages on that subdomainExclude query parameters by entering the path up to the parameter, like "agencyanalytics.com/something?page=". Because we match using a contains rule, this skips ?page=1, ?page=2, and so on
To remove an exclusion, go to the crawl settings page and click the Remove Icon (the garbage can) next to the rule you want to delete.
Can I use wildcards or regular expressions in the exclusions box?
Not yet. The exclusions box doesn't currently accept wild cards or regular expressions, but this is on our roadmap and will likely be released at a future date.
How do I Ignore a Test or Page in my Results?
Ignoring hides a test, page, or issue from your results and removes it from your audit score. This is useful when you don't want a test to affect the score, or you want to hide something from your reporting. Ignored items can also be hidden from dashboards and reports (see the "How Do I Show or Hide Ignored Tests or Pages?" section below).
There are three ways to ignore items:
Ignore an Entire Test
Head to the Tests section on the Site Auditor dashboard, click the checkbox next to the test (for example, "page is not canonical"), then click Ignore.
Ignore an Entire Page
Head to the Pages section on the Site Auditor dashboard. Scroll to the page you want to ignore, or search for it using the search box in the top right of the table. Click the checkbox next to the page, then click Ignore.
Ignore a Specific Test on a Page
In the Pages section, click the page you want. A slide-out appears with the page details. Scroll down to the specific test you'd like to ignore on that page.
How do I Show or Hide Ignored Tests or Pages?
Once you've ignored items, you can choose whether they still appear on your dashboard. This helps you show only the tests or pages that matter to you or your client.
Step 1: Click the settings button in the top-right corner of the Site Auditor dashboard.
Step 2: Under Filters, select Show Ignored Tests or Hide Ignored Tests from the dropdown menu.
Does hiding an ignored issue add it back to my score?
No. Whether ignored issues are shown or hidden, they're never included in your site audit scores.
How do I View Site Audit Issues and Errors?
Once an audit finishes, you can drill into each problem to see exactly what needs fixing. Make sure you've run an audit for the Client first.
Step 1: From within your Client, open the Site Auditor dashboard.
Step 2: You'll land on the Tests page, with widgets for score, page depth, and page and test summaries at the top.
Step 3: Scroll down to the Tests and Failures section. Each test shows the affected pages, along with additional columns that help you resolve the issue. Click the title of any test to see more.
Step 4: A slider opens on the right with a page breakdown, an explanation of the test, and details to help you locate the issue. For example, clicking a Missing Alt Attributes test with 34 failures shows the 34 images that lack alt tags.
After you fix an issue on the site, the next audit won't return that error. You can then compare the two audits (see the "Compare to a Previous Audit" section below) for more insight.
How do I Analyze an Individual Page?
Beyond the test-level view, you can drill down into any single page for granular, page-level stats. Make sure you've run an audit for the Client first.
Once the audit is finished, open the Pages section under the Site Auditor dashboard to see every page that was crawled. Pages are sorted by audit score from lowest to highest by default. Click any column name to change the sort order, or click the Settings button in the top right to add or remove metrics and show or hide page results.
Click any page in the table to open the Page Details slide-out.
The Page Details slide-out has the following tabs:
Test Tab
A quick overview of the flags on a page. It shows the page's overall score, including the number of critical issues, errors, and warnings. Scroll down to see each individual flag currently affecting the page.
Insights Tab
A high-level technical summary of the crawled page. It includes crawl metadata like the date it was last scanned, status code, and load time, plus on-page metrics like title length, page size, URL length, number of internal and external links, and total image count. Use this tab to quickly spot page structure and basic performance issues.
Inlinks Tab
Lists all the "inlinks," the URLs that point to the page you're reviewing. You'll also see link attributes like the anchor text, whether it's a dofollow or nofollow link, and the link type.
Outlinks Tab
Shows your "outlinks," the links to other pages or websites. Like the Inlinks tab, it identifies link attributes such as anchor text, whether links are nofollow, and link type.
Images Tab
A detailed list of all images found on the page. For each image, it shows the alt text (if available), the HTTP status code confirming whether the image loaded, the file size, and the wait time it took to load during the crawl. Use this tab to find missing alt attributes, oversized images that may slow load speed, and slow-loading assets.
How do I View Previous Audit Results?
Site Auditor keeps your past audits so you can review earlier pages and tests.
Step 1: Click the History section from the Site Auditor dashboard.
Step 2: Click the past audit you'd like to view.
Step 3: You'll return to the Tests section, where the selected audit is now shown. A dropdown at the top-left shows the audit date you're viewing. The auditor keeps showing this audit's data until you return to the History tab.
How long is detailed audit data kept?
The detailed report has a shelf life of 60 days. When a report passes 60 days, you'll see a pop-up alert. It's just a notification you can close, and you can still run a new audit at any time.
After 60 days, the system keeps the overall score in the History tab, but the detailed results (pages, errors, and so on) are archived and cannot be accessed. To keep long-term records, download your results (see below).
How do I Compare to a Previous Audit?
Comparing your current results to those from an earlier audit is a great way to track progress on client work.
Step 1: Click the Add Filter button at the top of your Site Auditor dashboard.
Step 2: Under Filters, choose Compare To, then select the past audit you want to compare against.
Once Compare To is enabled, a change percentage indicator appears in the table widget on your dashboard, showing how results have changed between the two audits.
How do I Download or Export Site Audit Results?
You can export your audit as a PDF or an XLS file, or share it via a link. Detailed results can be downloaded for up to 60 days, after which only the score and basic stats are kept.
Step 1: From within the Client, click the Share button in the upper right, next to the Perform Audit button. In the dropdown, click Download PDF or Download XLS.
Step 2: An XLS file downloads straight to your computer. A PDF is queued for generation.
Step 3: Once the export status shows Completed, click the file to download it. You can find it under your Exports (formally called Downloads).
To download the details of one test: Click the test title in the table to open its slide-out, then click Export in the upper right and choose PDF or XLS. The report is generated and can be downloaded from the Exports tab.
Why isn't the Auditor Crawling my Site?
In most cases, robots.txt files won't block our site auditor, and our IP addresses won't be restricted. But if you have robots.txt rules that block bots, or any IP restrictions, use the steps below so the auditor can still run. We strongly recommend using an experienced web developer to make these changes, since the process varies by website.
You can also stop our crawler from scanning parts of your site through the Excluded URLs / Paths setting (see the "How Do I Exclude Specific Pages From Being Crawled?" section above).
Use robots.txt to Allow the Crawler
You can allow or block our crawler specifically using the user agent "Mozilla/5.0 (compatible; RSiteAuditor)" in your robots.txt file. This example allows our auditor to crawl your whole website:
User-agent: RSiteAuditor
Allow: /
If you've tried this and the auditor still won't crawl, your server may be blocking it. If you see 4xx or 5xx status codes across all pages in the auditor, this is likely the cause. Whitelist the user agent "Mozilla/5.0 (compatible; RSiteAuditor)" (without quotation marks) on your server. This whitelisting is done on your server, not in robots.txt.
Use robots.txt to Allow Only a Specific Path
If you only need the auditor to crawl a certain path, you can specify it in robots.txt. This example allows our auditor to crawl only the pages under example.com/categories:
User-agent: RSiteAuditor
Disallow: /
Allow: /categories
If your robots.txt changes aren't working, check your cache. A different version of robots.txt may be served from your website, server, or CDN cache (like Cloudflare). Clearing the cache usually fixes this.
Whitelist Our IP Addresses
If the auditor still can't connect, whitelist the following IP addresses on your website or server firewall.
IPv4
94.130.93.30
168.119.141.170
168.119.99.190
168.119.99.191
168.119.99.192
168.119.99.193
168.119.99.194
68.183.60.34
134.209.42.109
68.183.60.80
68.183.54.131
68.183.49.222
68.183.149.30
68.183.157.22
68.183.149.129
IPv6
2a01:4f8:c17:f386::1/128
2a01:4f8:c17:f387::1/128
2a01:4f8:c17:f38a::1/128
2a01:4f8:c17:f394::1/128
2a01:4f8:c17:f395::1/128
2a01:4f8:251:5d3::2/128
2604:a880:800:10::eb:9001/128
2604:a880:800:10::596:4001/128
2604:a880:800:10::e9:1001/128
2604:a880:800:10::65b:f001/128
2604:a880:800:10::695:7001/128
2604:a880:800:10::6da:6001/128
2604:a880:800:10::6ee:8001/128
2604:a880:800:10::6f7:3001/128
What's Next
Need to troubleshoot something in Site Auditor? Check out this article.
💬 Need additional help?
If you have any questions, please contact our friendly support team by following these instructions! We're available 24/5 to help 😄