This article covers excluding pages from a crawl, ignoring tests or pages, showing or hiding ignored items, viewing issues and errors, viewing and comparing past audits, downloading and exporting results, and allowing our crawler through robots.txt and firewalls.

The auditor crawls every page it can reach on the site, starting from the client's URL.

Here's what it includes:

The only pages it skips are those blocked via robots.txt, added to the Excluded URLs / Paths setting, or using a non-HTTP/HTTPS URL.

If the client's root URL returns an X-Robots-Tag: noindex HTTP header, the crawl stops immediately and returns 0 pages. This is different from noindex pages found mid-crawl, which are crawled normally. If you're seeing an empty crawl result, check whether the root URL has a noindex HTTP header set at the server level.

These two sound similar but do different things:

Excluding a path or URL stops the auditor from scanning that part of the site. You can also use robots.txt to allow or block our crawler (see the "Why Isn't the Auditor Crawling My Site?" section below).

Step 1: Click the settings button at the top right of your Site Auditor dashboard, click Settings, then select Manage to open the crawl settings.

Step 2: Click Edit.

Step 3: Enter the path or URL you want to skip in the Excluded URLs / Paths box, then click the + button to the right. Add as many as you need, then click Save at the bottom.

Paths are excluded using a contains rule, so anything matching the path you enter is skipped.

Example 1: If the Client URL is www.agencyanalytics.com and the blog lives at www.agencyanalytics.com/blog (with posts like /blog/myblogpost1, /blog/myblogpost2, and so on), enter "agencyanalytics.com/blog" to exclude the entire blog and all its posts.

Example 2: If the site hosts listings like www.agencyanalytics.com/usa/listing/entry1 and www.agencyanalytics.com/canada/listing/entry2, enter "/listing/" to exclude all listing pages. To exclude only Canadian listings, enter "/canada/listing".

Other examples:

To remove an exclusion, go to the crawl settings page and click the Remove Icon (the garbage can) next to the rule you want to delete.

Not yet. The exclusions box doesn't currently accept wildcards or regular expressions, but this is on our roadmap and will likely be released at a future date.

Ignoring hides a test, page, or issue from your results and removes it from your audit score. This is useful when you don't want a test to affect the score, or you want to hide something from your reporting. Ignored items can also be hidden from dashboards and reports (see the "How Do I Show or Hide Ignored Tests or Pages?" section below).

There are three ways to ignore items:

Head to the Tests section on the Site Auditor dashboard, click the checkbox next to the test (for example, "page is not canonical"), then click Ignore.

Head to the Pages section on the Site Auditor dashboard. Scroll to the page you want to ignore, or search for it using the search box in the top right of the table. Click the checkbox next to the page, then click Ignore.

In the Pages section, click the page you want. A slide-out appears with the page details. Scroll down to the specific test you'd like to ignore on that page.

Once you've ignored items, you can choose whether they still appear on your dashboard. This helps you show only the tests or pages that matter to you or your client.

Step 1: Click the settings button in the top-right corner of the Site Auditor dashboard.

Step 2: Under Filters, select Show Ignored Tests or Hide Ignored Tests from the dropdown menu.

No. Whether ignored issues are shown or hidden, they're never included in your site audit scores.

Once an audit finishes, you can drill into each problem to see exactly what needs fixing. Make sure you've run an audit for the Client first.

Step 1: From within your Client, open the Site Auditor dashboard.

Step 2: You'll land on the Tests page, with widgets for score, page depth, and page and test summaries at the top.

Step 3: Scroll down to the Tests and Failures section. Each test shows the affected pages, along with additional columns that help you resolve the issue. Click the title of any test to see more.

Step 4: A slider opens on the right with a page breakdown, an explanation of the test, and details to help you locate the issue. For example, clicking a Missing Alt Attributes test with 34 failures shows the 34 images that lack alt tags.

After you fix an issue on the site, the next audit won't return that error. You can then compare the two audits (see the "Compare to a Previous Audit" section below) for more insight.

Beyond the test-level view, you can drill down into any single page for granular, page-level stats. Make sure you've run an audit for the Client first.

Once the audit is finished, open the Pages section under the Site Auditor dashboard to see every page that was crawled. Pages are sorted by audit score from lowest to highest by default. Click any column name to change the sort order, or click the Settings button in the top right to add or remove metrics and show or hide page results.

Click any page in the table to open the Page Details slide-out.

A quick overview of the flags on a page. It shows the page's overall score, including the number of critical issues, errors, and warnings. Scroll down to see each individual flag currently affecting the page.

A high-level technical summary of the crawled page. It includes crawl metadata like the date it was last scanned, status code, and load time, plus on-page metrics like title length, page size, URL length, number of internal and external links, and total image count. Use this tab to quickly spot page structure and basic performance issues.

Lists all the "inlinks," the URLs that point to the page you're reviewing. You'll also see link attributes like the anchor text, whether it's a dofollow or nofollow link, and the link type.

Shows your "outlinks," the links to other pages or websites. Like the Inlinks tab, it identifies link attributes such as anchor text, whether links are nofollow, and link type.

A detailed list of all images found on the page. For each image, it shows the alt text (if available), the HTTP status code confirming whether the image loaded, the file size, and the wait time it took to load during the crawl. Use this tab to find missing alt attributes, oversized images that may slow load speed, and slow-loading assets.

Site Auditor keeps your past audits so you can review earlier pages and tests.

Step 1: Click the History section from the Site Auditor dashboard.

Step 2: Click the past audit you'd like to view.

Step 3: You'll return to the Tests section, where the selected audit is now shown. A dropdown at the top-left shows the audit date you're viewing. The auditor keeps showing this audit's data until you return to the History tab.

The detailed report has a shelf life of 60 days. When a report passes 60 days, you'll see a pop-up alert. It's just a notification you can close, and you can still run a new audit at any time.

After 60 days, the system keeps the overall score in the History tab, but the detailed results (pages, errors, and so on) are archived and cannot be accessed. To keep long-term records, download your results (see below).

Comparing your current results to those from an earlier audit is a great way to track progress on client work.

Step 1: Click the Add Filter button at the top of your Site Auditor dashboard.

Step 2: Under Filters, choose Compare To, then select the past audit you want to compare against.

Once Compare To is enabled, a change percentage indicator appears in the table widget on your dashboard, showing how results have changed between the two audits.

You can export your audit as a PDF or an XLS file, or share it via a link. Detailed results can be downloaded for up to 60 days, after which only the score and basic stats are kept.

Step 1: From within the Client, click the Share button in the upper right, next to the Perform Audit button. In the dropdown, click Download PDF or Download XLS.

Step 2: An XLS file downloads straight to your computer. A PDF is queued for generation.

Step 3: Once the export status shows Completed, click the file to download it. You can find it under your Exports (formally called Downloads).

To download the details of one test: Click the test title in the table to open its slide-out, then click Export in the upper right and choose PDF or XLS. The report is generated and can be downloaded from the Exports tab.

In most cases, robots.txt files won't block our site auditor, and our IP addresses won't be restricted. But if you have robots.txt rules that block bots, or any IP restrictions, use the steps below so the auditor can still run. We strongly recommend using an experienced web developer to make these changes, since the process varies by website.

You can also stop our crawler from scanning parts of your site through the Excluded URLs / Paths setting (see the "How Do I Exclude Specific Pages From Being Crawled?" section above).

You can allow or block our crawler specifically using the user agent "Mozilla/5.0 (compatible; RSiteAuditor)" in your robots.txt file. This example allows our auditor to crawl your whole website:

If you've tried this and the auditor still won't crawl, your server may be blocking it. If you see 4xx or 5xx status codes across all pages in the auditor, this is likely the cause. Whitelist the user agent "Mozilla/5.0 (compatible; RSiteAuditor)" (without quotation marks) on your server. This whitelisting is done on your server, not in robots.txt.

If you only need the auditor to crawl a certain path, you can specify it in robots.txt. This example allows our auditor to crawl only the pages under example.com/categories:

If your robots.txt changes aren't working, check your cache. A different version of robots.txt may be served from your website, server, or CDN cache (like Cloudflare). Clearing the cache usually fixes this.

If the auditor still can't connect, whitelist the following IP addresses on your website or server firewall.

94.130.93.30

168.119.141.170

168.119.99.190

168.119.99.191

168.119.99.192

168.119.99.193

168.119.99.194

68.183.60.34

134.209.42.109

68.183.60.80

68.183.54.131

68.183.49.222

68.183.149.30

68.183.157.22

68.183.149.129

2a01:4f8:c17:f386::1/128

2a01:4f8:c17:f387::1/128

2a01:4f8:c17:f38a::1/128

2a01:4f8:c17:f394::1/128

2a01:4f8:c17:f395::1/128

2a01:4f8:251:5d3::2/128

2604:a880:800:10::eb:9001/128

2604:a880:800:10::596:4001/128

2604:a880:800:10::e9:1001/128

2604:a880:800:10::65b:f001/128

2604:a880:800:10::695:7001/128

2604:a880:800:10::6da:6001/128

2604:a880:800:10::6ee:8001/128

2604:a880:800:10::6f7:3001/128

If the Site Auditor is flagging duplicate content but you don't see an obvious issue, it's because we use a similarity algorithm, the same one Google uses, to compare pages and detect content that's highly similar, even if it's not an exact copy. Each flagged page is given a similarity score from 0 to 10. A score of 6 or 7 is generally not a concern, but scores of 9 or 10 may be worth addressing for SEO purposes. To see which pages are flagged and their scores, click on the page in the Site Auditor slideout. If the score is low and you'd prefer it not to affect your audit score, you can ignore the test in your settings.

These two numbers are measuring different things. The summary count shows how many internal links point to pages returning 4xx or 5xx errors. The Pages section shows all pages that returned those errors: including pages that only exist in your sitemap but aren't linked to from anywhere on the site.

If the Pages section shows more errors than the summary, it means some of those broken pages are sitemap-only: no internal links point to them directly.

To clean this up: remove the affected pages from your sitemap, and fix any internal links pointing to 4xx or 5xx pages.

AgencyAnalytics may show a lower word count than tools like Ahrefs. This is expected, our Site Auditor counts visible and hidden text (including text hidden by CSS, like hover menus), but excludes content that isn't considered part of the page document, such as video fallback text. Other tools may include this content in their count.

Your Site Audit page limit resets at midnight UTC, at the start of each billing cycle.

If you've allowed the RSiteAuditor user-agent in your robots.txt but are still seeing 4xx errors, your hosting provider's firewall, CDN, or security settings may be blocking the crawler based on request volume. A single allowed request might pass, but a full crawl can trigger bot detection.

Ask your hosting provider to whitelist the RSiteAuditor user-agent in their firewall or CDN settings. If you're on WP Engine, their Web Rules Engine can help, or contact WP Engine support directly.

Yes, you can set a crawl limit between 100 and 30,000 pages per audit in the client's Site Auditor settings. If you need to crawl more than 30,000 pages, contact our support team and we'll look into options for your account.

This error means your crawl limit for that client is set higher than the pages remaining in your account's monthly allowance. For example, if a client's crawl limit is set to 10,000 pages but only 9,977 pages are available in your account, the audit won't run.

To fix it, go to the client's Site Auditor settings and lower the Page Crawl Limit to match your available pages.

This error occurs when Site Auditor is connected, but the initial setup parameters were not finalized by clicking Save in the lower left corner of the settings window.

When this step is skipped, the system cannot initialize the audit task in the background.

This can be resolved by re-configuring the auditor settings and clicking Save.

If you have any questions, please contact our friendly support team by following these instructions! We're available 24/5 to help 😄