Seeing this error means that our bot encountered an unexpected issue when trying to reach that page or domain.

There might be a bug or server misconfiguration blocking the request, even though the page may actually exist on the server. This is different from a 404 error which means the page simply doesn't exist on the server.

If you're unsure whether our site auditor is reporting correctly, you can manually verify the link status at https://httpstatus.io/. If this page confirms a 500 error, then we're reporting it correctly at our end.

If the dead link checker does not show a 500 error, this indicates that the web server where the site resides is likely blocking our crawler. You'll need to whitelist our useragent in your web host's firewall settings.

How to find our crawler's useragent

The useragent is specified in the site audit crawl settings, and is usually on the Default setting. In this case, the useragent is: Mozilla/5.0 (compatible; aa/1.0)

How to whitelist our crawler's useragent

You may need to contact your web host or web developer in order to make this change as security policies vary between websites.

Once you've white listed the user agent, you should try running the site audit again (ensuring you've chosen the white listed user agent when running the crawl).

If you are still seeing a problem after doing this, we recommend that you consult your web host or web developer to check the server logs and see what is ultimately causing the issue.

What's next:

Did this answer your question?