Crawling audit HTTP status codes and errors
Crawling audits flag every case where Marfeel crawlers cannot obtain a satisfactory response from servers when requesting main article HTML or related assets (images, author pages, and similar resources). Some of these errors are caused by WAFs blocking Marfeel crawlers, which can be resolved by whitelisting the IPs from which they crawl all the content. For further diagnosis of crawler-related failures, see the editorial crawling troubleshooting guide.
Connection failed
Section titled “Connection failed”code: connectionFailed
The system has not been able to validate the information in these URLs.
Connection timed out
Section titled “Connection timed out”code: connectionTimeout
The system has received a timeout when trying to connect to these URLs.
Received an empty response
Section titled “Received an empty response”code: emptyResponse
The system has not been able to trace the content of these pages as they are apparently blank. Please double-check to ensure this is the expected behaviour.
Redirection to same URL found
Section titled “Redirection to same URL found”code: selfRedirection
URL responds with a redirection HTTP code (3XX) and a location header pointing to the same URL, which results in an endless loop.
301 response code
Section titled “301 response code”The system has found 301 redirects in these URLs. This type of redirect tells search engines such as Google that it should ignore the crawled URL and is now part of a different URL.
This does not have to be incorrect, but we recommend that you double-check the behaviour to make sure that this is what you were looking for.
302 response code
Section titled “302 response code”The system has found 302 redirects in these URLs. This type of redirect tells search engines such as Google that it should ignore the crawled URL and is now part of a different URL.
This does not have to be incorrect, but we recommend that you double-check the behaviour to make sure that this is what you were looking for.
400 response code
Section titled “400 response code”Your server has returned a 400 HTTP status code (Bad Request) for these URLs.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
401 response code
Section titled “401 response code”Your server has returned a 401 HTTP status error code (Unauthorized) for these URLs.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
403 response code
Section titled “403 response code”Your server has returned a 403 HTTP status error code (Forbidden) for these URLs.
Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
404 response code
Section titled “404 response code”Your server has returned a 404 HTTP status error code (Not Found) for these URLs.
This behaviour is not necessarily wrong, especially if your intention is that this content is not visible to anyone. Double-check these URLs since both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
410 response code
Section titled “410 response code”code: 410
Your server has returned a 410 HTTP status error code (Gone) for these URLs.
This behaviour is not necessarily wrong, especially if your intention is that this content should not be visible to anyone. Double-check these URLs, since both search engines and social networks will not be able to index or publish this content correctly as this error is considered invalid for the functioning of their systems.
429 response code
Section titled “429 response code”Your server has returned a 429 HTTP status error code (Too Many Requests) for these URLs.
Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
451 response code
Section titled “451 response code”Your server has returned a 451 HTTP status error code (Unavailable For Legal Reasons) for these URLs.
Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
500 response code
Section titled “500 response code”Your server has returned a 500 HTTP status error code (Internal Server Error) for these URLs.
Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
503 response code
Section titled “503 response code”Your server has returned a 503 HTTP status error code (Service Unavailable) for these URLs.
Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent, as this could be the reason for the issue detected.
Both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.
image.url did not return an HTTP 200 response
Section titled “image.url did not return an HTTP 200 response”code: image_url_correct_response
The url property is a required property of the image structured data type which, in turn, is part of the Article type. This image is used by some search engines such as Google to identify the main image to be used in Top Stories, Google News and Google Discover carousels and other Google products.
The system has found problems accessing the image. Make sure that Google can access the URL correctly, otherwise it can negatively impact the performance of your Search and Discover traffic.
What causes crawling audit errors in Marfeel?
Crawling audit errors occur when Marfeel crawlers cannot obtain a satisfactory response from servers when requesting article HTML or related assets such as images and author pages. Common causes include WAFs blocking Marfeel crawlers, server timeouts, HTTP error status codes (4xx and 5xx), redirect loops, and empty responses.
Why do new audit errors appear after fixing a crawling issue?
Any URL that results in a crawling error is not analyzed further, so other audits do not run on that page. Once a crawling issue is resolved, the full audit suite becomes available for that URL, which may reveal additional errors that were previously hidden.
How do WAFs affect Marfeel crawling audits?
Web Application Firewalls (WAFs) can block Marfeel Compass crawlers, resulting in HTTP 403, 429, 451, 500, or 503 errors. You can resolve this by whitelisting the IP addresses from which Marfeel crawlers operate.