Many search engine optimization (SEO) companies (such as ahrefs or semrush) use a web crawler (or "bot") to index pages on your site to collect data for SEO recommendations.
coreFORCE includes a number of features to aggressively block hacking attempts and Denial of Service attacks. These features may interfere with the desired functionality of a SEO crawler, for one of several reasons:
- The web crawler may be overcrawling your site (that is, requesting pages too rapidly one after another). This behavior will cause coreFORCE to blacklist the bot after a few minutes. To resolve this, the SEO company may need to configure their crawler to reduce its rate for your site.
- The web crawler may be running from an IP address outside of the US or Canada. By default, a coreFORCE site that sells firearms is only available to IP addresses in the USA and Canada. To resolve this, there are two preferences that need to be set: Allow World Access = true, and US Canada Access only = false. Contact support if you are unable to set these preferences in coreFORCE.
- Due to a history of overcrawling by certain bots, such as semrush, the bot may have been explicitly added to a list of "hacking terms" which automatically result in blacklisting. In this case, it will show up in the IP address blacklist in coreFORCE. Contact support to remove the entries from the blacklist and to remove the bots name from the list of hacking terms.
If a bot triggers one of these features, it will get a 503 (Service Unavailable) error.
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article