As part of automating the content ingestion process, your site may be visited by the Cxense web robot.
The web robot will only fetch the pages where the Insight JavaScript snippet is present.
A page is identified by means of an URL. It is often the case that the same page is known under many different URLs, typically because the URL contains many parameters that are irrelevant to the retrieval or display of the page. To this end, the Cxense platform applies a number of URL normalization heuristics designed to detect if two or more different URLs all really refer to the same logical page. In the following, when we talk about the number of pages, unless otherwise stated such numbers are to be interpreted as being post-normalization.
Pages are periodically recrawled to ensure that we have an up-to-date view of the content. New pages are fetched immediately, as capacity allows. Note, however, that we currently require that a new page has been viewed at least 3 times (within a window of approximately 6 hours) before it is crawled. If the article has been modified after the last crawling and contains the meta property modified_time (see Event data), then it will be recrawled as soon as possible.
Individual pages can be triggered for recrawling manually on demand through the Insight GUI, or through an API method invocation.
External resources
Most crawled pages will reference external resources such as images and scripts that are loaded and possibly executed as the page gets rendered. Such external resources will normally be loaded and possibly executed, but Cxense reserves the right to block the loading of external resources that are known to be problematic for some reason. Examples include third-party analytics tags (where counts would otherwise get inflated) or external resources residing on slow or irresponsive domains (where loading would significantly slow down crawler throughput).
If a customer wants to prevent a certain part of their code to be run by the Cxense crawler, then that code can be wrapped with a condition as shown here:
if (navigator.userAgent.toLowerCase().indexOf('cxensebot') == -1) {
// Code here will not be run by the Cxense Crawler
}
Paywall-protected content
There are several methods to allow our web robot access to paywall-protected content. The exact details of doing this depends on details of the paywall solution, but the paywall will need to be able to identify if a particular visit comes from the Cxense web robot or not, so that suitable access can be granted. This section lists, in decreasing order of preference, several strategies that the paywall can use for this.
User-Agent header
Visits from the Cxense web robot can be clearly identified by looking at the User-Agent HTTP header: Simply look for the text cXensebot as part of the overall User-Agent string.
Reverse DNS lookup
Visits from the Cxense web robot will come from IP addresses that reverse-resolve to *.cxensebot.com. This is a more future-proof check than using IP addresses, but is not supported by all systems.
IP addresses
Visits from the Cxense web robot will come from any of the IP addresses listed below.
This list of IP addresses may change in the future. The list will normally be updated at least one week before the new IP addresses are used for crawling. Please monitor this page if you use this identification method.
5.9.190.101
5.9.190.106
5.9.190.107
5.9.190.127
For simplicity, you can also use the network specifications 5.9.190.96/27 or 5.9.190.96, netmask 255.255.255.224.
Basic access authentication
In special circumstances it is possible to use HTTP Basic Access Authentication as described in RFC 1945 section 11.1.
This is generally not a recommended approach. Please contact your account manager if this is the only suitable solution for your site.
Soft 404 errors
Some pages deliver a "not found" error under the guise of an HTTP status code 200 OK, instead of the usual 404 NOT FOUND. This is known as a soft 404 error. The crawler attempts to detect such errors by analyzing the page content, looking for the presence of prominently positioned sentences such as "Page not found".