Overview
Composer 1x is the result of the integration of Piano Composer and Piano Audience. The crawling process described here is also relevant for Piano Content and Piano Insight.
A page is identified by means of URL. It is often the case that the same page is known under many different URLs, typically because the URL contains many parameters that are irrelevant to the retrieval or display of the page. To this end, the platform including Piano Audience, Insight, and Content applies a number of URL normalization heuristics designed to detect if two or more different URLs all really refer to the same logical page. In the following, when we talk about the number of pages unless otherwise stated, such numbers are to be interpreted as being post-normalization.
Pages are periodically recrawled to ensure that we have an up-to-date view of the content. New pages are fetched immediately, as capacity allows. Note, however, that we currently require that a new page has been viewed at least 3 times (within a window of approximately 6 hours) before it is crawled. If the article has been modified after the last crawling and contains the meta property modified_time (see Event data), then it will be recrawled as soon as possible.
Individual pages can be triggered for recrawling manually on demand through the Insight GUI, or through an API method invocation.
Rate limiting for force push requests
To protect the stability and performance of the crawling infrastructure, rate limiting is applied to force push requests made through the /profile/content/push API.
The rate limit is enforced per URL. If the same URL is pushed multiple times in rapid succession, only a limited number of re-crawls will be processed within a given time window. For example, if a publisher pushes the same URL every second, only approximately 4 crawl requests will be honored within one hour. Any additional force push requests for that URL during this period will be declined.
This rate limit applies only to repeated pushes of the same URL. Pushing different URLs is not affected by this limit.
Best practices for using force push:
-
Only push a URL when the content has actually changed.
-
Avoid pushing the same URL more than a few times per day.
If you are pushing multiple URLs, include a short delay (0.5–1 second) between requests to avoid overwhelming the API.
It is not possible to add additional data into the crawler.
External resources
Most crawled pages will reference external resources such as images and scripts that are loaded and possibly executed as the page gets rendered. Such external resources will normally be loaded and possibly executed, but Piano reserves the right to block the loading of external resources that are known to be problematic for some reason. Examples include third-party analytics tags (where counts would otherwise get inflated) or external resources residing on slow or irresponsive domains (where loading would significantly slow down crawler throughput).
If a customer wants to prevent a certain part of their code to be run by the Piano crawler, then that code can be wrapped with a condition as shown here:
if (navigator.userAgent.toLowerCase().indexOf('cxensebot') == -1) { // Code here will not be run by the Piano Crawler }
Paywall-protected content
There are several methods to allow our web robot access to paywall-protected content. The exact details of doing this depend on details of the paywall solution, but the paywall will need to be able to identify if a particular visit comes from the Piano web robot or not, so that suitable access can be granted. This section lists, in decreasing order of preference, several strategies that the paywall can use for this.
User-Agent header
Visits from the Piano web robot can be clearly identified by looking at the User-Agent HTTP header: Simply look for the text cXensebot as part of the overall User-Agent string.
Reverse DNS lookup
Visits from the Piano web robot will come from IP addresses that reverse-resolve to *.cxensebot.com. This is a more future-proof check than using IP addresses but is not supported by all systems.
IP addresses
Visits from the Piano web robot will come from any of the IP addresses listed below.
This list of IP addresses may change in the future. The list will normally be updated at least one week before the new IP addresses are used for crawling. Please monitor this page if you use this identification method.
5.9.190.101
5.9.190.106
5.9.190.107
5.9.190.127
For simplicity, you can also use the network specifications 5.9.190.96/27 or 5.9.190.96, netmask 255.255.255.224.
Basic access authentication
In special circumstances it is possible to use HTTP Basic Access Authentication as described in RFC 1945 section 11.1.
This is generally not a recommended approach. Please contact your account manager if this is the only suitable solution for your site.
Soft 404 errors
Some pages deliver a "not found" error under the guise of an HTTP status code 200 OK, instead of the usual 404 NOT FOUND. This is known as a soft 404 error. The crawler attempts to detect such errors by analyzing the page content, looking for the presence of prominently positioned sentences such as "Page not found".