An important part of ensuring that analytics, targeting and recommendations are relevant is that we can detect and parse out the relevant content from a page's HTML. An HTML page is full of text that we normally don't care about, e.g., footers, sidebars and menus. Thus, an intelligent HTML parser needs to be able to identify and extract the relevant portions of text on the page so that these can be semantically analyzed and indexed.
We can roughly divide the data on a page into two categories:
-
Unstructured text, e.g., paragraphs of natural language text that we may want to semantically analyze. This is typically text found within the HTML document's
<title>and<body>sections. -
Structured data, e.g., (key, value) pairs where the semantics are already defined. This is typically data found in the
<meta>tags in the HTML document.
Unstructured text
Sections
By default, the Cxense crawler applies a parsing heuristic that is good at detecting blocks of relevant text and strips away navigation, sidebars, and other irrelevant content on the page. The heuristic works particularly well for pages that have layouts similar to news articles. In some cases, though, the default parsing heuristic might yield sub-optimal results, and you as a site owner might want to have explicit control over which fragments of the page that are detected as being relevant. This can be achieved by marking up the relevant content within one or more <div> tags having the class cXenseParse.
<p>This is not relevant and will be ignored.</p>
<div class="cXenseParse"><p>This is relevant and will not be ignored.</p></div>
<p>This is <div class="cXenseParse" style="display:inline">inlined and relevant</div>, but this is not.</p>
If any such <div> tags are found, the textual content within those tags (and no other unstructured text) will be parsed out from the document.
Note that, currently, text that occurs within nested such <div> tags will be extracted multiple times.
As an alternative to using a cXenseParse div tag, we can also markup entire sections, as illustrated below. This markup allows the customer to explicitly define what page content that is to be extracted, all other content will be disregarded.
Content that would otherwise be considered for processing by the rules above can be explicitly excluded with the CSS class cxenseignore.
<p>This is not relevant and will be ignored.</p>
<!-- cxenseparse_start -->
<p>This is relevant and will not be ignored.</p>
<p class="cxenseignore">But this will be ignored</p>
<!-- cxenseparse_end -->
Descriptions
In addition to parsing out content from the title and body of the document, text is also parsed out from the <meta> tags named keywords, news_keywords and description. This text is treated as being unstructured.
<html>
<head>
<datalayer-description> <-- this value is taken from composer if it is enabled --> <!-- Priority one. -->
<meta name="cXenseParse:description" content="This could be the document's description"/> <!-- Priority two. -->
<meta property="og:description" content="This could be the document's description"/> <!-- Priority three. -->
<meta name="description" content="This could be the document's description"/> <!-- Priority four. -->
<datalayer-keywords> <-- this value is taken from composer if it is enabled --> <!-- Priority one. -->
<meta name="cXenseParse:keywords" content="a keyword, another keyword"/> <!-- Priority two. -->
<meta property="news_keywords" content="a keyword, another keyword"/> <!-- Priority three. -->
<meta name="keywords" content="a keyword, another keyword"/> <!-- Priority four. -->
</head>
<body>
</body>
</html>
Titles
A document's title is treated as unstructured text, and can serve an important role for presentation purposes. Document titles are therefore extracted and stored separately. The title extraction logic operates in the order of precedence shown below.
<html>
<head>
<datalayer-title> <-- this value is taken from composer if it is enabled --> <!-- Priority one. -->
<meta name="cXenseParse:title" content="This could be the document's title"/> <!-- Priority two. -->
<meta property="og:title" content="This could be the document's title"/> <!-- Priority three. -->
<title>This could be the document's title</title> <!-- Priority four, unless it contains the domain name. -->
</head>
<body>
<h1>This could be the document's title</h1> <!-- Priority four, but only if it's the first such tag, and only if it's fully contained within the document title -->
<p>Some text goes here</p>
<h1>This is not the document's title</h1> <!-- Not considered. -->
<p>Some more text goes here</p>
</body>
</html>
Structured data
In some cases you as a publisher and site owner might want to explicitly provide Cxense with structured data items that you want to end up in the content profile as-is. This can be achieved by marking up the structured data items within one or more <meta> tags.
Adding meta tags requires that you can update the site's HTML. In cases where this for some reason is inconvenient or impossible, structured data items that you want to end up in the content profile as-is can also be provided via an API call.
Note the following:
-
The supplied keys and values will be normalized (i.e. upper case letters will be normalized to lower case) before they end up in the content profiles.
-
Keys not explicitly mentioned in this documentation must be prefixed with a Cxense-assigned customer prefix. Please contact support@cxense.com for details.
-
If you choose to prefix with a string different than your Cxense-assigned customer prefix, you risk that the data become unavailable to you.
-
The keys may have a maximum length of 30 characters, and should consist of the dash-separated prefix, followed by the key name, using characters
a-z,0-9and_. -
The values/items may contain letters of any character set, up to a maximum of 100 characters.
-
This markup mechanism should be used for structured data only, not unstructured data. Data that seems to violate this, e.g., because of length considerations, can get dropped.
Custom markup
You can supply arbitrary (key, value) pairs through meta tags with names having the prefix cXenseParse:, including the colon. The prefix cXenseParse: gets stripped away and does not end up as part of the profile element keys. For example, to have the (key, value) pairs (xyz-fruit, orange), (xyz-fruit, apple) and (xyz-climate, tropical) appear as elements in the content profile, the page could contain the following HTML markup:
<meta name="cXenseParse:xyz-fruit" content="orange"/>
<meta name="cXenseParse:xyz-fruit" content="apple"/>
<meta name="cXenseParse:xyz-climate" content="tropical"/>
The special data-separator property can be used to split content based on the provided separator character. The markup above can be written like this:
<meta name="cXenseParse:xyz-fruit" content="orange,apple" data-separator=","/>
<meta name="cXenseParse:xyz-climate" content="tropical"/>
The data types number, time and geopoint (as semicolon pair of latitude/longitude) are supported
<meta name="cXenseParse:number:xyz-temperature" content="30.1"/>
<meta name="cXenseParse:time:xyz-updated" content="2018-01-01T00:00:00Z"/>
<meta name="cXenseParse:geopoint:xyz-location" content="21.29;-157.72"/>
Keys defined via this markup mechanism (and not explicitly mentioned in this documentation) must be prefixed with a Cxense-assigned customer prefix, so that their visibility across Cxense customers becomes limited and causes less confusion. By prefixing with a different string than the assigned prefix you risk losing access to retrieve the data in the future. Please contact support@cxense.com for details.
The values in the custom markup content should have a maximum length of 100 characters.
Special exceptions might apply for the cXenseParse:recs: prefix. See, e.g., [here](page://Cxense Content - Review and Refinement) for more information.
Page class
Based on properties such a URL structure and the internal HTML document, a typical page may be classified as article or frontpage. This labeling will have consequences for the further processing stages and utilization of the page. For Cxense Content a page classified as frontpage is not recommendable. The custom markup tags cXenseParse can be used to define that a page is either article or frontpage. This can be achieved with the meta tag described below. Note that the customer prefix is not used in this case.
<meta name="cXenseParse:pageclass" content="article"/>
<meta name="cXenseParse:pageclass" content="frontpage"/>
In cases where there is no cXenseParse:pageclass meta tags, and our classification algorithm is uncertain if a page is a front page or an article. Specifying the Open Graph type "article" will result in a pageclass of "article".
<meta property="og:type" content="article"/>
Canonical URL
The URL that identifies a document is normally based on the canonical URL. This is found from either explicit cXenseParse:url markup, canonical link or the open graph og:url meta data. The supported meta tags are, in prioritized order:
<meta name="cXenseParse:url" content="https://www.foo.com/onion/123.html"/>
<link rel="canonical" href="https://www.foo.com/onion/123.html" />
<meta property="og:url" content="https://www.foo.com/onion/123.html"/>
Publication date
A document's publishing date is a piece of meta data of particular importance in many applications. We recommend explicitly specifying the publishing date and time to ensure that it is correct, especially when using Content to increase quality of recommendations.
Examples of meta tags to specify the publishing date are given below in prioritized order. Dates are best specified using yyyy-MM-ddTHH:mm:ssZ (ISO 8601 4.3.2a complete representation of date and time of day using calendar dates in extended format), to avoid ambiguities.
<meta name="cXenseParse:publishtime" content="2012-03-01T13:00:00Z"/>
<meta name="cXenseParse:recs:publishtime" content="2012-03-01T13:00:00Z"/>
After cXenseParse there is an option for Piano customers in the tp notation. Then in the following order:
<meta property="article:published_time" content="2012-03-01T13:00:00Z"/>
<meta name="date" content="2012-03-01T13:00:00Z"/>
<meta name="dc.date" content="2012-03-01T13:00:00Z"/>
<meta name="dc.date.created" content="2012-03-01T13:00:00Z"/>
<meta name="dc.terms.issued" content="2012-03-01T13:00:00Z"/>
<meta name="pub_date" content="2012-03-01T13:00:00Z"/>
<meta name="article.published" content="2012-03-01T13:00:00Z"/>
<meta itemprop="datePublished" content="2012-03-01T13:00:00Z"/>
<meta itemprop="og:article:published_time" content="2012-03-01T13:00:00Z"/>
As an alternative to meta tags, the HTML5 <time> element combined with the hAtom microformat related to specifying publishing dates is supported.
<p>This article was published on <time class="published" datetime="2012-03-01T13:00:00Z">March 1st, 2012</time>.</p>
If no explicit markup indicating publishing dates are present, a best-effort extraction heuristic is applied to the unstructured article body as a last resort.
Extracted publishing dates do not end up in the content profiles, but are made available for use by Cxense Content and Cxense Search.
Modification date
A document's modification date is extracted from article:modified_time.
<meta property="article:modified_time" content="2008-05-05T06:51:49.000Z"/>
Article identifiers
An article identifier is used as an external reference ID for a document. This is used in several products to automatically remove duplicates and merge events across duplicate URLs, in combination with information about canonical URL.
The article identifier can only contain letters, digits and any of =@+-_./, and no more than 64 characters are allowed.
<meta name="cXenseParse:articleid" content="24757772"/>
<meta name="cXenseParse:recs:articleid" content="24757772"/> <!-- Deprecated format. -->
Article authorship
The author of the document, e.g., the names of the journalists that wrote the news story, can be captured and included in the content profiles by adding relevant meta tags. The supported options are, in prioritized order:
<datalayer-authors> <-- this value is taken from composer if it is enabled --> <!-- Priority one. -->
<meta property="cXenseParse:author" content="John Smith"/> <!-- Priority two. -->
After cXenseParse:author there is an option for Piano customers in the tp notation. Then in the following order:
<meta property="og:article:author" content="John Smith"/>
<meta property="article:author" content="John Smith"/>
<meta property="og:book:author" content="John Smith"/>
<meta property="book:author" content="John Smith"/>
<meta name="author" content="John Smith"/>
<meta name="dc.creator" content="John Smith"/>
<meta name="article.author" content="John Smith"/>
As an alternative to meta tags, subsets of the hAtom microformat (used by the hNews microformat) and the HTML5 rel tag related to article authorship are supported.
Author value is limited to 50 characters
You could declare multiple authors (if applicable) within single tag by using "data-separator" attribute. Example:
<meta property="cXenseParse:author" content="John|Smith" data-separator="|"/>
<p>This article was written by <span class="vcard author"><span class="fn">John Smith</span></span>.</p>
<p>This article was written by <a rel="author" href="/">John Smith</a>.</p>
Lastly, selected Google+ links are also interpreted as indicating article authorship.
<p>This article was written by <a href="https://plus.google.com/12345678?rel=author">John Smith</a>.</p>
Dominant image
The dominant image of a page is extracted and a thumbnail generated and uploaded to our CDN. This can be used in Content when displaying the returned recommendations.
If the dominant image of the page is known, we recommend marking up the page with the open graph og:image field. This will then be used as the dominant image. It is also possible to override og:image using cXenseParse:image.
<meta property="cXenseParse:image" content="https://www.foo.com/images/onion195x71.png"/>
<meta property="og:image" content="https://www.foo.com/images/onion195x71.png"/>
Overriding generated fields
The search fields taxonomy, company, person, location, concept, entity and classification are normally generated by content processing. They can however be overridden with meta tags prefixed with cXenseParse: in the following way:
<meta name="cXenseParse:taxonomy" content="news/sports/golf" />
<meta name="cXenseParse:company" content="Tiger Inc" />
<meta name="cXenseParse:person" content="Tiger Woods" />
<meta name="cXenseParse:location" content="California" />
<meta name="cXenseParse:concept" content="golf" />
<meta name="cXenseParse:entity" content="tgr" />
<meta name="cXenseParse:classification" content="sports" />
The fields are overridden in the sense that they will no longer be generated by content processing, and instead take on the provided values. Note that the category (and related fields) will be derived from the overriding taxonomy.
Multiple values can be specified by including multiple tags with the same property.
Content specific markup
Certain fields are specific to Content and will mainly be used when providing recommendations.
Block page from recommendation results
Pages that should not be returned in recommendations can be explicitly marked as not recommendable. For section front pages and similar it is recommended to instead configure the pageclass as "frontpage" as these will also be removed from recommendations.
<meta name="cXenseParse:recs:recommendable" content="false"/>
If you do not want to recommend a page after a certain date, you can specify an expiration time. This must be specified using the format yyyy-MM-ddTHH:mm:ssZ (ISO 8601 4.3.2a).
<meta name="cXenseParse:recs:expirationtime" content="2014-03-11T23:00:00.000Z"/>
Disable contextual recommendations
On pages that contain sensitive topics with words with multiple meanings, it is possible to explicitly say that we do not wish to display contextual recommendations at all. As an example, for articles about a natural disaster in some city, you might not want to suggest recommendations from the travel section about going on vacation to that city. So for such articles you can explicitly mark them up not to deliver contextual recommendations.
<meta name="cXenseParse:recs:recommending" content="false"/>
Data Layer extension for Content metadata
Contains a common contract between various Piano client SDKs. There’s a list of fields we collect on Composer behalf.
<ol> <li>id</li> <li>type</li> <li>zone</li> <li>createdAt</li> <li>modifiedAt</li> <li>authors</li> <li>section</li> <li>tags</li> <li>keywords</li> <li>title</li> <li>description</li> <li>isNative</li> </ol>
It takes values according to the given priority.
<ol> <li>cxenseParse</li> <li>meta tags</li> <li>ld+json</li> </ol>