We’ve migrated our documentation to a new site, which means some URLs have changed.
Audience
Audience
Breadcrumbs

Audience, Insight, Content: Document Parsing

Data categories

The data on a page can be roughly divided into two categories:

  1. Unstructured text, e.g., paragraphs of natural language text that you may want to semantically analyze. This is typically text found within the <title> and <body> sections of the HTML document.

  2. Structured data, e.g., (key, value) pairs where the semantics are already defined. This is typically data found in the <meta> tags in the HTML document.

Unstructured text

  • Sections

By default, the Piano crawler applies a parsing heuristic which is good at detecting blocks of relevant text and strips away navigation, sidebars, and other irrelevant content on the page. The heuristic works particularly well for pages that have layouts similar to news articles. In some cases, though, the default parsing heuristic might yield sub-optimal results, and you as a site owner might want to have explicit control over the fragments of the page detected as relevant. This can be achieved by marking up the relevant content within one or more <div> tags having the cXenseParse class.

<p>This is not relevant and will be ignored.</p>
<div class="cXenseParse"><p>This is relevant and will not be ignored.</p></div>
<p>This is <div class="cXenseParse" style="display:inline">inlined and relevant</div>, but this is not.</p>

If any <div> tags are found, the textual content within those tags (and no other unstructured text) will be parsed out from the document.

Note that, currently, text within nested <div> tags will be extracted multiple times.

As an alternative to using a cXenseParse div tag, we can also markup entire sections, as illustrated below. This markup allows you to explicitly define what page content is to be extracted, all other content will be disregarded.

Content that would otherwise be processed by the rules above can be explicitly excluded with the cxenseignore CSS class.

<p>This is not relevant and will be ignored.</p>
<!-- cxenseparse_start -->
<p>This is relevant and will not be ignored.</p>
<p class="cxenseignore">But this will be ignored</p>
<!-- cxenseparse_end -->

  • Descriptions

In addition to parsing out content from the title and body of the document, text is also parsed out from the <meta> tags named keywords, news_keywords, and description. This text is treated as unstructured.

<html>
  <head>
    <meta name="cXenseParse:description" content="This could be the document's description"/> <!-- Priority one. -->
    <meta property="og:description" content="This could be the document's description"/> <!-- Priority two. -->
    <meta name="description" content="This could be the document's description"/> <!-- Priority three. -->
 
    <meta name="cXenseParse:keywords" content="a keyword, another keyword"/> <!-- Priority one. -->
    <meta property="news_keywords" content="a keyword, another keyword"/> <!-- Priority two. -->
    <meta name="keywords" content="a keyword, another keyword"/> <!-- Priority three. -->
  </head>
  <body>
  </body>
</html>

  • Titles

The document title is treated as unstructured text and often plays an important role in the presentation. Document titles are therefore extracted and stored separately. The title extraction logic operates in the order of precedence shown below.

<html>
  <head>
    <meta name="cXenseParse:title" content="This could be the document's title"/> <!-- Priority one. -->
    <meta property="og:title" content="This could be the document's title"/> <!-- Priority two. -->
    <title>This could be the document's title</title> <!-- Priority three, unless it contains the domain name. -->
  </head>
  <body>
    <h1>This could be the document's title</h1> <!-- Priority four, but only if it's the first such tag, and only if it's fully contained within the document title -->
    <p>Some text goes here</p>
    <h1>This is not the document's title</h1> <!-- Not considered. -->
    <p>Some more text goes here</p>
  </body>
</html>

Structured data

In some cases, you as a publisher and site owner might want to explicitly provide Piano with structured data items to be put in the content profile as-is. This can be achieved by marking up the structured data items within one or more <meta> tags.

Adding <meta> tags requires that you are able to update the site HTML. When, for some reason, this is inconvenient or impossible, structured data items you want to put in the content profile as-is can also be provided via an API call.

Note the following:

  • The supplied keys and values will be normalized (upper case letters replaced with lower case) before they end up in the content profiles.

  • Keys not explicitly mentioned in this documentation must be prefixed with a Piano-assigned customer prefix. Please contact Piano support at support@piano.io for details.

  • If you choose to prefix with a string different than your Piano-assigned customer prefix, there is a risk that the data may become unavailable to you.

  • Keys may have a maximum length of 30 characters and should consist of a dash-separated prefix followed by the key name using characters a-z, 0-9, and _.

  • The values/items may contain letters of any character set, up to a maximum of 100 characters.

  • This markup mechanism should be used for structured data only, not unstructured data. Data that seems to violate it, e.g., because of length considerations, can get dropped.

  • If you have fields of unstructured text that you want to display as part of recommendations, make sure to prefix them with cxenseparse:recs.

  • Custom markup

You can supply arbitrary (key, value) pairs through <meta> tags with names having the cXenseParse: prefix, including the colon. The prefix gets stripped away and does not end up as part of the profile element keys. For example, to have the (key, value) pairs (xyz-fruit, orange), (xyz-fruit, apple) and (xyz-climate, tropical) as elements in the content profile, you can use the following HTML markup:

<meta name="cXenseParse:xyz-fruit" content="orange"/>
<meta name="cXenseParse:xyz-fruit" content="apple"/>
<meta name="cXenseParse:xyz-climate" content="tropical"/>

The special data-separator property can be used to split content based on the provided separator character. The markup above can be written like this:

<meta name="cXenseParse:xyz-fruit" content="orange,apple" data-separator=","/>
<meta name="cXenseParse:xyz-climate" content="tropical"/>

The data types number, time and geopoint (as semicolon pair of latitude/longitude) are supported.

<meta name="cXenseParse:number:xyz-temperature" content="30.1"/>
<meta name="cXenseParse:time:xyz-updated" content="2018-01-01T00:00:00Z"/>
<meta name="cXenseParse:geopoint:xyz-location" content="21.29;-157.72"/>

Keys defined via this markup mechanism (and not explicitly mentioned in this documentation) must be prefixed with an assigned customer prefix so that their visibility across Piano customers can become limited and cause less confusion. Prefixing with a different string than the assigned prefix causes a risk of inability to retrieve the data in the future. Please contact Piano support at support@piano.io for details.

The values in the custom markup content should have a maximum length of 100 characters.

Special exceptions might apply for the cXenseParse:recs: prefix. See, e.g., here for more information.

  • Page class

Based on properties like URL structure and internal HTML document, a typical page may be defined as article or frontpage. This labeling will have consequences for further processing stages and utilization of the page. For Piano Content, pages classified as frontpage are not recommended. The cXenseParse:pageclass meta tag can be used to explicitly classify a page as either an article or a frontpage. Note that no customer prefix is used in this case.

Add the following tag in the <head> of the page to include it in recommendations (classify as an article):

<meta name="cXenseParse:pageclass" content="article"/>

To exclude a page from recommendations (classify as a frontpage):

<meta name="cXenseParse:pageclass" content="frontpage"/>

This is particularly useful for preventing section pages, topic hubs, and homepages from being recommended while keeping true article pages eligible.

In cases where there is no cXenseParse:pageclass meta tag and the classification algorithm cannot determine whether the page is a frontpage or an article, specifying the Open Graph type article will result in a pageclass of article:

<meta property="og:type" content="article"/>

Use lowercase article for the og:type value. Some configurations require the value to be exactly "article" in lowercase to be correctly indexed.

You can use both tags together for compatibility with other platforms while ensuring correct Piano classification. When both are present, cXenseParse:pageclass takes precedence:

<meta property="og:type" content="article"/>
<meta name="cXenseParse:pageclass" content="article"/>

If you want non-article pages (for example, TV programs, films, or series pages) to appear in Piano Content recommendations, classify them as article using cXenseParse:pageclass.

Re-crawling required: After changing page classification tags, Piano must re-crawl those URLs before the updated classification takes effect. For broad changes (for example, adding tags across a large section or a new content type), a full site re-crawl may be required.

  • Canonical URL

The URL that identifies a document is normally based on a canonical URL. It is found from explicit cXenseParse:url markup, the canonical link or the open graph og:url metadata. The supported meta tags are, in order of priority.

<meta name="cXenseParse:url" content="http://www.foo.com/onion/123.html"/>
<link rel="canonical" href="http://www.foo.com/onion/123.html" />
<meta property="og:url" content="http://www.foo.com/onion/123.html"/>

  • Publication date

The publishing date of a document is a piece of metadata of particular importance for many applications. We recommend explicitly specifying the publishing date and time to ensure that it is correct, especially when using Content to increase the quality of recommendations.

Examples of meta tags to specify publishing date (in prioritized order) are given below. Dates are best specified using yyyy-MM-ddTHH:mm:ssZ (ISO 8601 4.3.2a complete representation of date and time using calendar dates in extended format), to avoid ambiguity.

<meta name="cXenseParse:publishtime" content="2012-03-01T13:00:00Z"/>
<meta name="cXenseParse:recs:publishtime" content="2012-03-01T13:00:00Z"/>

After cXenseParse there is an option for Piano customers in the tp notation. The other option's order is as follows:

<meta property="article:published_time" content="2012-03-01T13:00:00Z"/>
<meta name="date" content="2012-03-01T13:00:00Z"/>
<meta name="dc.date" content="2012-03-01T13:00:00Z"/>
<meta name="dc.date.created" content="2012-03-01T13:00:00Z"/>
<meta name="dc.terms.issued" content="2012-03-01T13:00:00Z"/>
<meta name="pub_date" content="2012-03-01T13:00:00Z"/>
<meta name="article.published" content="2012-03-01T13:00:00Z"/>
<meta itemprop="datePublished" content="2012-03-01T13:00:00Z"/>
<meta itemprop="og:article:published_time" content="2012-03-01T13:00:00Z"/>

As an alternative to meta tags, the HTML5 <time> element combined with the hAtom microformat related to specifying publishing dates is supported.

<p>This article was published on <time class="published" datetime="2012-03-01T13:00:00Z">March 1st, 2012</time>.</p>

If no explicit markup indicating publishing dates is present, a best-effort extraction heuristic is applied to the unstructured article body as a last resort.

Extracted publishing dates do not end up in the content profiles but are made available for use by Piano Content.

  • Modified date

The modified date of a document is extracted from article:modified_time.

<meta property="article:modified_time" content="2008-05-05T06:51:49.000Z"/>

If the article has been modified after the last crawling and contains the meta property modified_time (for more information, see Event data), then it will be recrawled as soon as possible.

  • Article identifiers

An article identifier serves as an external reference ID for a document. It is used in several products to automatically remove duplicates and merge events across duplicate URLs, in combination with information about canonical URLs.

The article identifier can only contain letters, digits and any of =@+-_./; no more than 64 characters are allowed.

<meta name="cXenseParse:articleid" content="24757772"/>
<meta name="cXenseParse:recs:articleid" content="24757772"/> <!-- Deprecated format. -->

  • Article authorship

The names of the authors of the document, e.g., the journalists that wrote the news story, can be captured and included in the content profiles by adding relevant meta tags. The supported options are, in order of priority:

<meta property="cXenseParse:author" content="John Smith"/>

After cXenseParse:author there is an option for Piano customers in the tp notation. The order is as follows:

<meta property="og:article:author" content="John Smith"/>
<meta property="article:author" content="John Smith"/>
<meta property="og:book:author" content="John Smith"/>
<meta property="book:author" content="John Smith"/>
<meta name="author" content="John Smith"/>
<meta name="dc.creator" content="John Smith"/>
<meta name="article.author" content="John Smith"/>

As an alternative to meta tags, subsets of the hAtom microformat (used by the hNews microformat) and the HTML5 rel tag related to article authorship are supported.

<p>This article was written by <span class="vcard author"><span class="fn">John Smith</span></span>.</p>
<p>This article was written by <a rel="author" href="/">John Smith</a>.</p>

Author value is limited to 50 characters

You could declare multiple authors (if applicable) within a single tag by using the "data-separator" attribute. Example:

<meta property="cXenseParse:author" content="John|Smith" data-separator="|"/>

If no data separator is specified, then two default separators , and ; are used. Only for the author.

  • Dominant image

The dominant image of a page is extracted and a thumbnail is generated to be uploaded to our CDN. It can be used in Piano Content when displaying returned recommendations.

If the dominant image of a page is known, we recommend marking up the page with the open graph og:image field. This will then be used as the dominant image. It is also possible to override og:image using cXenseParse:image.

<meta property="cXenseParse:image" content="http://www.foo.com/images/onion195x71.png"/>
<meta property="og:image" content="http://www.foo.com/images/onion195x71.png"/>

  • Overriding generated fields

The search fields taxonomy, company, person, location, concept, entity and classification are normally generated by content processing. They can however be overridden with meta tags prefixed with cXenseParse: in the following way:

<meta name="cXenseParse:taxonomy" content="news/sports/golf" />
<meta name="cXenseParse:company" content="Tiger Inc" />
<meta name="cXenseParse:person" content="Tiger Woods" />
<meta name="cXenseParse:location" content="California" />
<meta name="cXenseParse:concept" content="golf" />
<meta name="cXenseParse:entity" content="tgr" />
<meta name="cXenseParse:classification" content="sports" />

The fields are overridden in the sense that they will no longer be generated by content processing and instead take the provided values. Note that the category (and related fields) will be derived from the overriding taxonomy.

Multiple values can be specified by including multiple tags with the same property.

Content specific markup

Certain fields are specific to Piano Content and will mainly be used when providing recommendations.

  • Blocking a page from recommendation results

Pages that should not be returned in recommendations can be explicitly marked as not recommended. For section frontpages and similar, it is recommended to instead configure the pageclass as "frontpage" as they will also be removed from recommendations.

<meta name="cXenseParse:recs:recommendable" content="false"/>

If you do want to stop recommending a page after a certain date, you can specify an expiration time. This must be specified using the format yyyy-MM-ddTHH:mm:ssZ (ISO 8601 4.3.2a).

<meta name="cXenseParse:recs:expirationtime" content="2014-03-11T23:00:00.000Z"/>

  • Disabling contextual recommendations

On pages that contain sensitive topics with words with multiple meanings, it is possible to explicitly say that we do not wish to display contextual recommendations at all. As an example, for articles about a natural disaster in some city, you might not want to suggest recommendations from the travel section about going on vacation to that city. So for such articles, you can explicitly mark them up to not deliver contextual recommendations.

<meta name="cXenseParse:recs:recommending" content="false"/>

  • Troubleshooting page classification

If pages are not being recommended as expected, work through the following checks.

  1. Confirm classification values

    • Article pages intended for recommendations should use content="article".

    • Section, homepage, and topic pages you want excluded should use content="frontpage".

  2. Check for accidental exclusion

    • Article pages mistakenly tagged with <meta name="cXenseParse:pageclass" content="frontpage"/> will be excluded from recommendations.

  3. Validate exact tag values

    • Ensure og:type uses lowercase article when used as a fallback.

  4. Confirm a re-crawl has occurred

    • Tag changes do not take effect until the crawler reprocesses the affected pages. If updated pages are still not appearing correctly, verify that the meta tags are present in the rendered HTML and request a re-crawl if needed.

Last updated: