We’ve migrated our documentation to a new site, which means some URLs have changed.
Audience
Audience
Breadcrumbs

Content/Insight: Examples of markup-assisted data extraction

Introduction

The Piano Crawler retrieves web pages with the Piano script deployed. Upon being downloaded these pages are then parsed and a variety of data is extracted. Normally the document processing functionality is good at extracting the right data. However, sometimes, and especially for pages that do not conform to the regular news story format, the document processor may fail in retrieving or interpreting the data correctly. In those cases, we can help out by adding auxiliary tags to the pages.

Here are some typical sources for erroneous data processing:

  • The data is not there – for instance, the name of the author does not appear anywhere on the page

  • The data is tagged as something else – for instance, the title tag does not hold the title, but the newspaper's slogan: First in News

  • The data is bad/wrong – for instance, the og-image tag often contains links to images not well suited for use in a content recommendation.

  • The data has a wrong or unfamiliar format – for instance, the date is spelled out in Italian: 12 gennaio 2016

  • The web page has more than one URL - for instance, one and the same web page can be accessed as http://server.com/page.html and http://server.com/news/article.html

  • The data is encapsulated in an HTML structure not understood by the document processor – for instance, non-news articles material is included on the pages in such a way that the processor takes it for being part of the news article.

The top 5 issue types listed can be fixed by adding tags in the header section (meta tags or canonical link tags) that specify the right value whereas the last one can be fixed with content tags in the body section. More detailed background information on most of the tags listed below can be found at Document parsing and Piano Content - Review and Refinement.

Meta and canonical URL tags

If multiple meta tags holding the same type of data are detected (for instance cXenseParse:title, og:title, title), then the order of the below sections determines which data value will be retrieved. Tags listed in earlier sections take precedence over tags listed in later sections.

"Cxense" meta tags

These are Piano meta tags and they take presedence above any other tags holding the same information. Some of them are general (for instance cXenseParse:title) whereas others are feature specific (for instance Content Recommendation specific tag cXenseParse:recs:image).

  • cXenseParse:title

  • cXenseParse:author

  • cXenseParse:description

  • cXenseParse:keywords

  • cXenseParse:pageclass

  • cXenseParse:recs:image

  • cXenseParse:recs:publishtime

  • cXenseParse:recs:articleid

  • cXenseParse:recs:recommendable

  • cXenseParse:recs:recommending

  • cXenseParse:recs:category

  • cXenseParse:recs:expirationtime

Facebook open graph meta data

Facebook provides an API for apps to integrate with the Facebook Platform to post or retrieve Facebook data. Part of this API is the Open Graph protocol in which developers instruct Facebook about what is the story title, the story image, etc. The Piano Crawler will pick up the following Open Graph meta data unless the corresponding Piano meta tags are present:

  • og:title

  • og:description

  • og:keywords

  • og:image

  • og:article:author, og:book:author

Standard HTML headers and others

These are the last line of defense type of tags that will only be relied upon if all other means have failed (the exception being the title and the date that are given yet another chance in the body text, see example below)

  • title

  • description

  • keywords

  • date, dc.date, dc.date.created, dc.terms.issued, pub_date, article.published

  • author, dc.creator, article.author

Canonical URL tag

A canonical URL tag (link tag) should be added if one and the same web page can be accessed via more than one URL. Otherwise, the page will be registered as two or more independent documents in Insight, each of which will have its portion of the page views.

Date/time format

The data format to be used in the meta tags is to be in accordance with the extended format variant of ISO 8601 4.3.2 (a):

2012-03-01T13:00:00Z

  • The components are listed in this order:

    • year, month, day of the month, time designator, hour, minute, second, zone designator

  • The time designator is an upper case T

  • The zone designator is an upper case Z (UTC/Greenwich time) or a +/- time difference '+04', '06:00', '-08:30', etc)

  • The components have dashes between date components and colons between time components and + or - before zone (if not Z)

    • YYYY-MM-DDThh:mm:ss±hh:mm

    • YYYY-MM-DDThh:mm:ssZ

  • Even though not recommended due to feature dependencies, the notation can be truncated:

    • YYYY-MM-DDThh:mm:ssZ

    • YYYY-MM-DD

  • All components have to occupy the maximum possible number of characters by adding leading 0's (hence, 08 instead of 8 for the month of August, 01 instead of 1 for 1 PM)

    • 2016-08-21T01:00:00Z

  • 24 hours notation is used (thus 9:00 PM is 21:00)

    • 2016-08-21T21:00:00Z

Example showing a full meta-tag waterfall

Below follows an example HTML page showing the various levels of meta-tagging/data extraction.

<html>
  <head>
 
    <!-- CANONICAL URL TAG -->
    <link rel="canonical" href="http://www.domain.com/some/dir/somePage.html" />
 
    <!-- GENERAL META TAGS (CXENSE INSIGHT AND BEYOND) -->
 
    <!-- Page Class -->
    <meta name="cXenseParse:pageclass" content="article"/> <!-- The alternative content value is "frontpage" -->
 
    <!-- Title -->
    <meta name="cXenseParse:title" content="This could be the document's title"/> <!-- priority 1 -->
    <meta property="og:title" content="This could be the document's title"/> <!-- priority 2 -->
    <title>This could be the document's title</title> <!-- priority 3 -->
 
    <!-- Author -->
    <meta name="cXenseParse:author" content="John Smith"/> <!-- priority 1 -->
    <meta property="og:article:author" content="John Smith"/> <!-- priority 2 -->
    <meta property="og:book:author" content="John Smith"/> <!-- priority 3 -->
    <meta name="author" content="John Smith"/> <!-- priority 4 -->
    <meta name="dc.creator" content="John Smith"/> <!-- priority 5 -->
    <meta name="article.author" content="John Smith"/> <!-- priority 6 -->
 
    <!-- Description -->
    <meta name="cXenseParse:description" content="This could be the document's description"/> <!-- priority 1 --> 
    <meta property="og:description" content="This could be the document's description"/> <!-- priority 2 -->
    <meta name="description" content="This could be the document's description"/> <!-- priority 3 -->
 
    <!-- Keywords --> 
    <meta name="cXenseParse:keywords" content="This could be the document's keywords"/> <!-- priority 1 --> 
    <meta property="og:keywords" content="This could be the document's keywords"/> <!-- priority 2 -->
    <meta name="keywords" content="This could be the document's keywords"/> <!-- priority 3 --> 
 
    <!-- CXENSE CONTENT RECOMMENDATION RELATED META TAGGING -->
 
    <!-- Publication Date -->
    <meta name="cXenseParse:recs:publishtime" content="2012-03-01T13:00:00Z"/> <!-- priority 1 -->
    <meta property="article:published_time" content="2012-03-01T13:00:00Z"/> <!-- priority 2 -->
    <meta name="date" content="2012-03-01T13:00:00Z"/> <!-- priority 3 -->
    <meta name="dc.date" content="2012-03-01T13:00:00Z"/> <!-- priority 4 -->
    <meta name="dc.date.created" content="2012-03-01T13:00:00Z"/> <!-- priority 5 -->
    <meta name="dc.terms.issued" content="2012-03-01T13:00:00Z"/> <!-- priority 6 -->
    <meta name="pub_date" content="2012-03-01T13:00:00Z"/> <!-- priority 7 -->
    <meta name="article.published" content="2012-03-01T13:00:00Z"/> <!-- priority 8 -->
 
    <!-- Article id -->
    <meta name="cXenseParse:recs:articleid" content="24757772"/>
 
    <!-- Image -->
    <!-- There is no cXenseParse alternative for image -->
    <meta property="og:image" content="http://something/really/cool.jpg"/> <!-- priority 1 -->
  </head>
 
  <body>
    <h1>This could be the document's title</h1> <!-- Title, priority 4 (but just the first h1 tag on the page) -->
    <p>Mar 03, 2012, 8:34 PM ET </p> <!-- Date, priority 9 -->
    <h1>This is not the document's title</h1> <!-- Not considered for title -->
    <p>Some more text goes here</p>
  </body>
</html>

Content tags

An HTML page normally consists of a mixture of meaningful content and text that we don't care about such as footers, sidebars, and menus. The Piano Crawler normally does a good job at extracting the relevant content from an HTML page. This is particularly true when the page is a news article. However, there are times when is beneficial to help the crawler with identifying the content to be extracted.

Content tagging example

Relevant Content can be tagged with the html div attribute cXenseParse or with the HTML comments pair cxenseparse_start / cxenseparse_end anywhere within the <body> element.

<body>
        :
    <p>This is not relevant and will be ignored.</p>
    <div class="cXenseParse">
        <p>This is relevant and will not be ignored.</p>
    </div>
    <p>This is <div class="cXenseParse" style="display:inline">inlined and relevant</div>, but this is not.</p>
    <!-- cxenseparse_start -->
        <p>This is relevant and will not be ignored.</p>
    <!-- cxenseparse_end -->
        :
</body>

Custom markup

The purpose of all the markup covered so far has been to correct or in other ways compensate for the crawler's inability to extract the data that we want it to extract. In this section, we will look at the markup that is added for the purpose of supplying additional data beyond what the crawler extract by default.

Below see an example taken from an imaginary fruit website where each web page is tagged with the one or several fruits that the page content is covering.

<meta name="cXenseParse:xyz-fruit" content="orange"/>
<meta name="cXenseParse:xyz-fruit" content="apple"/>

The 'xyz' part of "cXenseParse:xyz-" is the customer prefix. Documents with this meta tag will end up with the key xyz-fruit in the content profile. This information can be viewed via the Piano Insight GUI and it is possible to make traffic queries and search queries based on it.

Recrawling updated pages

For any of the changes described above to have the intended effect, the page(s) must first be re-crawled and re-processed. The following 4 actions will make that happen:

  • Do nothing - you heard right, doing nothing will make your page get re-crawled. However, you may have to wait for quite sometime before that happens as there is no fixed time schedule (less than 60 days ought to be a safe guess though).

  • Do single page manual re-crawl - see figure below for how to re-crawl a single page that one has just updated. A good approach for testing one's changes during development.

  • Write a script - use the Piano(Cxense) API function /profile/content/push from within a Piano (Cxense) API script

  • Add re-crawl meta tag - Any page with a modified time meta tag with a time more recent than the previous crawl time will be re-crawled soon after the next user visit to the page:

<meta property="article:modified_time" content="2016-12-16T14:18:00-05:00">

Last updated: