Markup Assisted Data Extraction Examples

English | Español | 日本語

THIS PAGE IS DEPRECATED

For up-to-date information, please follow this link.

Introduction Meta and Canonical URL Tags Cxense Meta Tags Facebook Open Graph Meta Data Standard HTML Headers and Others Canonical Url Tag Date/Time Format Example Showing Full Meta Tag Waterfall Content Tags Content Tagging Example Custom Markup Recrawling Updated Pages

Introduction

The Cxense Crawler retrieve web pages with the Cxense script deployed. Upon being downloaded these pages are then parsed and a variety of data is extracted. Normally the document processing functionality is good at extracting the right data. However, sometimes, and especially for pages that do not confirm to the regular news story format, the document processor may fail in retrieving or interpreting the data correctly. In those cases we can help out by adding auxiliary tags to the pages.  

Here are some typical sources for erroneous data processing:

• The data is not there – for instance, the name of the author does not appear anywhere on the page

• The data is tagged as something else – for instance, the title tag does not hold the title, but the newspaper's slogan: First in News

• The data is bad/wrong – for instance, the og-image tag often contains links to images not well suited for use in a content recommendation.

• The data has a wrong or unfamiliar format – for instance, the date is spelled out in Italian: 12 gennaio 2016

• The web page has more than one url - for instance, one and the same web page can be accessed as http://server.com/page.html and http://server.com/news/article.html
  

• The data is encapsulated in a html structure not understood by the document processor – for instance, non-news articles material is included on the pages in such a way that the processor takes it for being part of the news article.

The top 5 issue types listed can be fixed by adding tags in the header section (meta tags or canonical link tags) that specify the right value whereas the last one can be fixed with content tags in the body section. More detailed background information on most of tags listed below can be found at Document parsing and Cxense Content - Review and Refinement.

Meta and Canonical URL Tags

If several meta tags that hold the same type of data are detected (for instance cXenseParse:title, og:title, title), then the section order below decides which data value that will be retrieved. Tags listed in earlier sections take precedence over tags listed in later sections.

Cxense Meta Tags

These are Cxense meta tags and they take presedence above any other tags holding the same information. Some of them are general (for instance cXenseParse:title) whereas others are feature specific (for instance Content Recommendation specific tag cXenseParse:recs:image).

• cXenseParse:title

• cXenseParse:author

• cXenseParse:description

• cXenseParse:keywords

• cXenseParse:pageclass

• cXenseParse:recs:image

• cXenseParse:recs:publishtime

• cXenseParse:recs:articleid

• cXenseParse:recs:recommendable

• cXenseParse:recs:recommending

• cXenseParse:recs:category

• cXenseParse:recs:expirationtime

Facebook Open Graph Meta Data

Facebook provides an API for apps to integrate with the Facebook Platform to post or retrieve Facebook data. Part of this API is the Open Graph protocol in which developers instruct Facebook about what is the story title, the story image, etc. The Cxense Crawler will pick up the following Open Graph meta data unless the corresponding Cxense meta tags are present:

• og:title

• og:description

• og:keywords

• og:image

• og:article:author, og:book:author

Standard HTML Headers and Others

These are last line of defence type of tags that will only be relied upon if all other means have failed (the exception being the title and the date that are given yet another chance in the body text, see example below)

• title

• description

• keywords

• date, dc.date, dc.date.created, dc.terms.issued, pub_date, article.published

• author, dc.creator, article.author

Canonical Url Tag

A canonical url tag (link tag) should be added if one and the same web page can be accessed via more than one URL. Otherwise the page will be registered as two or more independent documents in Insight, each of which will have its portion of the page views.

Date/Time Format

The data format to be used in the meta tags is to be in accordance with the extended format variant of ISO 8601 4.3.2 (a):

• The components are listed in this order: 

  • year, month, day of the month, time designator, hour, minute, second, zone designator

• The time designator is an upper case T

• The zone designator is an upper case Z (UTC/Greenwich time) or a +/- time difference '+04', '06:00', '-08:30', etc)

• The components have dashes between date components and colons between time components and + or - before zone (if not Z)
  

  • YYYY-MM-DDThh:mm:ss±hh:mm

  • YYYY-MM-DDThh:mm:ssZ

• Even though not recommended due to feature dependencies, the notation can be truncated:

  • YYYY-MM-DDThh:mm:ssZ

  • YYYY-MM-DD

• All components have to occupy the maximum possible number of characters by adding leading 0's (hence, 08 instead of 8 for the month of August, 01 instead of 1 for 1 PM)

  • 2016-08-21T01:00:00Z

• 24 hours notation is used (thus 9:00 PM is 21:00)

  • 2016-08-21T21:00:00Z

Example Showing Full Meta Tag Waterfall

Below follows an example html page showing the various levels of meta tagging/data extraction.

Meta Tagging Examples

Content Tags

An HTML page normally consists of a mixture of meaningful content and text that we don't care about such as footers, sidebars and menus. The Cxense Crawler normally do a good job at extracting the relevant content from a HTML page. This is particullary true when the page is a news article. However, there are times when is beneficial to help the crawler with identifying the content to be extracted.

Content Tagging Example

Relevant Content can be tagged with the html div attribute cXenseParse or with the HTML comments pair cxenseparse_start / cxenseparse_end anywhere within the <body> element.

Content Tagging Examples

Custom Markup

The purpose of all the mark-up covered so far has been to correct or in other ways compensate for the crawlers inability to extract the data that we want it to extract. In this section we will look at ,mark-up that is added for the purpose of supplying additional data beyond what the crawler extract by default. 

Below see see an example taken from an imaginary fruit web site where each web page is tagged with the one or several fruits that the page content is covering.

CustomTagging Examples

The 'xyz' part of "cXenseParse:xyz-" is the customer prefix. Documents with this meta tag will end up with the key xyz-fruit in the content profile. This information can be viewed via the Cxense Insight GUI and it is possible to make traffic queries and search queries based on it.

Recrawling Updated Pages

For any of the changes described above to have the intended effect, the page(s) must first be re-crawled and re-processed. The following 4 actions will make that happen:

• Do nothing - you heard right, doing nothing will make your page get re-crawled. However, you may have to wait for a quite some time before that happens as there is no fixed time schedule (less than 60 days ought to be a safe guess though).

• Do single page manual re-crawl - see figure below for how to re-crawl a single page that one has just updated. Good approach for testing one's changes during development.

• Write a script - use the Cxense API function /profile/content/push from within a Cxense API script

• Add re-crawl meta tag - Any page with a modified time meta tag with a time more recent than the previous crawl time will be re-crawled soon after the next user visit to the page:

  <meta property="article:modified_time" content="2016-12-16T14:18:00-05:00">

Last updated: August 12, 2021