Article content extraction
To be able to deliver good recommendations, we need to ensure the content we extract from each page properly represents the information the end user actually shows an interest in. We have a variety of algorithms, rules and dictionaries that are used to identify article content (and skip menus, ads etc).
For sites that have less common site structured where it is hard for us to automatically identify the article content, you may need to mark up the textual content using the cXenseParse div tag (see Document parsing).
Metadata extraction
In addition to extracting and processing the textual body of the documents, we also extract metadata from each page:
-
Standard html headers (e.g. title).
-
Specific markup of the pages (e.g. facebook open graph tagging).
-
Mapping of textual content into custom categories based on customer provided dictionaries (see Excel workbooks as dictionaries).
-
Custom metadata (e.g. author), see the section on structured data here.
This should only be used when there is a limited number of expected values.
In particular, it should not be used for unstructured text. In such cases, you may instead use: -
Custom, Content specific metadata (See separate section of Document Parsing).