Manual tuning of content profiles

Content profiles are automatically produced from the pages where the Piano Content script is present. They can be thought of as a page fingerprints, and form the basis for certain key operations in the platform, e.g.:

• Contextual ad targeting in Advertising. E.g., "show my ad on pages that have been classified as sports pages."

• Contextual recommendations of news articles in Piano Content. E.g., "retrieve articles that have content similar to this reference article's content profile."

• Traffic filtering in Piano Insight. E.g., "how much traffic do I have on pages that have negative sentiment?"

• Content analytics in Piano Insight. E.g., "what are the most read-about persons on my site right now?"

Additionally, elements from content profiles can migrate into user profiles and thus impact various personalization features of the platform.

High-quality content profiles are typically produced with little or no manual intervention, but in some cases one might want to manually tune the process. This page outlines how do achieve this, and how to troubleshoot some common situations where one might consider doing this.

Diagnosing crawling issues

A prerequisite for creating a content profile for a URL is that the URL has been crawled. Crawling happens based on normalized URLs. I.e., two or more different raw URLs might normalize to the same normalized URL (e.g., due to which URL parameters that are considered as being meaningful) and then one representative from each group of equivalent URLs are selected from crawling.

Symptoms of crawling not having taken place are, e.g.:

• No content profile is available at all for the URL.

• The content profile for the URL only contains elements derived from the URL and not from the actual content of the page.
  

This can easily be checked using the /profile/content/fetch API method, or from within the Piano Insight application. E.g.:

• In the Popularity view, the document doesn't have a title but only the raw URL is listed.

• In the Page Statistics view, the content profile only contains site, category and/or taxonomy keys.

Possible causes of this include the site not having been whitelisted and the "freebie limit" has been reached (this usually only applies to very large sites), or that the URL in question for some reason has been classified by Piano as being a page that shouldn't be crawled.

Currently, any issues regarding crawling are not self-service but require contacting Piano support for assistance. 

Diagnosing and addressing URL normalization issues

Similarly, symptoms of URL normalization problems include:

• URL parameters seem to be ignored and lots of different pages have their traffic collapsed down to the same URL. E.g., both http://www.example.com/?id=123 and http://www.example.com/?id=456 are really different pages, but they seem to considered as the same and their traffic statistics are conflated. 

• The same page is accessible through multiple different URLs and their traffic is not collapsed down to the same URL. E.g., both http://www.example.com/foo and http://www2.example.com/foo are really the same page, but their traffic statistics are being reported separately.

This is perhaps most easily checked by looking at the Popularity view in the Piano Insight application.

Currently, most issues regarding URL normalization are not self-service but require contacting Piano support for assistance. The possible exception is the JavaScript API's location override feature, which can be used in some circumstances.

Diagnosing issues with content not found in index

When a page has been crawled, the content profile gets created and the page is fed to the search index. If the page is not found in the index (via /document/search API), the problems could be that canonical or og:url does not match with the crawled URL. In which case, the page will not be indexed.

Diagnosing and addressing parsing issues

When a page has been crawled, the HTML is parsed and an algorithm is applied to automatically identify the relevant, unstructured text on the page that should undergo natural language processing. For typical news articles this process works well, but in some cases the parsing heuristic might fail. Extracting unwanted, irrelevant text will thus cause noisy and irrelevant content profiles.

This is best checked by looking at the content profile of a specific problem page, either using the /profile/content/fetch API method or from within the Piano Insight application. If the content profile seems to contain elements that are irrelevant to the page's logical content (e.g., perhaps it contain elements that clearly stem from the document's header or footer or from some piece of content found in a sidebar), that might be an indication of HTML parsing problems.

You can address HTML parsing issues by adding markup to your HTML that tells the HTML parser which elements on the page that contain the unstructured text to extract, as documented here. This basically amounts to sprinkling your markup with certain <div> tags, where relevant.

Diagnosing and addressing extraction issues

When a page has been crawled and the relevant unstructured text has been parsed out from the page, that text undergoes natural language processing and structured data items are extracted from that text using language-specific algorithms. In some cases you might want to tune this extraction process. This is best checked by looking at the content profile of a specific problem page, either using the /profile/content/fetch API method or from within the Piano Insight application.

To manually add structured data elements to a content profile, there are two different options available:

• You can add certain <meta> tags to the HTML markup of the page, as documented here.

• You can use the /profile/content/external/update API method and its enrich mode.

In both cases, the pages in question will have to be recrawled and reprocessed before the changes take effect. There are three different options available for this:

• You can wait until the crawler does this automatically according to its recrawling strategies. 

• You can manually trigger this by using the /profile/content/push API method.

• You can manually trigger this by using the Refetch button in the Page Statistics view in the Piano Insight application.

To manually override all structured data elements in a content profile (and thus completely specifying the content profile), you can use the /profile/content/external/update API method and its replace mode. Note that the same issue as noted above regarding propagation of this data throughout the backend systems applies.

Last updated: June 25, 2026