Natural Language Processing (NLP)
Piano uses sophisticated NLP algorithms to process the text of pages that are crawled and produce structured data that can be utilized in multiple ways across all Piano products. Below you’ll find more details about the different techniques in use.
Content Classification
The article classification system is built on top of state-of-the-art Machine Learning models with the goal of accurately classifying news articles into pre-defined categories based on the IAB Content Taxonomy. More details about the categories supported and what they are about can be found here.
The model is fine-tuned on thousands of news articles from Piano customers across the globe, which allows the model to learn the context associated with each category and take that into account when assigning the most suitable category to each article.
This applies to the content profile group called "classification".
Entity Extraction
The entity extraction detects the company, location, and person names mentioned in the text. It is based on a state-of-the-art deep learning model with support for identifying keywords of each of the types in many different languages.
This applies to the content profile groups called: "company", "location", and "person".
Concept Detection
Concept detection is used to detect concepts (typically, but not necessarily a grammatical noun-phrase, or frequent 2–3-word phrases) mentioned in the text. It is based on state of the art pre-trained text analysis models for multiple languages. These models extract candidate concepts from the input text, which is then ranked on its importance to the article. Only the most important and relevant concepts are added to the content profile.
This applies to the content profile group called "concept".
Quality-issue Detection
The article quality-issue detection is built using state of the art transformer models which learn from hundreds of examples of articles containing various kinds of quality issues. These articles are from publishers within Piano customers, with content in multiple languages.
Types of quality issues that are detected with this approach:
-
empty-body (when the article body is empty);
-
cookie-notice (when the article body contains cookie-notice text);
-
paywall-notice (when the article body contains paywall-notice text);
-
incorrect-site-language (when the article language does not match the one assigned to the site).
This applies to the content profile group called "quality-issue".
Sentiment
The sentiment analysis reflects the tone or the emotional stance of each article and classifies it as positive, neutral, or negative. The solution is powered by a Large Language Model to understand and determine the sentiment of the content as well as its weight.
Transparency
Sentiment Analysis for Content Profiles is generated in Piano’s own data centers using a large language model. In this particular case, we utilize the Llama open-source model by Meta. The model is queried by providing:
-
Contextual information: Metadata that includes sentiment descriptions and task-specific instructions, ensuring the model accurately extracts the sentiment as intended.
-
Crawled article data: Full text (including both title and body) is analyzed to assess the overall sentiment.
-
Rules and guidelines: Specific guidelines are provided to ensure the model only considers relevant data, reducing any ambiguity in classification.
All data is processed within the EU region, reinforcing our commitment to data privacy and transparency.
Representation and usage
Logically, a content profile can be thought of as a set of:
-
Group name: A string that somehow indicates the type of the item. This typically identifies how to semantically interpret the item. For example, an item john smith might have the group name person.
-
Item: A string, often a normalized version of something actually found on the page, but it could equally well be a category name or something else.
-
Weight: A number that somehow reflects the relative prominence of the item on the page, as measured via the well-defined concepts from information retrieval.
Content profiles are used in a number of settings and applications in the Piano platform, e.g.:
-
As input to many of Piano’s Machine Learning models.
-
As a foundation for content recommendations.
-
As a foundation for Lookalike modeling.
-
As a foundation for contextual targeting.
-
As a component for building user profiles.
-
As a component for building user interest segments.
-
As a component for building contextual third-party applications.
-
As dimensions in Insight reporting and ad-hoc analysis.
Different elements in a content profile can be used for different things. In general, their use is application-specific, and not all elements in a profile are used for all applications.
Group names
Group names found in content profiles generally vary depending on the language since the linguistic processing that takes place to produce them is language-specific. One of the first steps during the content processing is language identification since the way we process a Japanese text might vary a lot from how we process a text in Spanish.
Groups commonly found in content profiles are listed in the table below, together with a description of how to interpret items belonging to these groups.
|
Group |
Description |
|---|---|
|
|
A field that is present in content profiles that are considered safe for brands to advertise on, according to the IAB Tech Lab guidelines around brand safety. The field contains the value Brand safety is determined using a Piano ML model trained to identify articles that are not safe for brands to advertise on, according to the IAB Content Taxonomy v3. |
|
|
A string whose value indicates a category name of some kind. Typically inferred from the URL structure. Note that the inference from URL is done by splitting the URL into path components and applying heuristics to remove components which are unlikely categories. The current heuristic will only extract up to 5 components, and components will be skipped if they are too short, too long or look like numbers. In the example URL http://www.example.com/2015/11/sports/baseball/yankees-beat-red-sox, the categories sports and baseball gets extracted. It is possible to override categories by including a taxonomy meta tag in the documents, see Document parsing. |
|
|
A string indicating an automatically detected content topic, e.g., Classification categories tend to match the top-level of IAB Tech Lab Content Taxonomy. For The list of all supported categories can be found here. |
|
|
A proper name believed to be the name of a company or organization. E.g., |
|
|
A string indicating a concept of some kind. Concepts refer to abstract ideas or general notions that convey meaning beyond specific entities or individuals, it encapsulates broader themes, categories, or topics that help to understand the subject matter of the text. They often represent relationships, processes, or ideologies rather than tangible, named entities. E.g., |
|
|
A string containing the prediction from Piano’s Content Likely to Convert machine learning model, as values from “0” to “9” signifying 10 propensity buckets. This ML model predicts which articles are the most likely to drive subscription conversions. The model analyzes not just data related to the type of content and the author, but also incorporates key performance indicators for each article just after publication. Since the model automatically detects high-potential articles, the scores it produces can be leveraged in content locking rules, content recommendation logic, and promotional strategies. |
|
|
A string believed to be a proper name of some unspecified kind. E.g., a brand or product name, or the name of a movie, or something else. |
|
|
A string indicating the language of the page. E.g., Spanish pages would have |
|
|
A proper name believed to be the name of a location. E.g., |
|
|
Indicates if the page is believed to be an article or a frontpage. E.g., |
|
|
A proper name believed to be the name of a person. E.g., |
|
|
An indication of issues with the content profiles, which can be:
|
|
|
A name of an event, celebration or special occasion. E.g., Only available for English, German, French, Dutch, Spanish, Czech, Polish, Danish, Swedish, Norwegian, Finnish, Russian, and Greek sites. |
|
|
The sentiment reflects the tone or emotional stance of the entire article content. Sentiment can be: neutral, positive, or negative. Only available for Czech, Dutch, French, and German. |
|
|
The name of the site or domain where the page lives. E.g., |
|
|
A string whose values indicate a full category path of some kind. Typically, but not necessarily, inferred from the URL structure. See |
|
|
Keywords for a page, typically explicitly specified through keyword meta tags. |
|
|
An omniture channel from the javascript variable as defined here . |
|
|
The schema.org itemtype metadata. |
|
|
The date of article publication in UTC formatted like |
Other keys might also occur. E.g., there are mechanisms that include customer-defined group names having customer-defined semantics. Such data can either be derived from the page content, or be explicitly supplied.
Supported languages
As noted above, the processing of unstructured text uses language-specific algorithms and lexical resources. The currently supported languages are:
Afrikaans, Arabic, Bengali, Chinese (simplified and traditional), Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Thai, Turkish, and Urdu.