Piano strives for maximum transparency with our clients. In that spirit, the documentation here strives to provide your data science team with detailed information about the nature and performance of Piano’s Content Likely to Convert model, which predicts the likelihood of each article driving subscription conversions.
This documentation is primarily intended for any data scientists you may have on staff. More general documentation on the content likelihood to convert model can be found here.
Features
There are 3 types of features calculated for each article: traffic features, content features, and conversion features. Each set of features is detailed below.
Traffic Features
Features based on traffic (pageview) data on the article. Each traffic feature is calculated for two time periods:
-
first 30 minutes after publication, e.g. androidPvs_first30min
-
first 2 hours after publication, e.g. androidPvs
Both of these feature groups are also expressed as relative features:
-
share of pageviews from Android in the first 2 hours (the number of Android pageviews divided by total pageviews), e.g. androidPvs-rate
-
share of pageviews from Android in the first 30 minutes, e.g. androidPvs-rate_first30min
In order to express the dynamics of the traffic on the article, we calculate growth features for both absolute and relative features:
-
growth features (percentual growth in the first 30 minutes vs 2 hours), e.g. androidPvs_growth
-
growth features for relative features (percentual growth of the relative values - first 30 minutes vs 2 hours), e.g. androidPvs-rate_growth
Content Features
Features based on the metadata of the article. Content Features are available at the moment of publication.
Content features can be divided into the following categories:
-
A: Features based on author data
-
B: Features based on IAB classification
-
C: Features based on the time and day of publication
A: Features based on author data
There are 10 features based on author data. These are binary features based on the top 10 most popular authors (most popular in terms of number of pageviews on their articles).
B: Features based on IAB classification
There are 25 features based on IAB classification data. These are binary features representing the article's IAB category, which is predicted when the article is initially crawled and the article’s content profile is created.
C: Features based on the time and day of publication
There are 6 binary features representing the day of the week and 23 binary features representing the time of day.
Conversion Features
There are 2 features leveraging the subscription conversion data Piano collects:
-
Total number of paid subscription conversions on the article, 2 hours after publication
-
The ratio of the total number of conversions and the total number of pageviews 2 hours after publication
Feature Table
Here is the complete list of features in the model:
|
Feature Name |
Feature Description |
Feature Type |
|
Android pageviews |
Number of pageviews from android 2 hours after publishing |
Traffic |
|
Android pageviews (first 30 minutes) |
Number of pageviews from android 30 minutes after publishing |
Traffic |
|
Android pageviews growth |
Percentual growth of pageviews from android (30 minutes vs 2 hours) |
Traffic |
|
Share Android pageviews |
Share of pageviews from android 2 hours after publishing |
Traffic |
|
Share Android pageviews (first 30 minutes) |
Share of pageviews from android 30 minutes after publishing |
Traffic |
|
Share Android pageviews growth |
Percentual growth of the share of pageviews from android (30 minutes vs 2 hours) |
Traffic |
|
Author |
Is an article of a given author (10 features, based on the top 10 most visited authors) |
Content |
|
Bounce rate |
Share of bounce pageviews on a given article within 2 hours after publishing |
Traffic |
|
Bounce rate (first 30 minutes) |
Share of bounce pageviews on a given article within 30 minutes after publishing |
Traffic |
|
Bounce rate growth |
Percentual growth of the share of bounce pageviews (30 minutes vs 2 hours) |
Traffic |
|
Share attractions articles |
Is attractions article |
Content |
|
Share automotive articles |
Is automotive article |
Content |
|
Share books and literature articles |
Is books and literature article |
Content |
|
Share business and finance articles |
Is business and finance article |
Content |
|
Share careers articles |
Is careers article |
Content |
|
Share communication articles |
Is communication article |
Content |
|
Share crime articles |
Is crime article |
Content |
|
Share disaster articles |
Is disaster article |
Content |
|
Share education articles |
Is education article |
Content |
|
Share entertainment articles |
Is entertainment article |
Content |
|
Share events articles |
Is events article |
Content |
|
Share family and relationships articles |
Is family and relationships article |
Content |
|
Share fine art articles |
Is fine art article |
Content |
|
Share food and drink articles |
Is food and drink article |
Content |
|
Share healthy living articles |
Is healthy living article |
Content |
|
Share hobbies and interests articles |
Is hobbies and interests article |
Content |
|
Share holidays articles |
Is holidays article |
Content |
|
Share home and garden articles |
Is home and garden article |
Content |
|
Share law articles |
Is law article |
Content |
|
Share medical health articles |
Is medical health article |
Content |
|
Share personal celebrations and life events articles |
Is personal celebrations and life events article |
Content |
|
Share personal finance articles |
Is personal finance article |
Content |
|
Share politics articles |
Is politics article |
Content |
|
Share pop culture articles |
Is pop culture article |
Content |
|
Share real estate articles |
Is real estate article |
Content |
|
Share science articles |
Is science article |
Content |
|
Share sports articles |
Is sports article |
Content |
|
Share style and fashion articles |
Is style and fashion article |
Content |
|
Share technology and computing articles |
Is technology and computing article |
Content |
|
Share television articles |
Is television article |
Content |
|
Share travel articles |
Is travel article |
Content |
|
Share video gaming articles |
Is video gaming article |
Content |
|
Share war and conflicts articles |
Is war and conflicts article |
Content |
|
Chrome pageviews |
Number of pageviews from chrome 2 hours after publishing |
Traffic |
|
Chrome pageviews (first 30 minutes) |
Number of pageviews from chrome 30 minutes after publishing |
Traffic |
|
Chrome pageviews growth |
Percentual growth of pageviews from chrome (30 minutes vs 2 hours) |
Traffic |
|
Share Chrome pageviews |
Share of pageviews from chrome 2 hours after publishing |
Traffic |
|
Share Chrome pageviews (first 30 minutes) |
Share of pageviews from chrome 30 minutes after publishing |
Traffic |
|
Share Chrome pageviews growth |
Percentual growth of the share of pageviews from chrome (30 minutes vs 2 hours) |
Traffic |
|
Number of conversions |
Number of paid subscription conversions 2 hours after publishing |
Conversion |
|
Ratio of conversions vs pageviews |
Ratio of paid subscription conversions and pageviews 2 hours after publishing |
Conversion |
|
Direct pageviews |
Number of direct pageviews 2 hours after publishing |
Traffic |
|
Direct pageviews (first 30 minutes) |
Number of direct pageviews 30 minutes after publishing |
Traffic |
|
Direct pageviews growth |
Percentual growth of direct pageviews (30 minutes vs 2 hours) |
Traffic |
|
Share Direct pageviews |
Share of direct pageviews 2 hours after publishing |
Traffic |
|
Share Direct pageviews (first 30 minutes) |
Share of direct pageviews 30 minutes after publishing |
Traffic |
|
Share Direct pageviews growth |
Percentual growth of the share of direct pageviews (30 minutes vs 2 hours) |
Traffic |
|
Firefox pageviews |
Number of pageviews from firefox 2 hours after publishing |
Traffic |
|
Firefox pageviews (first 30 minutes) |
Number of pageviews from firefox 30 minutes after publishing |
Traffic |
|
Firefox pageviews growth |
Percentual growth of pageviews from firefox (30 minutes vs 2 hours) |
Traffic |
|
Share Firefox pageviews |
Share of pageviews from firefox 2 hours after publishing |
Traffic |
|
Share Firefox pageviews (first 30 minutes) |
Share of pageviews from firefox 30 minutes after publishing |
Traffic |
|
Share Firefox pageviews growth |
Percentual growth of the share of pageviews from firefox (30 minutes vs 2 hours) |
Traffic |
|
Internal pageviews |
Number of internal pageviews 2 hours after publishing |
Traffic |
|
Internal pageviews (first 30 minutes) |
Number of internal pageviews 30 minutes after publishing |
Traffic |
|
Internal pageviews growth |
Percentual growth of internal pageviews (30 minutes vs 2 hours) |
Traffic |
|
Share Internal pageviews |
Share of internal pageviews 2 hours after publishing |
Traffic |
|
Share Internal pageviews (first 30 minutes) |
Share of internal pageviews 30 minutes after publishing |
Traffic |
|
Share Internal pageviews growth |
Percentual growth of the share of internal pageviews (30 minutes vs 2 hours) |
Traffic |
|
iPhone OS pageviews |
Number of pageviews from iPhone OS 2 hours after publishing |
Traffic |
|
iPhone OS pageviews (first 30 minutes) |
Number of pageviews from iPhone OS 30 minutes after publishing |
Traffic |
|
iPhone OS pageviews growth |
Percentual growth of pageviews from iPhone OS (30 minutes vs 2 hours) |
Traffic |
|
Share iPhone OS pageviews |
Share of pageviews from iPhone OS 2 hours after publishing |
Traffic |
|
Share iPhone OS pageviews (first 30 minutes) |
Share of pageviews from iPhone OS 30 minutes after publishing |
Traffic |
|
Share iPhone OS pageviews growth |
Percentual growth of the share of pageviews from iPhone OS (30 minutes vs 2 hours) |
Traffic |
|
Linux pageviews |
Number of pageviews from linux 2 hours after publishing |
Traffic |
|
Linux pageviews (first 30 minutes) |
Number of pageviews from linux 30 minutes after publishing |
Traffic |
|
Linux pageviews growth |
Percentual growth of pageviews from linux (30 minutes vs 2 hours) |
Traffic |
|
Share Linux pageviews |
Share of pageviews from linux 2 hours after publishing |
Traffic |
|
Share Linux pageviews (first 30 minutes) |
Share of pageviews from linux 30 minutes after publishing |
Traffic |
|
Share Linux pageviews growth |
Percentual growth of the share of pageviews from linux (30 minutes vs 2 hours) |
Traffic |
|
MS IE pageviews |
Number of pageviews from MS IE 2 hours after publishing |
Traffic |
|
MS IE pageviews (first 30 minutes) |
Number of pageviews from MS IE 30 minutes after publishing |
Traffic |
|
MS IE pageviews growth |
Percentual growth of pageviews from MS IE (30 minutes vs 2 hours) |
Traffic |
|
Share MS IE pageviews |
Share of pageviews from MS IE 2 hours after publishing |
Traffic |
|
Share MS IE pageviews (first 30 minutes) |
Share of pageviews from MS IE 30 minutes after publishing |
Traffic |
|
Share MS IE pageviews growth |
Percentual growth of the share of pageviews from MS IE (30 minutes vs 2 hours) |
Traffic |
|
Opera pageviews |
Number of pageviews from opera 2 hours after publishing |
Traffic |
|
Opera pageviews (first 30 minutes) |
Number of pageviews from opera 30 minutes after publishing |
Traffic |
|
Opera pageviews growth |
Percentual growth of pageviews from opera (30 minutes vs 2 hours) |
Traffic |
|
Share Opera pageviews |
Share of pageviews from opera 2 hours after publishing |
Traffic |
|
Share Opera pageviews (first 30 minutes) |
Share of pageviews from opera 30 minutes after publishing |
Traffic |
|
Share Opera pageviews growth |
Percentual growth of the share of pageviews from opera (30 minutes vs 2 hours) |
Traffic |
|
Other OS pageviews |
Number of pageviews from other OS 2 hours after publishing |
Traffic |
|
Other OS pageviews (first 30 minutes) |
Number of pageviews from other OS 30 minutes after publishing |
Traffic |
|
Other OS pageviews growth |
Percentual growth of pageviews from other OS (30 minutes vs 2 hours) |
Traffic |
|
Share Other OS pageviews |
Share of pageviews from other OS 2 hours after publishing |
Traffic |
|
Share Other OS pageviews (first 30 minutes) |
Share of pageviews from other OS 30 minutes after publishing |
Traffic |
|
Share Other OS pageviews growth |
Percentual growth of the share of pageviews from other OS (30 minutes vs 2 hours) |
Traffic |
|
Other browser pageviews |
Number of pageviews from other browsers 2 hours after publishing |
Traffic |
|
Other browser pageviews (first 30 minutes) |
Number of pageviews from other browsers 30 minutes after publishing |
Traffic |
|
Other browser pageviews growth |
Percentual growth of pageviews from other browsers (30 minutes vs 2 hours) |
Traffic |
|
Share Other browser pageviews |
Share of pageviews from other browsers 2 hours after publishing |
Traffic |
|
Share Other browser pageviews (first 30 minutes) |
Share of pageviews from other browsers 30 minutes after publishing |
Traffic |
|
Share Other browser pageviews growth |
Percentual growth of the share of pageviews from other browsers (30 minutes vs 2 hours) |
Traffic |
|
Number of pageviews |
Total number of pageviews 2 hours after publishing |
Traffic |
|
Number of pageviews (first 30 minutes) |
Total number of pageviews 30 minutes after publishing |
Traffic |
|
Number of pageviews growth |
Percentual growth of the number of pageviews (30 minutes vs 2 hours) |
Traffic |
|
Safari pageviews |
Number of pageviews from safari 2 hours after publishing |
Traffic |
|
Safari pageviews (first 30 minutes) |
Number of pageviews from safari 30 minutes after publishing |
Traffic |
|
Safari pageviews growth |
Percentual growth of pageviews from safari (30 minutes vs 2 hours) |
Traffic |
|
Share Safari pageviews |
Share of pageviews from safari 2 hours after publishing |
Traffic |
|
Share Safari pageviews (first 30 minutes) |
Share of pageviews from safari 30 minutes after publishing |
Traffic |
|
Share Safari pageviews growth |
Percentual growth of the share of pageviews from safari (30 minutes vs 2 hours) |
Traffic |
|
Search pageviews |
Number of search pageviews 2 hours after publishing |
Traffic |
|
Search pageviews (first 30 minutes) |
Number of search pageviews 30 minutes after publishing |
Traffic |
|
Search pageviews growth |
Percentual growth of search pageviews (30 minutes vs 2 hours) |
Traffic |
|
Share Search pageviews |
Share of search pageviews 2 hours after publishing |
Traffic |
|
Share Search pageviews (first 30 minutes) |
Share of search pageviews 30 minutes after publishing |
Traffic |
|
Share Search pageviews growth |
Percentual growth of the share of search pageviews (30 minutes vs 2 hours) |
Traffic |
|
Session starts |
Number of pageviews that are starting a new session of a user within 2 hours after publishing |
Traffic |
|
Session starts (first 30 minutes) |
Number of pageviews that are starting a new session of a user within 30 minutes after publishing |
Traffic |
|
Session starts growth |
Percentual growth of the number of pageviews that are starting a new session of a user (30 minutes vs 2 hours) |
Traffic |
|
Social pageviews |
Number of social pageviews 2 hours after publishing |
Traffic |
|
Social pageviews (first 30 minutes) |
Number of social pageviews 30 minutes after publishing |
Traffic |
|
Social pageviews growth |
Percentual growth of social pageviews (30 minutes vs 2 hours) |
Traffic |
|
Share Social pageviews |
Share of social pageviews 2 hours after publishing |
Traffic |
|
Share Social pageviews (first 30 minutes) |
Share of social pageviews 30 minutes after publishing |
Traffic |
|
Share Social pageviews growth |
Percentual growth of the share of social pageviews (30 minutes vs 2 hours) |
Traffic |
|
Unique users |
Total number of users 2 hours after publishing |
Traffic |
|
Unique users (first 30 minutes) |
Total number of users 30 minutes after publishing |
Traffic |
|
Unique users growth |
Percentual growth of the number of users (30 minutes vs 2 hours) |
Traffic |
|
Unique users vs pageviews ratio |
Ratio of unique users and pageviews 2 hours after publishing |
Traffic |
|
Unique users vs pageviews ratio (first 30 minutes) |
Ratio of unique users and pageviews 30 minutes after publishing |
Traffic |
|
Unique users vs pageviews ratio growth |
Percentual growth of the ratio of unique users and pageviews (30 minutes vs 2 hours) |
Traffic |
|
Unique users with multiple pageviews |
Number of users with 2 or more pageviews 2 hours after publishing |
Traffic |
|
Unique users with multiple pageviews (first 30 minutes) |
Number of users with 2 or more pageviews 30 minutes after publishing |
Traffic |
|
Unique users with multiple pageviews growth |
Percentual growth of the number of users with 2 or more pageviews (30 minutes vs 2 hours) |
Traffic |
|
Windows pageviews |
Number of pageviews from windows 2 hours after publishing |
Traffic |
|
Windows pageviews (first 30 minutes) |
Number of pageviews from windows 30 minutes after publishing |
Traffic |
|
Windows pageviews growth |
Percentual growth of pageviews from windows (30 minutes vs 2 hours) |
Traffic |
|
Share Windows pageviews |
Share of pageviews from windows 2 hours after publishing |
Traffic |
|
Share Windows pageviews (first 30 minutes) |
Share of pageviews from windows 30 minutes after publishing |
Traffic |
|
Share Windows pageviews growth |
Percentual growth of the share of pageviews from windows (30 minutes vs 2 hours) |
Traffic |
|
Publication Day |
Day of week of the day of publishing |
Content |
|
Publication Hour |
Time of the day of publishing |
Content |
Target Variable
The model predicts whether or not an article will drive subscription conversions within the first week of publication.
The model is trained using a binary target variable derived from the total number of paid subscription conversions attributed to each article. An article is attributed with a conversion event if it meets the following criteria:
-
The article is visited during the conversion session (either before or during the conversion event).
-
The conversion happens within a 7-day timeframe following the article's publication, with the initial two hours after publication being excluded. (These initial two hours are reserved for feature calculation purposes.)
Based on this attribution criteria, the total number of conversions within the 7-day post-publication window is computed for each article. Articles with zero conversions are assigned a target variable value of '0,' while articles with one or more conversions are assigned a value of '1.'
Training Data Storage
Upon initiation of the model training process, features and the target variable are computed for all articles published in the prior month. This data, organized at the article level, is subsequently stored and utilized as the dataset for model training.
The model training job is executed on a daily basis. Each day, a dataset containing articles published on the prior day is appended to the existing dataset. This iterative process ensures that the model is trained on an expanding dataset, which increases the model’s quality and accuracy over time.
Publications with lower editorial output and fewer conversions on articles will likely need a longer training period in order for this process to produce a quality model.
Machine Learning Algorithm
In the initial steps of model training, 20% of the dataset is set aside as a validation dataset. The remaining 80% of the dataset is then divided into a training and test set, in a ratio of 80:20.
The classification algorithm used is XGBoost. For each training iteration, hyperparameter optimization is carried out using the Bayesian hyperparameter optimization algorithm. The optimized parameters include: max_depth, gamma, eta, and others.
Model Quality Evaluation
The validation dataset is used for evaluating model quality. Within each training job, the resulting model is used for predicting scores for articles in the validation dataset. The model is pushed to production if the following criteria are met:
-
Recall is at least 0.52 (this value will commonly be 80-90%)
-
There is at least one article predicted in the highest CLtC segment
-
There is at least one article predicted in the lowest CLtC segment
Recall measures the share of articles with conversion that were correctly identified by the model. The threshold of over 50% means that the model is, at a minimum, performing better than chance.
By setting those quality thresholds, we can ensure that only higher-quality models with a wide distribution of article scores are sent into production for use.