Piano's Likelihood to Subscribe (LtS) model is trained based on conversion events passed to the Piano Audience. For clients using Piano Management + Billing for payments, this occurs automatically and Piano's engineering and data science teams monitor this event flow. For clients who do not use Management + Billing to process payments and are passing conversion events to Piano as a bespoke integration, this document outlines an approach to monitor conversions.
The approach outlined here is to make regular API calls to the Piano Audience to get the number of conversion events in a given time period. That number should then be compared to the expected number of conversions to ensure your integration is sending conversion events as expected.
When devising alerts in relation to this data, it is recommended to alert on sharp declines (no conversions in the prior 24 hours) rather than minor variances in conversions that do not have a measurable impact on algorithm performance.
Prerequisites
The following is needed to be able to follow this approach. Reach out to Piano support if you need assistance getting this information:
-
A Piano user account that has a CX API key
-
The site ID for your site in Piano Audience
In this guide, we will illustrate the necessary API calls using the command line API client for Audience called cx.py. See this page for instructions on how to get cx.py.
Find the product IDs that are relevant
Conversion events can be reported for a variety of different terms, both subscription terms and registration terms. Each term is represented by a product ID in Audience, with an associated type. We need to fetch the list of currently used products to find all the product IDs that we want to check the number of conversion events for.
Example API query to the /conversion/product/read endpoint, with the siteId as the only request argument, to get all products for this site:
$ cx.py /conversion/product/read '{"siteId": "112233445566778899"}'
{
"products": [
{
"id": "8m1a2b3c4d5e",
"siteGroupId": "112233445566778888",
"siteId": "112233445566778899",
"type": "subscription",
"name": "Monthly Term",
"deleted": false
},
{
"id": "8m6a7b8c9d10e",
"siteGroupId": "112233445566778888",
"siteId": "112233445566778899",
"type": "registration",
"name": "Register to Read More",
"deleted": false
}
]
}
Explanation of response object
The JSON response object contains a list of product objects. Each object has a unique ID which is the product ID. It also has a type and a name. In this example, we are only interested in the products of type "subscription" of which there is one, with product ID "8m1a2b3c4d5e". See API documentation here for more details.
Query for the number of conversion events
Here is an example API call for checking the number of conversion events for one product ID in the last hour. This API call can be repeated for each product if needed. You can also supply a list containing several product IDs in one request to get a total number for all of them together if you don't need the per-product numbers.
$ cx.py /conversion/traffic '{"siteId": "112233445566778899", "start": "-1h", "stop": "-0h", "fields": ["uniqueUsers"], "filters": [{"type": "conversion-event", "group": "productId", "items": ["8m1a2b3c4d5e"]}]}'
{
"start": 1669829476,
"stop": 1669833076,
"data": {
"events": 100,
"uniqueUsers": 99
}
}
Explanation of response object
The JSON response object contains the time period in unixtime and the number of conversion events and unique anonymous visitors (meaning, unique browsers). Here we can see that for this particular product ID there were 100 conversion events in the last hour, made by 99 unique visitors. The slight difference between these two numbers indicates that there was one visitor that somehow managed to trigger two conversion events for the same product in the same hour.
Piano recommends checking the "events" number against your subscription management system to verify that the numbers are as expected. Note that minor deviations can happen and are usually not a cause for concern. This is especially true for the "uniqueUsers" metric due to differences between systems in how anonymous users are tracked and identified. The most important change to monitor and alert for is whether the "events" number goes to zero, indicating that something has stopped working in the technical integration.
See the API documentation here for more details.