How do I interpret A/B test and conversion metrics in Piano Composer?

Key Concepts and Definitions

Exposures

An exposure is logged when a template is loaded on a page (even if the user does not actively view it). Exposures are derived from backend event records.

Conversions (General)

A "conversion" in Composer reporting is a successful user action attributed to an experience. Depending on configuration and context, conversions can include:

• Completing an offer call-to-action (CTA)

• Accepting terms within an offer

• Clicking an external link in a template

• Triggering configured micro-conversions such as EXTERNAL_LINK or EXTERNAL_EVENT

User-Level vs. Pageview-Level Metrics

Composer reporting commonly offers two aggregation modes:

User-level metrics:

• Count unique users.

• A user is generally counted once for a given metric, even if they repeat the action multiple times.

• In cumulative charts, totals can occasionally decrease because the system deduplicates and recalculates as identities are reprocessed.

Use user-level metrics when you want to answer: "How many unique users converted?"

Pageview-level metrics:

• Count every occurrence of an action across pageviews.

• The same user can be counted multiple times if the event occurs on multiple pageviews (or multiple times across visits).

• In cumulative charts, totals typically only increase.

Use pageview-level metrics when you want to answer: "How often did this happen overall?"

How A/B Test Reports Differ from Conversion Reports

A/B Test Report: Optimized for Winner Selection

A/B Test reporting is designed to compare variants and determine which performs best using the chosen KPI. Common metrics include:

• Clicks: Clicks on a template (excluding close actions)

• CTR: clicks / exposures

• Conversions: Completion of an offer CTA (if no offer exists, this may be 0)

• Conversion rate: conversions / exposures

A/B tests are experience-specific and are intended to report outcomes within the context of that test.

Composer Conversion Report: Optimized for Conversion Analysis

The Composer Conversion report focuses on actions recorded as conversions/micro-conversions, such as:

• Offer-related actions (e.g., terms accepted, CTA completion)

• Template link clicks (e.g., EXTERNAL_LINK)

• Tracked custom events (e.g., EXTERNAL_EVENT)

A practical way to investigate differences is to use the report's grouping/breakdown options, such as:

• Group by template (to validate click counts per variant/template)

• Break down by conversion type (to separate CTA clicks vs. other conversion events)

Why Numbers May Not Match Across Reports

Different Counting Level (User vs. Pageview)

A/B Test reporting and Conversion reporting may not be using the same aggregation mode. If one report is counting pageviews and the other is counting users, clicks/conversions will not match.

"Conversion" May Represent Different Actions

In A/B testing, "conversion" can be used as an umbrella term for successful interactions (for example: registering, clicking through, dismissing a message, starting a purchase flow), depending on how the experience and KPI are configured.

Separately, if you compare Composer-reported conversions to Management + Billing subscription outcomes, the numbers may diverge because:

• Composer may count an interaction with a CTA as a conversion (even if the purchase is abandoned)

• Management + Billing subscription logs count successful transactions only

Multiple Clickable Elements Can Inflate Click Counts

If a template includes multiple clickable elements, click totals can be higher than expected, especially in pageview-level reporting.

Use the Conversion report grouped by template (and, when relevant, by conversion type) to verify what is being counted.

Attribution Effects When Multiple Experiences Run on the Same Page

If multiple Composer experiences run on the same page, conversions may appear in an experience's report due to attribution mechanisms (for example, redirects or tracking parameters such as _ptid) that credit conversions back to the original experience where engagement began.

This can make it look like an experience is receiving conversions "from elsewhere," when they are being attributed back to the initial interaction context.

Timing and Update Cadence Differences

Always align date ranges and be cautious when comparing partial-day snapshots between report types.

Flow Report Is Not Designed to Match Conversion Report Totals

The Flow report is a visualization of movement through the experience (segments, branches, events, actions). It is expected that Flow and Conversion reports do not correlate exactly because they capture and present different data.

Also note:

• Only actions that occur via ShowTemplate or ShowOffer cards are captured in Composer reporting.

• Templates displayed via tp.template.show() inside a Run JS action card may not appear in Composer reports.

Implementation Notes for Conversion Tracking (A/B Tests)

Tracking specific clicks with external-event:

For click tracking that should be treated as a measurable event in reporting, add an external-event in your template to capture specific click values (for example, to track a particular button/link).

Management + Billing clients: Tracking IDs are not necessary for this setup.

Troubleshooting Checklist

1. Confirm the report type you're using (A/B Test vs. Conversion vs. Flow).

2. Align date ranges across reports.

3. Check the metric level (user-level vs. pageview-level).

4. In Conversion reports:

   • Group by template

   • Break down by conversion type

5. If comparing to subscriptions, confirm whether your KPI measures CTA interaction or successful payment.

6. Validate event firing behavior (for purchases in particular):

   • Use browser developer tools to confirm only one conversion call fires per completed purchase.

7. If A/B test results look inconsistent, ensure you are using the latest A/B test cards (see Upgrading A/B Test Cards).

Operational Guidance: When a Test Is "Not Ready to Conclude"

If the A/B test report indicates "Not ready to conclude test," the system considers results partial and needs more data to determine a statistically valid outcome. As a best practice, run A/B tests long enough to capture representative traffic patterns (often several weeks to a month, depending on volume).

Last updated: June 25, 2026