Introduction
A segment can informally be defined as a specified subset of users, i.e., a "targeted audience". In the following we'll describe:
-
How segment matching and segment annotation works, i.e., how we define and keep track of that a given user belongs to a segment.
-
How to create segments that combine properties of users, content or page view activity. I.e., how complex segment definitions can be expressed.
-
Nuances on estimating the size of the targeted audience, in particular on how many events the audience generates.
Segment types
The Cxense DMP currently supports two types of segments:
-
External segments: Such segments are defined externally, i.e., keeping track of which users that belong to an external segment is done by some external system. Information about segment membership is then imported into the Cxense DMP by means of the /segment/data/update API method.
-
Traffic segments: Such segments are managed by the Cxense DMP itself, i.e., the Cxense DMP computes and keeps track of which users that belong to which segments. The Cxense DMP provides a "language" that describes the properties of the users that belong to the traffic segment. Thus, a complete enumeration of which users that belong to the segment isn't needed, but is inferred through the definition language.
This note concerns itself with traffic segments only. Unless otherwise noted, when we talk about segments we mean traffic segments.
Traffic segment membership
The "language" that is used to define a traffic segment is a traffic filter, and traffic filters operate on events and not users. An event could be a "page view event" (e.g., somebody opens a URL in their browser) or a "DMP performance event" (e.g., somebody clicks on a particular ad that has been shown to them.) The translation from events to users is simple, because we keep track of which users that generate which events:
A user is a member of the segment if that user has generated at least one event that matches the traffic filter.
This means also that a user is not a member of the segment if he/she has not generated any events that match the traffic filter. This definition is important to keep in mind, as one quickly gets into scenarios where only some (but not all) events match the traffic filter. We explore this in more detail below.
Traffic filters are associated with segments by means of the /segment/create or /segment/update API methods, or via the interactive segment editor in the DMP UI.
Time periods and cross-site gotchas
A segment's traffic filter is tied to a site group and has a time span associated with it. Thus, the filter captures specific events (and therefore users) emitted from sites within the referenced site group and the given time period. For example, "events generated by users visiting my sites within the last two days."
Regarding the time span and site group associations, note that there are some restrictions:
-
At the time of writing, and without any kind of special configuration, time periods up to 31 days in the past can be specified. Such segments will work across all sites in associated site group.
-
For time periods beyond 31 days in the past, this is at the time of writing only supported for sites that have been explicitly both "LTS-enabled" and "DMP-enabled" by Cxense support. Furthermore, such segments will not work across multiple sites if they are defined in terms of certain operators. (Specifically, if the
explodeUsersfilter or thehavingconstruct are used. If this sounds like gibberish to you, it's because we haven't explained what those things are yet. Relax, they're explained below.)
These restrictions are subject to change.
Note that these same restrictions also apply to segments that are defined over performance data supplied via the /dmp/push API method, i.e., if the segment's traffic filter makes use of the dmp-event or dmp-custom type filters. For segments whose filters join performance data and ordinary traffic data, these restrictions are also explained elsewhere.
Segment tagging and expiration
|
This section was written before support for real-time segments were deployed. For real-time segments, segment membership is evaluated in real-time. Whether or not a segment is real-time depends on the constructs used to define the segment. |
With hundreds of millions of active users and a large number of segments, it becomes quite a task to efficiently and continuously keep track of which users that belong to which segments. At any given point in time we want, among other things, the system to blazingly fast answer the following question:
The user that we see right now, which segments are he/she currently a member of?
Incidentally, the API method that answers this question is /profile/user/segment. In order to provide a lightning-fast answer to such API calls, the platform currently precomputes the answer to this question on a regular basis by means of a continuously running "segment tagger" process:
-
The freshness with which we track that a user enters a segment is currently bounded by how often the segment tagger runs. At the time of writing, this typically varies between 10 and 30 minutes depending on the current capacity and time of day. Also note that due to current capacity restrictions only segments that contain 2 million users or less are processed by the segment tagger.
-
The freshness with which we track that a user leaves a segment is currently bounded by the segment's expiry period. At the time of writing, this is the time period defined for the segment, but not more than 2 days.
The current approach and capacity limitations are subject to change.
For segments that define user engagement (e.g., "users that have generated 50 events or less") the way that the segment tagger currently deals with segment exits might pose a challenge. The workaround is to define a segment that starts where the other segment stops (e.g., "users that have generated 51 events or more") and to have the application combine these using and and not operators (e.g., "users that have generated 50 events or less events, and not 51 events or more.") In other words, users that belong to the first segment but not the second segment.
Note that due to the segment tagging process there is generally a small a difference between how many users that a segment's traffic filter captures, and how many users that the tagger process has marked to belong to the segment. (The segment type filter has options in case you really need to discern between the two cases.)
Event-based matching
Consider the following example:
|
For simplicity, in the following examples we will shorten this to just the traffic filter part:
|
Here, we have created a segment that matches the users that had events originating from Oslo within the last two days. As time goes by, a user may become a member of the segment if he had any events from Oslo within the last two days, or he may stop being a member if he had no events originating from Oslo within the last two days.
Expanding the set of events
So far, so good. However, a given user might move around in the physical world, e.g., the same user might within the given time period have been observed both in Oslo and in Stockholm. In some case we might need to differentiate such users from users that have only been observed in Oslo. The tools we have at our disposal for this are the explodeUsers filter and the having construct, so let's explore these a bit first.
The explodeUsers filter "explodes" or "expands" the set of events to all events generated by users that satisfy the given reference filter. For example, a user that we observed in Oslo might also have generated events while visiting Stockholm or other places, so to capture all of these events we could use the following filter expression:
|
The filter expression above is equivalent to the following filter expression:
|
Roughly speaking, we can think of the explodeUsers filter as being a special case of the having construct. The having construct can be used to express a more detailed restriction on the number of events, though, e.g.:
|
Negations and segment complements
Now, what about the not operator? Negations can have tricky semantics when doing event-based matching, so it's worth studying in more detail. Consider the following example, which will match users that generated any events from any city other than Oslo:
|
In other words, if there are users who generated events from both Oslo and Stockholm within the last two days, these users will be members of both the "Oslo users" and "non-Oslo users" segments! This may sound counter-intuitive and is where the explodeUsers filter becomes useful.
So in the previous example the two segments are not mutually exclusive or complementary. If the not operator doesn't do this alone, how can we define segment complements? By combining not with explodeUsers! For example, the following segment will only match users that do not have any events originating from Oslo and will thus be a complement to the "Oslo users" segment:
|
Let's examine this in detail:
-
The innermost
eventfilter matches the events that originate from Oslo. -
The
explodeUsersfilter expands this event set to all events generated by the same users. -
Finally, the
notfilter inverts the expansion. I.e., we are only left with events from users that have not generated any events from Oslo.
Cross-event matching
The explodeUsers filter can be used for advanced matching across events. For example, let's say that we want to create a segment that identifies users that have been to both Oslo and Stockholm. (In either order. If order is important, the explodeUsers filter has a mode for that, but let's not complicate things for now.) Given that filter matching is applied to individual events and a single event is associated with a single location, combining the constraints using a simple and operator would not yield any matches. Now if only we had a filter that could expand the set of events for us... Hang on, we do!
|
Now are all these three segment definition alternatives equivalent? Well, no. They are almost the same but not in all respects: The three filters all identify the same sets of users, but not exactly the same sets of events! So, if you used the /traffic API method with these three filter alternatives, you would see that the unique user counts would be the same but the number of matching events would differ. Why is that? Let's examine this in detail:
-
In alternative 1, we match the events from Stockholm from the users who also visited Oslo.
-
In alternative 2, we match the events from Oslo from the users who also visited Stockholm.
-
In alternative 3, we cross-match events from users who visited Oslo against the events from users who visited Stockholm. In this case we can match events having neither Stockholm or Oslo on their own.
Profile-based matching
So segments are defined by traffic filters that operate on events. Traffic filters are extremely expressive, and there's a plethora of available filter types that can be combined. A page view event links a given user to a given piece of content (the fact that the user consumed or interacted with the content), so it's natural to want to express filtering constraints also on:
-
Properties of the user that generated the event, i.e., filters applied to user profiles. For example, "all events generated by male users." This can be done using, e.g., the
user-keywordoruser-externalfilters. -
Properties of the piece content that was consumed, i.e., filters applied to content profiles. For example, "all events that involve somebody reading a page where Betty Page is mentioned." This can be done using, e.g., the
keywordfilter.
It's important to note that we do not keep a change log or revision history of user profiles or content profiles when it comes to matching. We keep around the most recent version, and this is what is used also when filtering events that took place in the past (when the profiles might have looked different.) Hey, that's worth repeating:
Profile-based filters are applied to the most recent version of the profiles, and not necessarily how the profiles looked like when the events actually happened.
In other words, this means that filters applied to historical event data might give different results across invocations if the referenced profiles change.
Profile availability
A prerequisite for being able to evaluate profile-based filters is that the profiles actually exists and are available. Some situations where this is not the case are:
-
The content profile doesn't yet exist because we have not yet been able to crawl the page, or because the content profile has not been pushed to us. For example, for newly published pages there will be a brief period of time where we receive events that reference a given URL but where we don't actually have available a content profile associated with the URL.
-
Due to capacity reasons we might only keep around a subset of the overall set of content profile data that we have. At the time of writing, content profiles for pages that we crawled more than 60 days ago are not available for matching.
-
Due to capacity reasons we might only keep around a subset of the overall set of user profile data that we have. At the time of writing, profiles (or some sections thereof) for users that we have not seen in the last 7 days are not available for matching.
The current capacity limitations are subject to change.
Matching against content profiles
A segment definition containing a keyword filter will match all events related to pages meeting the specified constraints. For example, this filter will match the events related to the reading of all sports articles, where articles/sports is derived from the article's URL:
|
In other words, the filter above identifies users that have read at least one sports article.
But what about identifying the set of users that haven't read a single sports article within the same time period? Negations can be tricky, as you might recall from the discussion above, so a combination of not and explodeUsers filters are required to achieve the desired result:
|
Simply wrapping the original and innermost filter inside a not filter would have matched all events related to pages where taxonomy is not set or set to anything other than sports. However, a sports reader might have read such non-sports pages, too, and since we want to exclude sports readers the additional explodeUsers filter is required.
Similar to how cross-event matching was discussed above, the explodeUsers filter must be used when doing cross-profile matching on mutually exclusive keywords. For example, to match users who read articles about both sports and technology:
|
Here, explodeUsers is needed because the categories are mutually exclusive, so there are no pages matching both keywords at once. Otherwise a simple filter without explode would do the job, for example in order to combine category spots and page class article.
Here, too, the user filter's having construct can be used instead of the explodeUsers filter if we want to define restrictions on, e.g., the number of times that sports articles have been read:
|
Matching against user profiles
A segment definition containing a user-keyword filter or a user-external filter will match all events generated by users having the specified properties. For example, to identify users that have shown a longterm interest in sports and that have visited the site within the last two days, we could define the following filter:
|
Because filters applied to a user profile automatically expand to all events from the matching user, the explodeUsers filter isn't required when working with negations or cross-profile matching. The having construct is only needed if we want to set a restriction on the number of events each matching user has generated:
|
|