Introduction
Cxense customers that have also deployed Google Analytics on their pages often report that the Cxense page views and unique users numbers are considerably lower than those of Google Analytics. There are several reasons for such an discrepancy, most of which we will cover in this document, but the main one is that Google Analytics has become a victim of their own success and thus a spam target. The main motivation behind the spam is to improve the ranking of the spammer's web site pages in Google Search search results. This the spammers achieve by creating false referral links back to their web pages which again will increase the perceived importance (authority) of these pages.
Below follows a grouping of the reasons why Google Analytics and Cxense provide different numbers. Not all reasons will necessarily work in the same direction. Even though the net outcome normally is that Google Analytics ends up with higher numbers than Cxense, depending on the deployment, some of the reasons described here could as well contribute towards higher Cxense number.
The groupings are:
-
The way the Cxense and Google scripts are deployed by the customer
-
Instrumentation errors
-
The Cxense script not configured to operate in "Google mode"
-
Google being a spam target
Script Deployment
For the two analytic scripts to count the same numbers they need to be:
-
Deployed on the same set of pages
-
Deployed at the same place on the page
If the two scripts are not deployed on the same set of pages, then the numbers will be different. The more popular the single script pages are, the greater an impact this will have.
If there is distance between the two script on the page, then over time, the script highest up on the page is likely to be executed more times than the one positioned lower on the page. The larger the distance is and the more content that separate them, the larger impact this could have, The reason for this is that more users will have a chance to abort the page somewhere between the two scripts either by closing the page or clicking on a link that takes the browser to another page before the second script has finished.
Instrumentation Error
-
The start and end time matters when comparing the output of the two systems, one must make sure that any difference in time zone is adjusted for
The way one compare numbers is normally by looking at the number shown in the repspective GUIs. If the site is set up with the same time zone in both systems, then no problem. However, if the time zone is differnet then the numbers will also be different. The effect of this is less the longer time intervals one compare. A one hour time zone difference when looking at one hour of data means that one are looking at two completey disjoint data set. The effect of the same time zone issue when looking at one month of data will be marginal.
Cxense Google Mode
-
Google count auto refresh (non-human initiated page refreshes), Cxense does not unless explicitly configured to do so
By setting the useAutoRefreshCheck parameter to false as shown below in line 4, the Cxense script will stop rejecting auto refreshes and report these as page views the same way as Google Analytics does.
<script>
var cX = cX || {}; cX.callQueue = cX.callQueue || [];
cX.callQueue.push(['setSiteId', '<SITE ID>']);
cX.callQueue.push(['sendPageViewEvent', { useAutoRefreshCheck: false }]);
</script>
Google Spam
A significant portion of the page views that Google Analytics reports stem from socalled Ghost Spam. These are page visits that never took place and only exists in the Google statistics. This is what Wikipedia (as of January 17th 2017) writes about ghost spam:
Referrer spam (also known as referral spam, log spam or referrer bombing) is a kind of spamdexing (spamming aimed at search engines). The technique involves making repeated web site requests using a fake referrer URL to the site the spammer wishes to advertise. Sites that publish their access logs, including referer statistics, will then inadvertently link back to the spammer's site. These links will be indexed by search engines as they crawl the access logs, improving the spammer's search engine ranking. Except for polluting their statistics, the technique does not harm the affected sites.
At least since 2014, a new variation of this form of spam occurs on Google Analytics. Spammers send fake visits to Google Analytics, often without ever accessing the affected site. The technique is used to have the spammers' URLs appear in the site statistics, inducing the site owner to visit the spam URLs. When the spammer never visited the affected site, the fake visits are also called Ghost Spam.
The motivation for the spam is to improve one's ranking in Google search result. As Cxense does not have an internet search engine there is no similar motivation for spaming Cxense. Since these page views actually never take place, Cxense will not be counting them.
Cxense Crawler as Google Spam
Cxense Insight does not count page view caused by its own crawler. Google Analytics will do so unless one select the option for excluding page views caused by known robots. This will not only stop counting page crawling by Cxense Insight, but also any other robot that Google is aware of (most "bad spirited" robots will not be stopped by this).
Spam Detection Code
The figure below shows how we can go about detecting ghost and other type of spam. In step 1 we see a user visits a page that contains both the Cxense and the Google Analytics scripts. The Google Analytic one fires of in step 2a and the Cxense one in step 2b. And in step 3 (orange arrow up to the right) we see possible ghost spam hitting the Google Analytics server (remember that Ghost spam never interact the with any of the web pages).
Our strategy is now to add Cxense DMP performance events (red arrows) to the call back function of the Google Analytics pixel requests (step 4a) and to piggy back another one to the the regular Cxense page view event (step 4b). That means that we in the Cxense DMP GUI will be able to see confirmation of any page view reported to either Cxense Insight or Google Analytics. By comparing these numbers with what Google Analytics is reporting in its GUI, we will be able to identify the amount of ghost spam hitting Google Analytics.
The JavaScript code below holds the original page view reporting in step 2a and 2b as well as the DMP performance events in step 4a and 4b:
<script type="text/javascript">
var _prefix = '<YOUR CXENSE CUSTOMER PREFIX>';
var _siteId = '<YOUR CXENSE SITE ID>';
var _persistedQueryId = '<YOUR CXENSE PERSISTED QUERY ID>';
var _googleTrackingID = '<YOUR GOOGLE TRACKING ID>';
function pushPageView(isGoogleEvent, source, sitePrefix, siteId, persistedQueryId, userId, articleId, timestamp) {
var origin = sitePrefix + '-' + source;
var eventType = 'view';
if (isGoogleEvent) {
var userIds = '/type:' + sitePrefix + '/id=' + userId;
var custParamArticleId = '/group:articleId/item:' + articleId + '/type=string';
var custParamTimestamp = '/group:timestamp/item:' + timestamp + '/type=string';
var img = new Image();
img.src = 'http://comcluster.cxense.com/dmp/push.gif?persisted=' + persistedQueryId + '&siteId=' + siteId + '&type=' + eventType + '&origin=' + origin + '&userIds' + userIds + '&customParameters' + custParamArticleId + '&customParameters' + custParamTimestamp;
} else {
cX.setEventAttributes({ 'origin': origin, 'persistedQueryId': persistedQueryId});
cX.callQueue.push(['sendEvent', eventType, {'articleId': articleId, 'timestamp': timestamp}]);
}
}
function getCookie(cookieName) {
var cookies = document.cookie.split(';');
for (var i = 0; i < cookies.length; i++) {
var cookie = cookies[i];
while (cookie.charAt(0) == ' ') {
cookie = cookie.substring(1);
}
var token = cookieName + "=";
if (cookie.indexOf(token) == 0)
return cookie.substring(token.length, cookie.length);
}
return "user ID not found";
}
var userId = getCookie('cX_P');
var articleId = 'articleID not set';
var metaTags = document.getElementsByTagName('meta');
for (var i=0, max=metaTags.length; i < max; i++) {
if (metaTags[i].name === 'cXenseParse:recs:articleid') {
articleId = metaTags[i].content;
}
}
var timestamp = String((new Date).getTime());
var cX = cX || {}; cX.callQueue = cX.callQueue || [];
cX.callQueue.push(['setSiteId', _siteId]);
cX.callQueue.push(['sendPageViewEvent']);
cX.callQueue.push(['invoke', function() {
pushPageView(false, 'cxense', _prefix, _siteId, _persistedQueryId, userId, articleId, timestamp);
}]);
</script>
<script type="text/javascript">
(function(d,s,e,t){e=d.createElement(s);e.type='text/java'+s;e.async='async';
e.src='http'+('https:'===location.protocol?'s://s':'://')+'cdn.cxense.com/cx.js';
t=d.getElementsByTagName(s)[0];t.parentNode.insertBefore(e,t);})(document,'script');
</script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', _googleTrackingID, 'auto');
ga('send', 'pageview', {
'hitCallback': function() {
pushPageView(true, 'google', _prefix, _siteId, _persistedQueryId, userId, articleId, timestamp);
}
});
</script>
The code above requires the use of a persisted query. This tutorial describes what a persisted query is and the screenshot below shows the configuration required in this case.
The JavaScript above will produce DMP performance events (step 4a and 4b) that can be looked at in the Cxense DMP GUI where we see every two lines represent one and the same page view event (same time stamp in the right most purple circle). Each pair consist of a Cxense and a Google performance event as identified by the left most purple circle.
Via the GUI above we can see that the performance events are working as intended. However, it is not a very practical interface for extracting the data we are looking for. For that we use the Python script below that using the Cxense API sums up all performance events to produce a report (step 5 in the overview figure above)
_username = '<YOUR CXENSE USER NAME>'
_secret = '<YOUR CXENSE API KEY>'
_siteId = '<YOUR CXENSE SITE ID>'
_prefix = '<YOUR CXENSE CUSTOMER PREFIX>'
_start = '<START OF TIME INTERVAL>' # Example, Jan 24th 2017 at 5:30 PM CET: 2017-01-24T17:30:00.000+0100
_stop = '<END OF TIME INTERVAL>'
import json, datetime, hmac, hashlib, http.client
def cxApi(path, obj):
date = datetime.datetime.utcnow().isoformat() + "Z"
signature = hmac.new(_secret.encode('utf-8'), date.encode('utf-8'), digestmod=hashlib.sha256).hexdigest()
headers = {"X-cXense-Authentication": "username=%s date=%s hmac-sha256-hex=%s" % (_username, date, signature)}
connection = http.client.HTTPSConnection("api.cxense.com", 443)
connection.request("POST", path, json.dumps(obj), headers)
response = connection.getresponse()
status = response.status
responseObj = json.loads(response.read().decode('utf-8'))
connection.close()
return status, responseObj
def errorHandling(status, response, message):
if status != 200:
raise Exception("%s (http status = %s, error details: '%s')" % (message, status, response['error']))
def getPageViewReportingData(dataType, siteId, start, stop):
if dataType == 'DMP':
status, response = cxApi('/dmp/traffic/data', {"siteIds":[siteId], "start":start, "stop":stop, "fields":["time","pageViewEventId","origin","customParameters"], "filters":[{"type":"dmp-event", "group":"type", "item":"view"}]})
if dataType == 'Traffic':
status, response = cxApi('/traffic/data', {"siteId":siteId, "start":start, "stop":stop, "fields":["eventId","url"]})
errorHandling(status, response, 'Unable to obtain %s Data' % dataType)
return response
def reportFinding(leadText, dataSet, addNewLine=False):
print(leadText + str(len(dataSet)) + ('\n' if addNewLine else ''))
def getGroupedEventData():
googleDmpEvents = {}
cxenseDmpEvents = {}
cxenseTrafficEvents = {}
for event in getPageViewReportingData('DMP', _siteId, _start, _stop)['events']:
eventData = {'time': event['time'], 'pageViewEventId': event['pageViewEventId']}
for customParameter in event['customParameters']:
if customParameter['group'] == 'articleId':
eventData['articleId'] = customParameter['item']
if customParameter['group'] == 'timestamp':
timestamp = customParameter['item']
if event['origin'] == _prefix + '-google':
googleDmpEvents[timestamp] = eventData
if event['origin'] == _prefix + '-cxense':
cxenseDmpEvents[timestamp] = eventData
googleDmpEventTimeStamps = googleDmpEvents.keys()
cxenseDmpEventTimeStamps = cxenseDmpEvents.keys()
cxenseDmpTrafficEventIDs = []
for timestamp in list(set(googleDmpEventTimeStamps) & set(cxenseDmpEventTimeStamps)):
cxenseDmpTrafficEventIDs.append(cxenseDmpEvents[timestamp]['pageViewEventId'])
for event in getPageViewReportingData('Traffic', _siteId, _start, _stop)['events']:
cxenseTrafficEvents[event['eventId']] = {'time': event['time'], 'url': event['url']}
cxenseTrafficEventIDs = cxenseTrafficEvents.keys()
return googleDmpEventTimeStamps, cxenseDmpEventTimeStamps, cxenseDmpTrafficEventIDs, cxenseTrafficEventIDs
def main():
googleDmpEventTimeStamps, cxenseDmpEventTimeStamps, cxenseDmpTrafficEventIDs, cxenseTrafficEventIDs = getGroupedEventData()
reportFinding('Identical Google & Cxense PageView Events = ', set(cxenseDmpTrafficEventIDs) & set(cxenseTrafficEventIDs))
reportFinding('Google Only PageView Events = ', set(cxenseDmpTrafficEventIDs) - set(cxenseTrafficEventIDs))
reportFinding('Cxense Only PageView Events = ', set(cxenseDmpEventTimeStamps) - set(googleDmpEventTimeStamps), True)
main()
Here we see an example output from running the script above:
Identical Google & Cxense PageView Events = 23
Google Only PageView Events = 3
Cxense Only PageView Events = 0
If the total of the first two lines matches with what the Google Analytics GUI shows (make sure you look at the same pages and time period), then you have accounted for all the times the Google script has been executed. If the Google Analytic GUI shows a higher number, then the surplus is ghost spam (step 3, the orange arrow in the overview figure above) where Google Analytics is registering page views that are not caused by the Google Analytics script on the page being executed.
Removing Spam from Google Analytics
There are several sites on the internet that claims to know how you can fix your Google Analytic spam issue. Here are a few:
-
Definitive Guide to Removing All Google Analytics Spam
-
Ultimate Guide to Getting Rid of All the Spam in Google Analytics
-
Ghost Spam & Bot Traffic – Be a Ghost Buster
The essence of them is this collection of advice:
-
Activate built-in filtering options for known robots
-
Filter out host names that don’t match your sites (most spammers don’t know your host names)
-
Add filters for spam crawlers as you detect them (normally by using the Campaign Source field)
-
Filter out page views with non-valid language codes
Single Page Applications
With AJAX and frameworks such as AngularJS, Ember.js, Meteor.js, ExtJS and React that build upon the same principle, single page applications has become quite common. In the context of analytics tools, the difference between a single page application and regular web pages is that the latter will take you to a new URL to show you the next content, whereas a single page application will just have JavaScript code change to content of one and the same page/URL. This creates a problem for analytics tools such as Cxense Insight as the page view counting is triggered by a new URL being accessed. To compensate for this, the Javascript code has to notify Cxense Insight whenever a "new page" is shown. This is done by executing the initializePage() for each "new page":
cX.initializePage();
cX.setSiteId('<YOUR CXENSE SITE ID>');
cX.sendPageViewEvent();
A complete AJAX example can be found at AJAX web applications.