THIS PAGE IS DEPRECATED
For up-to-date information, please follow this link.
Introduction Creating a persisted query programmatically Using a Persisted Query ID from a Web Page Using the Function cX.jsonpRequest() Allowing Request Object Parameters to be Set in the Web Page A more "Real World" Web Page Example Special Case: DMP Audience Segments Creating and Administrating Persisted Queries via the Cxense Admin GUI
Introduction
With a Cxense user account and the corresponding secret API key, one can query almost anything via the Cxense Insight API from a back end CGI script. However, the same approach cannot be used from a front end web page. Anyone can access the web page's code and in that way obtain the user id and the secret API key. This will give the ability to do whatever the owner of that id is authorized to do, and that would include all types of malicious data manipulation.
To safe guard against this we can register a particular API request with Cxense Insight. Cxense Insight will then provide an request id that can then be used to access the registered functionality from within a web page. If someone now extracts the request id from the open source code, it cannot be used to perform any action beyond what the registered API command can do. The figure below illustrate this. The API key gives us access to everything, including the ability to create the request, or the persisted query as we call it. However, unlike the API key, the persisted query id only gives access to the one specific functionality/operation.
The code examples in this tutorial will be HTML/Javascript and Python 3.x. All examples can however be done using the GUI instead of Python (check out the last section about how to create and administrate the persisted queries via the Cxense API GUI).
Creating a persisted query programmatically
The script below can be used to register any API command as a persisted query. The three vital parameters are the command path (/some/command), the request object ({ "param1":"value1", "param2", "value2", "param3":"value3" }), and the list of mutable parameters that can be changed from the web page code ({ "param1":True, "param3", True }).
The script below will return the persisted query ID to be used in the web page to access the registered functionality/operation.
# Access credentials _username = "<YOUR CXENSE USER NAME>" _secret = "<YOUR CXENSE API KEY>" # The persisted query to be registered _path = "<THE API COMMAND PATH>" _request = {<THE REQUEST OBJECT OR NOTHING>} _mutable = {<MUTABLE PARAMETERS OR NOTHING>} import json, datetime, hmac, hashlib, http.client def cxApi(path, obj): date = datetime.datetime.utcnow().isoformat() + "Z" signature = hmac.new(_secret.encode('utf-8'), date.encode('utf-8'), digestmod=hashlib.sha256).hexdigest() headers = {"X-cXense-Authentication": "username=%s date=%s hmac-sha256-hex=%s" % (_username, date, signature)} connection = http.client.HTTPSConnection("api.cxense.com", 443) connection.request("POST", path, json.dumps(obj), headers) response = connection.getresponse() status = response.status responseObj = json.loads(response.read().decode('utf-8')) connection.close() return status, responseObj def errorHandling(status, response, message): if status != 200: raise Exception("%s (http status = %s, error details: '%s')" % (message, status, response['error'])) def createPersitedQuery(path, request={}, mutable={}): status, response = cxApi('/persisted/create', {"path":path, "request":request, "mutable":mutable}) errorHandling(status, response, "Unable to create persisted query") return response['id'] persistedQueryId = createPersitedQuery(_path, _request, _mutable) print("Created persisted query id '%s'" % persistedQueryId)
We will be using the script above in the next few sections.
Using a Persisted Query ID from a Web Page
We will now run the script above with the persisted query configuration given below. With other words, we are requesting information for all sites that we have access to using the API function /site.
# The persisted query to be registered _path = "/site" _request = {} _mutable = {}
Running the script with the settings above will give us a persisted query id, let's say bda7639f305555555c981f86ff46ab7e6014fb97.
We will now use this persisted query id in a JSONP call into Cxense Insight in order to have data for all sites that we have access to being returned. The JSONP template looks like this:
<script src='https://api.cxense.com/COMMAND_PATH?persisted=PERSISTED_QUERY_ID&callback=CALLBACK_FUNCTION_NAME'></script>
We already know the api command path and the persisted query id. In the web page code below we have called the callback function listSites. We can try out the resulting JSONP call before implementing the page below by inserting the url in a browser. We will then see the JSON data that the code below will be receiving from Cxense Insight.
<!DOCTYPE HTML> <html> <body> <script> function listSites(data) { if (data.hasOwnProperty('httpStatus') && data['httpStatus'] == 200) { var names = ''; var sites = data['response']['sites']; for (var i in sites) { names += sites[i]['name'] + '<br>'; } document.getElementsByTagName('body')[0].innerHTML = names; } else { alert('No or bad http status'); } } </script> <script src='https://api.cxense.com/site?persisted=bda7639f305555555c981f86ff46ab7e6014fb97&callback=listSites'></script> </body> </html>
What the callback function does is to extract all the site names and list these on the web page as shown here:
In the approach above we asked to for all sites to which we had access. If we want to limit the listing to only one or a few sites then we can add the siteId or siteIds parameters to the request object as shown here:
# The persisted query to be registered _path = "/site" _request = {"siteId":"9222364973424511877"} _mutable = {}
Using the Function cX.jsonpRequest()
A more convenient way to create the JSONP request is to use the cx.js library function cX.jsonpRequest() as shown below. As we see this makes it possible to use variables rather than fixed text strings.
<!DOCTYPE HTML> <html> <body> <script type="text/javascript" src="http://cdn.cxense.com/cx.js"></script> <script> var path = '/site'; var requestObj = {}; var persistedQueryId = "bda7639f305555555c981f86ff46ab7e6014fb97"; var apiUrl = 'https://api.cxense.com' + path + '?callback={{callback}}' + '&persisted=' + encodeURIComponent(persistedQueryId) + '&json=' + encodeURIComponent(cX.JSON.stringify(requestObj)); cX.jsonpRequest(apiUrl, function(data) { if (data.hasOwnProperty('httpStatus') && data['httpStatus'] == 200) { var names = ''; var sites = data['response']['sites']; for (var i in sites) { names += sites[i]['name'] + '<br>'; } document.getElementsByTagName('body')[0].innerHTML = names; } else { alert('No or bad http status'); } }); </script> </body> </html>
This web page will show exactly the same output as the previous one. The only difference is that in this case we let the function cX.jsonpRequest() take care of all the pluming.
Allowing Request Object Parameters to be Set in the Web Page
In the examples so far there was no way for the (user of the) web page to influence what was being returned. The persisted query was fixed and it was take or leave it. In the next example we will let the (user of the) web page decide for which site we will obtain data. This we do by registering a new persisted query, but this time with the siteId as a mutable parameter. That is, we are allowed to set it from the web page.
# The persisted query to be registered _path = "/site" _request = {} _mutable = {"siteId":True}
Running the Python script with the new settings gives us a new persisted query id, let's say 31caec61462f60555556107384ad246dfe00d127.
The JSONP template for providing additional parameters on the fly is this:
<script src='https://api.cxense.com/COMMAND_PATH?persisted=PERSISTED_QUERY_ID&callback=CALLBACK_FUNCTION_NAME&json={"PARAM1"="VALUE1","PARAM2"="VALUE2"}'></script>
However, the json data has to be url encoded. Hence, the actual JSONP call in the web page will in our case (with a site id of 9222302702321341959) will look like this:
<script src='https://api.cxense.com/site?persisted=31caec61462f60555556107384ad246dfe00d127&callback=listSites&json=%7B"siteId":"9222302702321341959"%7D'></script>
A more "Real World" Web Page Example
The examples so far have been the bare minimum to prove the point. In this next example we will bring a bit more realism to it. At jqueryui.com there is a query completion example that could come pretty close to an actual Cxense Search front end auto completion implementation as soon as we have added the right persisted query to the mix.
The full modified example can be downloaded from here, but the part of essence is shown here below. As we can see, we here use jQuery to take care of the plumbing.
function( request, response ) { $.ajax({ url: "https://api.cxense.com/processing/dictionary/search", dataType: "jsonp", data: { persisted: 'aee984854ddfeecd16e074333f1d3bb0bd67f849', json: JSON.stringify({ input: request.term }) }, success: function( data ) { data = $.map(data['response']['matches'], function(value, i) { return value['key'] + ' :: ' + value['value'].split('|')[0]; }) response( data ); } }); }
However, before this code can work we need to do two things:
-
A query completion dictionary has to be created and uploaded
-
The dictionary lookup operation has to be turned into a persisted query
The example contains a query completion dictionary (the file afi.json) with movie titles that can be uploaded according to the instructions given in Dictionary Management Tutorial (you will have to swap the customer prefix used with one of your own). The persisted query configuration would be as shown below. The dictionary id is obtain from the first step when we created the dictionary.
# The persisted query to be registered _path = "/processing/dictionary/search" _request = {"dictionaryId": 1234567890123456789} _mutable = {"input": True}
Here we see the resulting web page in action:
Special Case: DMP Audience Segments
The extraction of the DMP audience segments is a special case for which there is a bit more support than for other persisted queries. The creation process is the very same one, but the usage in the web page is supported with a cx.js library function called getUserSegtmentId().
This would be the query configuration:
# The persisted query to be registered _siteGroupIds = [<COMMA SEPERATED LIST OF QUOTED SITE GROUP IDS>] _path = "/profile/user/segment" _request = {"identities":[{"id":"0","type":"cx"}],"siteGroupIds":_siteGroupIds} _mutable = {"identities":True}
Running the Python script with the new settings gives us a new persisted query id, let's say 5d2c2ebfdbc41cc555551ae63fdd0b5516d812de.
<!DOCTYPE HTML> <html> <body> <script type="text/javascript" src="http://cdn.cxense.com/cx.js"></script> <script> var segments = cX.getUserSegmentIds({persistedQueryId:'5d2c2ebfdbc41cc555551ae63fdd0b5516d812de'}); alert("The user is member of the following DMP audience segments: " + segments); </script> </body> </html>
Creating and Administrating Persisted Queries via the Cxense Admin GUI
It is possible to do everything we have done so far without writing/executing a line of Python code. Below we see how to access the Persisted Query administration page and how to create create/update a persisted query.
Below we see what it would be like to create the very same mutable site id site information query that we created earlier using the Python script.
