We’ve migrated our documentation to a new site, which means some URLs have changed.
Audience
Audience
Breadcrumbs

Dictionary Management Tutorial

English   | 日本語

Introduction

The following Cxense products facilitate customer management of dictionaries.

  • Cxense Insight

    • Custom taxonomy dictionary

      • user interest detection

      • user intent detection

  • Cxense Search

    • Query modification dictionaries

      • term completion

      • synonym expansion

Cxense supports two types of dictionaries, search related and taxonomy related dictionaries. Search related dictionaries are used for key/value lookups. That is, given a complete or partial search term as input key, an alternative search term (for instance a synonym) or a full term (a query completion term) will be returned as the output value. Taxonomy related dictionaries are different. In their case the input is not single query terms, but rather the full web page content (user interest detection) or the list of URL parameters (user intention detection).

Dictionary management involves the following operations:

  1. Instance creation - a Cxense Data Storage location is allocated and a reference id is produced to be used for all future references

  2. Purpose configuration - this operation only relates to taxonomy dictionaries: Determines if the input will be the web page content (user interest detection) or the web page url query parameters (user intent detection)

  3. Content uploading/updating - First time upload or later updates of the dictionary content in the form of a Excel spreadsheet or JSON

  4. Usage - this only relates to search as the use of taxonomy dictionaries is all managed by the system under the hood: Do dictionary key-value lookups and use the returned value to modify the search query

  5. Deletion - Dictionary content removal and dictionary instance deletion (dictionary id liberation)

With the exception of the operation purpose configuration in step 2) that only applies to taxonomies, all these operations can be carried out by the customer. Step 2) can only be carried out by someone from Cxense. Please contact your onboarder or support@piano.io if you are about to configure your own custom taxonomy.

Dictionary Input Formats

The dictionary content that is to be uploaded in step 3) can be uploaded as an Excel file or as a JSON data structure. Below we will  see the very same dictionary content as JSON and as Excel. However, in all further examples in this tutorial we will only be using the JSON format.

JSON as Input Format

Here we see a JSON formated sample dictionary. Notice how the dictionary consists of algorithm settings (global-properties) and data (synonyms). These dictionaries will only have one table for algorithm settings, but can have any number of data tables.

{
    "synonyms": {
        "tv": "television",
        "television": "tv",
        "car": "automobile",
        "automobile": "car",
        "ipod": "mp3 player"
    },
    "global-properties": {
        "group-prefix": "sds",
        "mode": "complete",
        "leftmost-longest-match": true,   
        "key-normalization-flags": 4,
        "value-normalization-flags": 23
    }
}

Excel as Input Format

Here we see the Excel version of the JSON formatted dictionary above. Notice how the algorithm settings and data tables above correspond to data sheets in this Excel version.

Unknown Attachment

The '@@' data sheet name prefix seen at the bottom of the figure above is mandatory for Excel, but optional for JSON.

Dictionary Management Operations - Function by Function

Before we start the practical part of this tutorial, we will in this section go through the API functions to use for each of the five dictionary management operations described earlier. After all the API functions, there will be a Python 3.x test script with all the same functionality. We will be using that test script actively as we learn about and try out the various configuration options.

In the examples in this section we use the cx.py tool.

Dictionary Instance Creation

One create a new dictionary instance with the API function /processing/dictionary/create:

python cx.py /processing/dictionary/create '{"siteGroupId":"1234567890123456789", "name":"My dictionary", "description":"Test dictionary"}'

The function above will return a dictionary id that is to be used for all later references to the dictionary.

Dictionary Purpose Configuration

If the dictionary is a taxonomy dictionary, then it is necessary to configure its purpose. It is either to be used for detecting user interest or user intent. The difference between the two is what is used as the input, the web page content (user interest) or the url query parameters (user intent). Only Cxense engineers can do the config operation (link only accessible to Cxense employees). Contact your onboarder or support@piano.io if you need to do this.

Dictionary Content Uploading / Updating

It is possible to upload dictionary content using any of two alternative formats: As A base64 encoded xslx file or as a JSON string.

python cx.py /processing/dictionary/update '{"dictionaryId":dictionaryId, "workbook":"<base 64 encoded xlsx spreadsheet formatted dictionary>"}'
python cx.py /processing/dictionary/update '{"dictionaryId":dictionaryId, "entries":"<dictionary as json formatted string>"}'

In this case, due to the dictionary content formatting requirements, one should consider use of this ready to execute Python dictionary uploading script. 

Dictionary Usage

One does dictionary lookups with the API function /processing/dictionary/search. In this example we are looking up the term restaur in the query completion dictionary:

python cx.py /processing/dictionary/search '{"dictionaryId":"1234567890123456789", "input":"restaur"}'

Dictionary Deletion

One delete a dictionary with the API function /processing/dictionary/delete:

python cx.py /processing/dictionary/delete '{"dictionaryId":"1234567890123456789"}'

Dictionary Configuration Test Script

The Python 3.x script below carries out all the operations listed above. In addition it provides a test framework for testing out the dictionary configurations we will be learning about in the next chapter. 

Python
_username       = '<YOUR USER NAME>'
_secret         = '<YOUR SECRET API KEY>'
_siteGroupId    = '<YOUR SITE GROUP ID>'
_prefix         = '<YOUR SITE GROUP PREFIX>'
 
# This will allow both json and Python syntax for the properties
false = False
true  = True
  
import json, base64, datetime, hmac, hashlib, http.client
  
def cxApi(path, obj):
    date = datetime.datetime.utcnow().isoformat() + "Z"
    signature = hmac.new(_secret.encode('utf-8'), date.encode('utf-8'), digestmod=hashlib.sha256).hexdigest()
    headers = {"X-cXense-Authentication": "username=%s date=%s hmac-sha256-hex=%s" % (_username, date, signature)}
    connection = http.client.HTTPSConnection("api.cxense.com", 443)
    connection.request("POST", path, json.dumps(obj), headers)
    response = connection.getresponse()
    status = response.status
    responseObj = json.loads(response.read().decode('utf-8'))
    connection.close()
    return status, responseObj
  
def errorHandling(status, response, message):
    if status != 200:
        raise Exception("%s (http status = %s, error details: '%s')" % (message, status, response['error']))
 
def createDictionary(siteGroupId, dictionaryName, description=''):
    status, response = cxApi('/processing/dictionary/create', {"siteGroupId":siteGroupId, "name":dictionaryName, "description":description})
    errorHandling(status, response, 'Unable to create dictionary')
    return response['dictionaryId']
 
def getAllDictionaries(siteGroupId):
    status, response = cxApi('/processing/dictionary/read', {"siteGroupId":siteGroupId})
    errorHandling(status, response, 'Unable to obtain dictionaries')
    return response['dictionaries']
 
def deleteDictionary(dictionaryId):
    status, response = cxApi('/processing/dictionary/delete', {"dictionaryId":dictionaryId})
    errorHandling(status, response, 'Unable to delete dictionaries')
 
def deleteAllDictionaries(siteGroupId):
    for dictionaryId in [dict['dictionaryId'] for dict in getAllDictionaries(siteGroupId)]:
        deleteDictionary(dictionaryId)
 
def uploadDictionary(dictionaryId, dictFilePath):
    if dictFilePath.lower().endswith('xlsx'):
        requestObj = {"dictionaryId":dictionaryId, "workbook":base64.b64encode(open(dictFilePath, 'rb').read()).decode('ascii')}
    else:
        requestObj = {"dictionaryId":dictionaryId, "entries":json.loads(open(dictFilePath, 'rb').read().decode('utf-8-sig'))}
    status, response = cxApi('/processing/dictionary/update', requestObj)
    errorHandling(status, response, 'Unable to upload new version of dictionary')
    if 'messages' in response and response['messages']:
        print ('\n'.join(response['messages']))
 
def dictionaryLookup(dictionaryId, input):
    status, response = cxApi('/processing/dictionary/search', {"dictionaryId":dictionaryId, "input":input})
    errorHandling(status, response, "Unable to do dictionary lookup")
    return response
 
def writeTestDictToFile(filepath, data, properties):
    testDict = {
        "myMappings": data,
        "global-properties": properties
    }
    testDict['global-properties']['group-prefix'] = _prefix
    with open(filepath, 'w') as f:
        f.write(json.dumps(testDict, indent=4, sort_keys=True))
 
def runLookupTest(dictionaryId, testData, testProperites, testQuery, expectedResult):
    # Creates file with new dictionary content and updates dictionary entry
    dictFilePath = 'testDict.json'
    writeTestDictToFile(dictFilePath, testData, testProperites)
    uploadDictionary(dictionaryId, dictFilePath)
    print('New dictionary uploaded "%s"' % testData)
    import time
    time.sleep(1)
 
    # Test to see if the lookup is working as expected
    lookupValue = dictionaryLookup(dictionaryId, testQuery)
    if len(lookupValue['matches']):
        lookupValue = [match['value'] for match in lookupValue['matches']]
    else:
        lookupValue = None
    expected = ''
    if str(lookupValue) == str(expectedResult):
        result = 'SUCCESS'
    else:
        result = 'FAILURE'
        expected = ', but expected "%s"' % expectedResult
    print('%s: Looking up "%s" gives "%s"%s' % (result, testQuery, lookupValue, expected))
    
def setUpAndRunTest(testData, testProperites, testQuery, expectedResult):
    # Deletes all dictionaries and verifies that they are gone
    deleteAllDictionaries(_siteGroupId)
    existingDictionaries = getAllDictionaries(_siteGroupId)
    if len(existingDictionaries):
        raise Exception('Not able to delete these dictionaries: %s' % existingDictionaries)
    # Creates test dictionary and execute test
    dictionaryId = createDictionary(_siteGroupId, 'test dict')
    runLookupTest(dictionaryId, testData, testProperites, testQuery, expectedResult)
     
def main():
 
	# YOUR DICTIONARY GOES HERE
    data = {
        "one": "first",
        "two": "second"
    }
    properties = {
        "group-prefix": "sds",
        "mode": "complete"
    }
 
	# YOUR DICTIONARY TESTS GO HERE
    setUpAndRunTest(data, properties, testQuery="one", expectedResult=['first'])
    setUpAndRunTest(data, properties, testQuery="one of many", expectedResult=None)
    setUpAndRunTest(data, properties, testQuery="twoooooooooo", expectedResult=['second'])
 
main()


Below you see the output from running the 3 tests listed at the end of the main() function above. Notice how we purposely set the third test up with erroneously data so that we got to see  what a failed test would look like. As you work your way through the next section, you will be using this script to test out the various dictionary content and configurations.

New dictionary uploaded "{'two': 'second', 'one': 'first'}"
Lookup test SUCCEEDED (looking up "one" gives "['first']")
New dictionary uploaded "{'two': 'second', 'one': 'first'}"
Lookup test SUCCEEDED (looking up "one of many" gives "None")
New dictionary uploaded "{'two': 'second', 'one': 'first'}"
Lookup test FAILED (looking up "twoooooooooo" gives "None")


The 'Hello World!' Dictionary Config

We will now make the simplest dictionary configuration possible. The 'Hello Word' dictionary example.

JavaScript
{
    "myMappings": {
        "hi world!": "hello world!"
    },
   "global-properties": {
        "group-prefix": "sds"
    }
}

In this dictionary we have one single lookup entry and no global properties beyond the customer site group prefix ('sds' in this case - Search Demo Site). 


We can now update the main() function of the test script as shown below with the dictionary and with a test.

Python
# YOUR DICTIONARY GOES HERE
data = {
    "hi world!": "hello world!"
}
properties = {
    "group-prefix": "sds"
}
# YOUR DICTIONARY TESTS GO HERE
setUpAndRunTest(data, properties, testQuery="hi world!", expectedResult=['hello world!'])

The output from running the test is shown here:

Python
New dictionary uploaded "{'hi world!': 'hello world!'}"
SUCCESS: Looking up "hi world!" gives "['hello world!']"

And here we see the exact output that is being returned when doing the dictionary lookup:

Python
print(dictionaryLookup(_dictionaryId, 'hi world'))
 
OUTPUT:
{'matches': [{'value': 'hello world!', 'sheet': 'sds-categories', 'key': 'hi world!'}]}

With this there is no need to shown any more Python code. For any of the cases explained below, update the dictionary and the tests and run the script.

Dictionary Property Settings

In this section we will be adding properties one by one to the property sheet and see how each setting influences the behavior of the dictionary. A complete list of the settings covered in this tuitorial as well as many others are described here.

tokenizer-context

The tokenizer needs to know the language of the data in order to process the data correctly. For this we use the tokenizer-context property. English ('en') is the default language.

JavaScript
{
    "myMappings": {
        "hi world!": "hello world!"
    },
   "global-properties": {
		"group-prefix": "sds",
		"tokenizer-context": "en"
    }
}

A full list of the ISO 639-1 language codes that we use for this property can be found here.

mode

The maybe most important property is the (matching) mode. It defines how the dictionary's keys should be matched against the content of the input string in order to produce a match.

JavaScript
{
    "myMappings": {
        "Tokyo": "Japanese city",
		"Beijing": "Chinese city"
    },
   "global-properties": {
        "group-prefix": "sds",
		"mode": "complete"
    }
}

Here is a list of some of the available modes (complete is the default mode setting). A full list of modes can be found here.

Key/Input, end to end matching modes

    • complete - the input text has to be identical to the key

      • Tokyo is the only input that will result in Japanese city being returned

    • didyoumean - the input and the key is within a certain spelling distance. This is the mode used for query spell checking.

      • Currently not supported

Input position modes

    • head - the key matches the beginning of the input text

      • The input Tokyo is a big city will result in Japanese city being returned

    • overlap - the key appear anywhere within the input

      • The input I go to Tokyo to every year will result in Japanese city being returned

    • tail - the key matches the end of the input text.

      • The input I love Tokyo will result in Japanese city being returned

Key position modes

    • prefix - the input match the beginning of the key. This is the mode used for query completion.

      • The fragments T, To, Tok, etc. from the beginning of the word Tokyo, as well as the complete word itself, will all result in Japanese city being returned

    • phrase - The input string is an infix of the key (is in the middle of the key)

      • The input oky will result in Japanese city being returned

    • keystroke - the input is the beginning of the key when entered using a CJK Input Method Editor (for instance 't', 'と', '東k' represent 3 different stages of writing the beginning of the Japanese word "東京" (Tokyo) using an Japanese IME). The mapping key is the complete word ("東京" in this case) and the mapping value the category/group or similar that the key is to part of (for instance 'Japanese cities').

      • The input 't', 'と' or '東k' will all result in japanese city being returned given the configuration below 

JavaScript
{
    "myMappings": {
        "東京": "Japanese city",
        "北京市": "Chinese city",
    },
   "global-properties": {
        "group-prefix": "sds",
        "tokenizer-context": "ja",
		"mode": "keystroke"
    }
}

count, unique-count, count-field

The mode property explained in the previous section determines what constitues a match. In this section we will look at the properties count and unique-count that determine the minimum number of matches (unique ones and in total) for a non‐empty result to be returned.

Below we have set the matching mode to overlap to be able to find a term anywhere in the input string. In addition we have added the properties count and unique-count. For an input string to produce a result with the configuration given below, the string needs to contain a minimum of 3 Japanese cities of which at least 2 of them need to be unique city . Hence, the input strings Tokyo Osaka Kyoto and Tokyo Tokyo Osaka will both produce results, whereas Tokyo Tokyo Tokyo (only one unique city) and Tokyo Osaka (only 2 cities in total) will both fail to return anything

JavaScript
{
    "myMappings": {
		"Tokyo": "Japanse city",
		"Osaka": "Japanse city",
		"Kyoto": "Japanse city",
    },
   "global-properties": {
		"mode": "overlap",
        "group-prefix": "sds",
		"count": 3,
        "unique-count": 2,
        "count-field": "none"
    }
}

Property explanations:

  • count - the minimum number of key matches mapping to one and the same value (the default value is 1)

    • The input string toyota toyota toyota will pass the limit of a minimum count of 3 and result in the return of ['car', 'car', 'car']. The input string toyota toyota will fail to return anything as the count is 2.


  • unique-count - the minimum number of unique key matches mapping to one and the same value (the default value is 1)

    • The mapping string toyota ford will pass the limit of a minimum count of 2 unique key matches and result in the return of ['car', 'car']. The input string toyota toyota will fail to return anything as the unique-count is 1.


  • count-field - Hard to explain. Use "none' for all search related dictionaries and "value" for taxonomies (the default value is "none")

key-normalization-flags, value-normalization-flags

So far we have been operating with exact matches. In our last example, the key toyota would not have given a match for the capitalized word Toyota in the input string. However, differences like this one can be ignored by setting the key-normalization-flags. 3 types of normalization is active by default and cannot be turned off, leading whitespace removal ("   Toyota" => "Toyota") and whitespace reduction between words ("Japanses     car" => "Japanses car") and Unicode canonization (including Japanese halfwidth and fullwidth normalization).

Below we see the various flags for the nine different type of normalization supported. Most of them come with a normalization example.

JavaScript
// Propterty values with corresponding normalization examples
// default        =    0        No normalization
// caseNorm       =    4        Madrid                          =>    madrid
// accentNorm     =    8        München                         =>    Munchen
// urlDecodedNorm =   64        Rio%20de%20Janeiro              =>    Rio de Janeiro
// urlNorm        =  128        http://where.no/city.html#Oslo  =>    where.no/city.html
// indicNorm      =  256        -                               =>    -
// scandiNorm     =  512        Måløy                           =>    Maloy
// arabicNorm     = 1024        -                               =>    -  
// turkishNorm    = 2048        -                               =>    -
// kanaNorm       = 4096        ロンドン (London)                =>    ろんどん (London)
 
{
    "myMappings": {
		"München": "German city"
    },
    "global-properties": {
        "group-prefix": "sds",
        "key-normalization-flags": 12,      // caseNorm + accentNorm,
        "value-normalization-flags": 4      // caseNorm
    }
}

In the configuration above we activated both case and accent normalization. Hence, any variation of Münich (Münich, münich, munich, MUNICH, etc) in the input string will result in a match. Likewise we can use the same flags with the value-normalization-flags to normalize the value. In the example above we apply case normalization to the output value and any match for Münich will return the term city rather than City.

leftmost-longest-match

With some modes there is no doubt about what should be returned. For instance, with mode complete only exact matches are valid. With other modes, it is not always that easy. Below we see a dictionary with mode set to 'overlap' and with mappings for the two cities York and New York. Which of these should be returned for the input string I love New York Yankees. In this case there is not much doubt, we would only want the US city to be returned, but what if the entries were Manchester ('UK city') and Manchester United ('UK soccer club'). In that case, depending on the usage of the dictionary, we may want both to be reported.

{
    "myMappings": {
        "New York": "US city",
        "York": "UK city",
    },
    "global-properties": {
        "group-prefix": "sds",
        "mode": "overlap",
        "leftmost-longest-match": True
    }
}

Setting the property leftmost--longest–match to true will make the dictionary only accept the longest match of the two (New York => US city). In case it is set to false, then both matches would be returned (New York => US city, York => UK city). True is the default value.

Configuring Dictionaries for their Purpose

At this point we have now learned most of the property settings that we will ever come across or that we will be using ourselves. It is now time to put that knowledge into action by implementing dictionaries for the various use cases. A few additional use case specific properties will be introduced in these sections beyond what has been covered earlier.

Taxonomy Dictionary

Cxense Insight support the ability to activate a standard Cxense taxonomy and/or upload a custom specific taxonomy. More on the use of taxonomies is covered in Taxonomy Setup Procedure. Below follows a typical taxonomy dictionary configuration.

{
    "location": {
		"Tokyo": "city/Japan",
		"Beijing": "city/China",
        "Mont Blanc": "mountain/France",
        "Nile": "river/Egypt"
    },
	"organization": {
		"FCC": "governmental/US",
		"Save the Children": "humanitarian/UK"
    },
    "global-properties": {
		"group-prefix": "sds",
        "tokenizer-context": "en",
        "mode": "overlap",
        "leftmost-longest-match": true,
        "key-normalization-flags": 4,
        "value-normalization-flags": 4,
        "count": 2,
        "unique-count": 2,
        "count-field": "value",
        "expand-paths": true,
        "annotate-paths": false
    }
}

Taxonomy specific property settings:

  • expand-paths - expands the taxonomy paths with all variations (the default value is false)

    • If set to true, the input string Tokyo will result in the values ['city', 'city/Japan'], if false, then only  ['city/Japan']


  • annotate-paths - insert the name of the mapping table as the root path element (the default value is false)

    • If set to true, the input string Tokyo will result in the values ['location/city/Japan'], if false, then only  ['city/Japan']

Synonym Expansion Dictionary

Below follows a typical synonym expansion dictionary.

{
    "synonyms": {
		"tv": "television",
        "television": "tv",
        "car": "automobile",
        "automobile": "car",
        "ipod": "mp3 player"
    },
    "global-properties": {
		"group-prefix": "sds",
		"tokenizer-context": "en",
		"leftmost-longest-match": true,    
        "allow-complete-match": true,
        "key-normalization-flags": 4,
        "value-normalization-flags": 12,
        "swap-fields": true,
        "count": 1,
        "unique-count": 1,
        "count-field": "none"
    }
}

Query Completion Dictionary

Query completion is done using two dictionaries, suggest-whitelist and suggest-blacklist. The former lists terms to be suggested, the latter terms that one don't want to suggest (for instance foul language). An excellent source for the suggest-whitelist are past queries that gave a non-zero-hits result (called matching queries), as these are proven queries. How to obtain all the matching queries is explained in Obtaining Query Statistics (see Cxense Search Query Side Tutorial). One can either loop over a long period of time (the report is per 24 hours based on UTC time) and accumulate the terms in that way, or one could have a daily run that adds all new terms from the day before and then upload the updated version of the dictionary.

Below we see an example of a suggest-whitelist dictionary. The dictionary will provide the search user with full length search terms based on the current partial input from the user. Hence, if someone starts entering "tok", the full term "tokyo" could be suggested.

{
    "whitelist": {
		"new york": "new york",
        "new delhi": "new delhi",
		"tokyo": "tokyo",
    },
    "global-properties": {
		"group-prefix": "sds",
		"tokenizer-context": "en",
		"mode": "prefix",
        "key-normalization-flags": 4,
        "value-normalization-flags": 4, 
    }
}


Last updated: