/profile/content/fetch | Documentation

Looks up and returns a precomputed content profile for a given URL, if this exists.

The user must be authenticated and have read permissions to the associated site.

Request

The request object has the following fields:

Name	Type	Required	Description
`url`	String	Yes	The URL to retrieve a content profile for. Url encoding needed for special characters (non-english). The URL can be specified with or without scheme.
`flattenProfile`	Boolean	No	Return profile objects in the flattened format
`groups`	Array of Strings	No	A list of strings that specify profile item groups to keep in the returned profile (see below for descriptions of the `profile` field in the response object). If not specified, all groups available for the content will be returned.

Response

The response object has the following fields:

Name	Type	Description
`url`	String	The URL of the page, corresponding to the URL of the request.
`id`	String	Unique identifier for the content profile. This will be the same for different URLs that are considered equivalent according to our normalization algorithm. For example http://www.example.com, http://www.example.com/ and https://example.com will all have the same `id` value.
`siteId`	String	The site identifier for the URL. This is either from the content of the page itself, or based on the site identifier of index pages on the same domain.
`httpStatus`	Integer	HTTP status response from the last time the page was fetched.
`lastFetched`	String (Date)	ISO 8601 timestamp representing the last time the page was fetched.
`firstFetched`	String (Date)	ISO 8601 timestamp representing the first time the page was fetched. (optional)
`canonicalUrl`	String	Canonical URL as calculated based on incoming events and the fetched page content.
`title`	String	Extracted title from the last time the page was fetched.
`profile`	Array of Object	Array of profile (or flattened profile, if `flattenProfile` request parameter is set to `true`) objects containing the generated content profile for the URL. Profile objects are defined below.
`thumbnails`	Array of Object	Array of thumbnail objects representing an image of the page from when the page was last fetched. Thumbnail objects are defined below.
`source`	String	Application source: documentUpdate (for API server handler `/document/update`), aurora-worker, dev-k8s-worker, prod-k8s-worker. If there is no field, then it is most likely aurora-worker.

If no content profile exists, an object containing only the url and id pairs will be returned.

A profile object has the following fields:

Name	Type	Description
`item`	String	Item extracted or generated form the page content. Usually a string or keyword extracted from the page.
`groups`	Array of Object	Array of profile group objects associated with the item.

A profile object group has the following fields:

Name	Type	Description
`group`	String	Represents the category or type of information.
`weight`	Number	Indicates the relative prominence of the item/group combination.

A flattened profile object has the following fields:

Name	Type	Description
`item`	String	Item extracted or generated form the page content. Usually a string or keyword extracted from the page - the same value as `item` in the raw (i.e. not flattened) profile object.
`group`	String	Represents the category or type of information - matches `group` in the raw profile group object.
`weight`	Number	Indicates the relative prominence of the item/group combination - matches `weight` in the raw profile group object.
`count`	Number	Represents the item (e.g. keyword) frequency in the given context - matches `count` in the raw profile group object.

A thumbnail object has the following fields:

Name	Type	Description
`width`	Integer	The width of the thumbnail in pixels.
`height`	Integer	The height of the thumbnail in pixels.
`url`	String	Either a data URL representing the thumbnail itself, or an URL pointing to an image service that will return the thumbnail.
`type`	String	Identifies the type or nature of the thumbnail. E.g., if it's a screenshot or something else. Currently valid values are `screenshot` or `dominant`.

Examples

Bash

$ python cx.py /profile/content/fetch '{"url":"https://www.example.com"}'
{
 "canonicalUrl": "https://www.example.com",
 "httpStatus": "200",
 "id": "0caaf24ab1a0c33440c06afe99df986365b0781f",
 "lastFetched": "2012-10-17T16:18:18.876Z",
 "firstFetched": "2012-08-02T11:14:42.711Z",
 "profile": [
 {
 "groups": [
 {
 "count": 1,
 "group": "concept",
 "weight": 1.0
 }
 ],
 "item": "examples in documents"
 },
 {
 "groups": [
 {
 "count": 1,
 "group": "site",
 "weight": 0.07038932789139794
 }
 ],
 "item": "example.com"
 },
 ],
 "siteId": "-1",
 "thumbnails": [
 {
 "type": "screenshot",
 "height": 150,
 "url": "https://analytics.cxense.com/analytics/ImageService/Thumbnail200x150_DKrySrGgwzRAwGr%29md%29YY2WweB8%3D",
 "width": 200
 }
 ],
 "title": "IANA \u2014 Example domains",
 "url": "https://www.example.com"
}

Last updated: December 28, 2023