Novelty Tutorial

This 12 min video demonstration walks through a Jupyter notebook powered scenario illustrating how to use thatDot Novelty to analyze CDN logs for anomalous activity.

Click here to download the CDN dataset for this example.

‍Download the Jupyter notebook and try the demo yourself with an AWS instance of thatDot Novelty.

Demo Summary

Novelty Score Endpoints

The demo interacts with thatDot Novelty through its interactive REST API. You can stream observations into thatDot Novelty using one of two API endpoints:

• Single observation: POST /api/v1/novelty/{context}/observe
• Bulk observations: POST /api/v1/novelty/{context}/observe/bulk

After streaming in a batch of observations, you can rescore observations given the context of the entirety of the dataset using Novelty’s read-only scoring endpoints:

• Single observation: POST /api/v1/novelty/{context}/read
• Bulk observation: POST /api/v1/novelty/{context}/read/bulk

Novelty Score Results

thatDot Novelty’s Score Results response returns the observation score, along with additional useful information. Here is some of that data:

• observation: The observation that was streamed in to generate the result. A list of string observation components
• score: score between 0 and 1 representing the most novel component of this observation. 1 is highly novel, 0 is not novel at all: the mostNovelComponent field contains more details for which component led to this result
• mostNovelComponent: which component of the observation was the most novel
• sequence: sequence number assigned to uniquely identify this observation as made within this context.
• uniqueness: scaled measure of uniqueness for the observation as a whole; ranges between 0 (no uniqueness) and 1 (totally unique)

Important Points

• Unique does not mean novel. Sometimes, completely unique and unseen observations can be normal, as described in the Demo when showing the normalcy of having completely unique IP addresses in a certain scenario
• thatDot Novelty does not require training, but does take a bit of time depending on the use case to adapt to the data