How does Anomaly Detection work?

At the foundation of every data science and business intelligence project is data accuracy and consistency. All too often our updates, releases and changes break or adversely affect various data sources and it takes days if not weeks to figure out when something is broken. This is why we invented Anomaly Detection as a product that allows businesses to monitor thousands of data streams at scale and get notified when something does not look right.

Anomaly Detection helps to:

  • Improve data accuracy
  • Quickly identify trends
  • Monitor 1000’s of data streams simultaneously
  • Get notified only about important changes in your data flows
  • Uncover the true causes of increases and decreases in data patterns
  • Better understand business drivers

The chart below represents daily number of visitors for a website ACME that sells products to various customers around the world. Around December 5 there was an unusual spike in number of users that have logged in. This will get easily noticed by analysts.

But what if ACME data is further broken down by gender, number of countries and devices used to connect with the website? How do they know when female users from Spain reduced their spend by 30% because your latest translation release misspelled the most popular product purchased by them? Or that the latest update of the website rendered it unusable for Android users in the UK?

Doing a simple math of 100 countries x 2 genders x 4 devices = 800 data streams to monitor! Even if we choose to look after TOP 20 countries, that is a LOT of data streams to look at every single day. And each data stream could have a unique seasonal or weekly pattern that it follows. Below is a summary of all data streams that ACME would have to monitor daily.

When ACME decided to use our Anomaly detection solution, one of the things they immediately noticed was the increased usage in early December from iOS users from Germany

In more technical terms, they were able to observe normality ranges for historical visits from iOS users in Germany and how early december results send the data outside the normal range, thus creating an anomaly.

Ironically, this coincided with the latest product release by ACME, where they improved the performance of their product for iOS devices and followed with an advertising campaign targeted at German users.

This was just a simple example of only a few countries, devices and genders. Imagine business that has multitude of products, sold by multitude of channels across countries. Or an online store that not only tracks device and country, but also browser type and version across multiple pages and services that it offers online. The effort required to monitor and notice an issue with a single data stream is intensive and could take days if not weeks for the analysts to figure out the true cause of increases or decreases in data trends.

How does it work?

Our approach to Business Intelligence is that every solution must be simple to use. This is how we designed our Anomaly Detection. You provide just one data set and our algorithm will not only parse the data, but it will create all of the necessary data streams for you and automatically monitor your data every day, sending you a notification when something appears unusual.

What about seasonality?

Our algorithm is designed to detect seasonality in each data stream and follow the seasonal trends building confidence intervals as the data fluctuates by day of the week. You will receive automatic notifications of anomaly only when the data is abnormal within the context of the seasonality trend of your business.

Monitoring 1000’s data streams daily

Let’s say you have a website that services millions of users per day and the latest update rendered your product unusable for a particular version of Internet Explorer. This may be insignificant decline in your overall data set as IE 10 (for example) accounts for less than 5% of the browser market share (reference), but for a company with 500,000 visitors daily this means that 25,000 users are not able to use the website. A small 5% drop within hundreds of thousands of sessions daily may be difficult to attribute manually to a particular browser type, our data parsing mechanism breaks up your data set automatically into relevant categories and within seconds will notify you about the cause of your discrepancies. We will also notify you when the data stops flowing altogether.

Features

One data file

Unlike many other companies, we do not force you to provide summary for each individual category to monitor. You set up just one file with all of the categories that you want to monitor and our data parsing system will summarise the data across all categories for you. In our earlier example, the company would provide one file with sales summarized by date, country, distributor and product, letting our system do the magic for you.

Flag known Anomalies

We understand that sometimes things do not work for known reasons. For example, a recent product issue may stop sales of product X for 2 days to 0 items per day, but this was a willful act of the company and should not be considered an anomaly. Our flexible system will allow you to flag those known anomalies so that the algorithm understand that they are unusual on purpose and these data points should not be used for machine learning.

Replacing data points

Once the known anomalies have been identified, our system allows the user to add correct measures for known anomalies. Building on the example above, if due to a recall the sales of product X dropped to 0 for Thursday and Friday. But the management knows that this is an unusual circumstance and usually the company sells 24,000 items of product X on thursday and 30,000 on Fridays (on average). We allow for the override of known anomalies in the data to ensure that our system learns from the accurate data that is in line with normal business expectations rather than unusual anomalies. Our system is also designed to disregard the known anomalies by adding its own interpretation:

  • If the point is a known anomaly and custom value is provided, it will be used during training
  • If the point is a known anomaly and no value is provided:
    • the value of the closest corresponding day from the week in the past will be used
    • If no information from the past is available and no custom value is provided, average day value for the current week will be used

Rollback to any historical data point

We understand that sometimes the data provided to us may be incomplete or partially correct and more accurate data set can be provided at a later point. Our flexible system allows the user to roll back the anomaly tracking to any point in the history and build new history based on an updated data set. So, if a company noticed that since Monday last week the data set excluded an important channel (by mistake), our algorithm and all history can be rolled back to last Monday and re-built from scratch with inclusion of new data.

Full history

We keep all history of

  • Data sets that are provided to us
  • All data streams that are built from it
  • Execution results of anomaly detection algorithms

Adjust algorithm sensitivity

When the business data fluctuates extensively, our algorithms can be fine-tuned to be more relaxed to detect only extreme anomalies. This way we avoid spamming our users with the notifications about anomalies that can be well within the business expectations.

Machine learning

We have build machine learning right into the core of our anomaly detection. As your data changes and your business growth, our system learns to adapt to those changes, so your product sales that significantly increase in time will be considered as the new norm moving forward and any further anomaly detections will be done against it.