Blog & News

Anomaly detection with automation and NLG: quickly identify and understand outliers

By Cammy Moore | April 13, 2022
detect anomalies and outliers with NLG

Sometimes, important facts in your data jump right out at you. But most of the time, you look at your dashboard or your spreadsheet, and while you’re finding answers to questions you have, you are simultaneously missing critical insights that answer questions you don’t have. That’s because anomalies and outliers are better detected by machines than humans. When the detection of anomalies is automated, you can’t miss them; when anomalies are explained in dynamic, natural language that updates instantly, you can’t miss their significance.

Making data-driven decisions means knowing what is happening in the data. It means knowing what, when, and why results deviate from the representative — or expected — model. The faster an anomaly is brought to your attention, the sooner you can respond.

As the volume of organizational data increases, so do the number of metrics needing to be analyzed. This critical information may lead to new opportunities or prevent potential losses, so efficiencies in surfacing anomalies are paramount to success. Keeping analytical pace with the surge in data volume and speed requires either impossible levels of human resources or faster analysis techniques. Although business intelligence (BI) platforms provide endless ways to view data in ever-refined, dynamic charts and graphs, anomalies and outliers in the data may not be apparent in the visualizations.

Spot the unexpected and capitalize on the opportunity

By providing scalable automation of anomaly detection, natural language generation (NLG) resolves these challenges. NLG can handle massive amounts of data, drilling down and teasing out the significant details from deep inside the database. This means details a human might not find even after days of digging can be found in milliseconds by the NLG system. Working at machine speed, NLG can analyze vast amounts of data and generate narratives explaining where and why anomalies occur.

As BI strategies move toward self-service, NLG offers a scalable solution that empowers all analytics stakeholders with the information they need — in a language they understand — as soon as the data is available. Arria’s Anomalies app provides on-demand real-time analytics on all data within BI platforms: no painstaking, manual drill-down analysis; no waiting for updates to insight reporting. As soon as decision-makers request the information, the answers are at their fingertips.

Remove distractions and get to the points that matter

Experienced analysts may know what to expect from their data and can predict future trends: they know when seasonal or regional peaks and troughs may occur. Inexperienced analysts or business users may not know when those peaks and troughs are the norm, or whether something else is happening within the data.

The Anomalies App assists with this task by detecting patterns within the dataset and calculating the values considered normal, including those expected ups and downs. Values that deviate from these norms, either at a single time point or across a time range, are identified based on three categories:

  • Global (point) outliers: when a single data point is significantly different to all other points in the dataset.
  • Contextual anomalies: when a single or range of data points may not be outside the normal global range, but they are abnormal within a specific context.
  • Collective anomalies: when a collection of values deviates from the norms of the whole dataset.

Adjusting the sensitivity of the narrative allows you to specify what deviation from the norm is significant in contextual and collective anomalies. You get more control over the number of anomalies flagged and reduce the risk of excessive data or potentially false alerts, ensuring efficient use of time and resources. Using the app, you can leverage the dynamic properties of the BI dashboard to explore the data across different time ranges and variables and get an instant explanation by generating the supporting narrative.

Scaling complex analytics with ease

Providing explanations of anomalies in plain, easy-to-understand natural language removes ambiguity or bias, ensures consistency and clarity in reporting, and means that everyone gets the same story and understanding from the data.

By automating the anomaly detection process, organizations can optimize the efforts of high-value analysts and data scientists, enabling them to utilize their experience and knowledge for decision-making instead of time-consuming manual work. The recent Forrester Total Economic Impact™ study commissioned by Arria stated an 80% reduction in manual reporting with Arria’s NLG solutions.

Arria Apps accelerate the complex calculations and processes of statistical analysis without the risk of inaccuracies or bias. They provide a scalable solution to the growing BI challenges and empower analysts and decision-makers across the organization with faster, effortless access to the information they need to develop a truly data-driven culture.

The new Anomalies app is available in Arria’s integrations with Power BI, MicroStrategy, Qlik Sense and Tableau; start your free trial today.

Register to keep up-to-date with our latest releases and news.