Not all analytics are created equal

Not all analytics are created equal

Aligning the right metrics to the right use case allows for timelier reporting and reduces application risk.

Credit: Dreamstime

There are many examples of Class B metrics, and there are many companies focused on generating these metrics, such as AppDynamics, Datadog, Dynatrace, and New Relic. Class B metrics can also include logging and other metrics from companies such as Elastic and Splunk.

Class C analytics

Class C analytics involve metrics that are used for offline application analysis and longer term planning purposes. Class C analytics are often used to determine the strategy and product direction of an application.

These metrics may be examined in real time, as Class A and Class B metrics are, or they may be issued and examined periodically, such as weekly, monthly, or quarterly.

Class C metrics are used for business analysis, such as analysing customer traffic patterns, time on site, referring sites, and bounce rates. They can be used for sales reports and sales funnels. They can be used for financial reports and auditing purposes.

Some shops test new application features or new wording for their websites by showing two or more different versions of the feature to customers, and analysing metrics to see which one performs better. This is called A/B testing, and the metrics used are Class C metrics.

There are many companies that provide Class C metrics, but by far the most well-known Class C metrics provider is Google Analytics.

Why is classifying analytics important?

Different metrics have different consumers. The consumer who cares about the metrics is specific to the category the metrics belong to:

  • Class A metrics are mostly consumed by automated systems and are used internally by systems and processes. They are used to dynamically and automatically update critical operational resources in order to keep a system healthy and scaled appropriately.
  • Class B metrics are mostly consumed by operations and support teams, along with development teams, as part of the incident response process. They can provide immediate assistance to teams in identifying and fixing problems, and generally help in preventing problems before they occur.
  • Class C metrics are mostly consumed by business planners, product managers, and corporate executives. They are used to drive longer term business decisions, business modelling, product design, and feature prioritisation.

Additionally, and perhaps most importantly, systems that collect and process analytics have different priorities within your application. Problems collecting Class A metrics are mission-critical problems. A failure of a Class A metric could result in automated infrastructure tools doing the wrong thing and ultimately result in brownouts or blackouts.

By contrast, problems collecting Class C metrics are not necessarily cause for alarm, and addressing a Class C issue could be postponed for hours, days, or even longer.

Be very careful when deciding how to use a metric; mistakes in using metrics for the wrong purposes can be disastrous. For example, don’t use a Class B metric, such as “application latency,” to dynamically and automatically allocate system resources, such as autoscaling up and down your server fleet. Why? Because using Class B metrics in mission-critical use cases such as this introduces unnecessary risk into your application.

Let’s say you are receiving metrics from an application performance monitoring company, which are typically classified as Class B metrics. Using their reported “application latency” to determine fleet scaling would leave you open to potential problems.

If your application performance monitoring company has an outage, you would not be able to correctly scale your fleet, and it could cause you to have an outage. This means that your application performance monitoring company is now a mission-critical component of your application, where before it may have just been a useful and valuable tool for diagnosing problems.

As another example, don’t rely on a Class C metric, such as “shopping cart abandon rate,” as the primary way of identifying an operations availability problem in your cart service. The metric is too far away from the problem, and would not give you the timely indication of a problem in need of resolution. Your report that “sales are down this week due to an increase in cart abandons” is too little and too late to assist you in debugging earlier cart service problems.

Using the right metric for the right purpose will increase the usefulness of your analytics, allow timely reporting, and reduce risk to your application and business.


Show Comments