Does is it Make Sense to Monitor Metrics and Logs Both for Making an Informed Decision?

Quite often, we are in dilemma that using either network monitoring or log monitoring is enough. If I have a log management solution do I need to look into metrics?

Even worse, if I get all the updates from my customers if something goes down, do I really need such solutions? Well, the point is to have answers for the following questions.

• What is the right approach?
• Is relying on either metric or log enough?
• If No – then when to use log and when to use metric analytics? Or do I need both?
• How do I monitor operations & performance in the best possible way?

In this blog, we will try to clear most of the problems discussed above & shall explore more on platforms to be considered based on source of data, cost efficiency, performance and key use cases.

Understanding Log & Metric Data Sources

Typically, vendors build monitoring solutions around either log or metrics data. The below table helps to understand the difference:

You would find a separate tool for monitoring each of these sources. A server monitoring tool wouldn’t have information about server log files. Similarly, a log management tool will not track the performance of your application or servers etc.

However, many tools available today, offer integration to bridge the gap between the two, but leave some blind spots behind. Let us have a deeper look at their differences & significance of monitoring the two together.

At a very basic level, the difference between a metric and a log lies in their data structure. Metrics are often tagged as simply a number or measure of some quantity.

They contain time stamp and other extra categorical information which is stored in a structured data format for eg. JSON for Amazon CloudWatch etc.

In case of logs, they are test files in unstructured or semi-structured format. Logs contain critical information which may be useful in the hour of the need. Most familiar use cases could be security forensics analysis or for cyber security threat prevention.

Stepping out of Traditional Monitoring Paradigm

Modern day monitoring solutions offer logs and metrics on the same platform, as they tend to offer exceptional functionality that impacts your understanding of your IT infrastructure.

Log management solutions can also parse each of the log field and convert them into metrics. Also, some log management tools are able to give out summary metrics.

But they miss out on critical information that might come into use later. This method averages out log data information. Similarly many metrics based solution track error logs in the form of events, which also gives the number of occurrences of a particular event. But there is so much more to it.

When to Use Metrics and When to go for Logs

Metrics are best suited for profiling, monitoring & alerting. The efficiency of summarizing data makes them great for monitoring and performance profiling because you can economically store data for longer duration.

On the other hand, logs give you detailed information required for debugging, troubleshooting and auditing. Having a unified solution could be great for alerting because they are faster and efficient. Single platform roughly offloads 90% of the monitoring stress.

Metrics + Logs for Continuous Delivery

To operate a highly reliable & prompt service for your end-customers, you can’t afford to leave out any of the use cases. In case if you monitor just the metrics you might miss out of important details desirable for debugging tough repetitive problems.

You might end up falling in a situation which is eating up all your precious time for no reason at all. Likewise, if you’ve implemented log management but you’re missing out on alerts and performance overview, then you may end up in a trap. A trap full of confusion & downtime.