Log Correlation: A Trend or a Need

There has been a growing importance of log data collection and management in an IT environment. Whether it’s for security, internal control or for even for compliance purposes, log collection and log correlation have become vital for companies. In an IT environment which is made up of a lot of different components like hardware and software, the logs can run into thousands in a day or sometimes in an hour.

The problem is that all these logs are not useful or relevant and as such to separate the useful data from not so useful one, we need to have intelligence in log management software which can just not collect the logs but process it to make them useful. This can be achieved by integrating, correlating or analysing the log data. The logs are collected of an event and as such sometimes the log correlation is also known as event log correlation. As stated, event correlation or log event correlation can serve many purposes. In layman terms, event correlation can help to discover and apply logical associations among disparate individual raw log events in order to:

1. Make informed security decisions
2. Identify and respond to security threats
3. Validate effectiveness of security controls
4. Measure and report on compliance with PCI, HIPAA, SOX, and other standards
5. Detect policy violations and in many more use cases.

Now that we have established that log correlation is indeed helpful and can assist in a number of scenarios, let’s look further as to why is it important? Since logs are generated from different sources, their output format also differs. While some of them can be directly read by humans, other are in machine readable format or some code language. Also, some logs can be in PDF, excel or other office formats while some others may be in completely different format.

In absence of a centralized log collection tool and the ability to see them all side by side, correlation would not be possible. Most logs are helpful in ascertaining security threats but they are useless in isolation without any connection established with other logs. For example, there is a log which says “Unauthorized Access detected” and shows the time when this event occurred. Unless it is established if this was internal or external breach and what led to such a breach, the extent of damage cannot be ascertained. For such instances, we need the event correlation and need to see logs from different sources. This can be done by human intervention or a logical analysis through event correlation rules. Event correlation helps in detection of the incident and follow the trail so that security analysts and incident responders can know the nature and extent of the damage occurred through such a breach and get to the right direction of investigation to be prepared in responding to such situation.

The use of event correlation rules helps in correlation of logs and is the most important ingredient to convert raw data into actionable insights and guide the IT administrators to make better decisions. The connecting of dots from different events and logs is what makes log correlation a powerful tool in aiding the IT personnel.

In a nutshell, log correlation is not a trend but has real useful value in a lot of different scenarios. And even though security issues like intrusion detection and network forensics might seem like the only advantages of log correlation, the use cases of log correlation can be extended to a lot of different scenarios such as recurrence of event, context of subsequent events etc.

Most companies today operate in several countries and as such they are bound by a lot of laws and regulations. These mandate them to follow regulatory compliance of all these countries in terms of data retention and maintenance. Log correlation helps IT personnel to measure and be aware of their compliance levels with different standards. Also with most companies around the world, no matter what sector or industry they operate in, being dependent on IT infrastructure, there is a lot of confidential data stored on electronic medium. Copyright materials, trade secrets, other classified documents and a lot more information which is vital for business and whose leak or being in the wrong hands can result in a loss of millions of dollars, are all stored in electronic format. All of these are protected by IT security policies and a breach anywhere would not lead to actionable insights unless the logs can be correlated.

Also, once an incident has happened where the security controls are breached, the IT policies have to be tightened to avoid any such incidents from recurring. The IT policies have to reevaluated and strengthened which cannot be done in the absence of log correlation. Only log correlation will give you an idea of the sequence of events as they happened. Log correlation can help IT staff to respond to security threats and devise better IT policies.

Thus Log correlation is an important need for organisations rather than just a trend. It has tangible benefits and should be one of the important criteria in the selection of a log management software by any company.