Log Forwarder

Motadata's Log Forwarder feature simplifies log management by efficiently collecting and forwarding logs from various sources to a centralized location. This ensures real-time analysis, enhanced security, and improved operational efficiency. With our tool, you can easily monitor and manage logs, helping you to quickly identify and resolve issues, and maintain a secure and compliant IT environment.

Try Now

Motadata AIOps’s Log Forwarder feature provides the capability to forward ingested logs to external, third-party software and systems. This is crucial for organizations that need to integrate their log data with other security, analysis, or compliance platforms. By forwarding logs, users gain the flexibility to perform a wide range of operations on their collected log data, such as advanced analysis, long-term storage, and correlation with other data sources, using their preferred tools.

Motadata AIOps supports industry-standard Syslog protocols for log forwarding, specifically:

  • Syslog-TCP: For reliable, connection-oriented log delivery.
  • Syslog-UDP: For higher-performance, but potentially less reliable, log delivery.

The Log Forwarder can transmit logs in the following formats:

  • JSON: A structured, human-readable format that is easily parsed by many systems.
  • Raw logs: The original, unstructured log data as it was received, providing maximum data fidelity.

Furthermore, Motadata AIOps allows you to apply source filters to selectively forward specific logs, ensuring that only relevant data is sent to the external system. This filtering capability optimizes bandwidth usage and reduces the processing load on the receiving system.

The Log Forwarder settings can be accessed within the Motadata AIOps interface via the following path:

  • Main Menu -> Settings -> Log Settings -> Log Forwarder

This navigation path will take you to the Log Forwarder configuration screen.

Log Forwarder Screen

The Log Forwarder screen presents a comprehensive overview of configured log forwarders and provides access to their settings. The following fields are displayed:

  • Forwarder Name: A user-defined, descriptive name for the log forwarder configuration. This name should be meaningful and easily identify the purpose or destination of the forwarded logs (e.g., “Splunk Forwarder”, “Security SIEM”).
  • Description: A user-provided description that provides additional context for the log forwarder. This description can include details such as the purpose of the forwarder, the destination system to which logs are being sent, and any specific configuration details.
  • Forwarder Type: Indicates the Syslog protocol being used for log forwarding. This field will display either “Syslog-UDP” or “Syslog-TCP”.
  • Forward As: Specifies the format in which the logs are being forwarded. This field will show either “JSON” or “Raw logs”.
  • Forwarder Status: Displays the current operational status of the log forwarder. This may include statuses such as “Running”, “Stopped”, “Error”, or “Pending”.
  • Action: Provides a set of actions that can be performed on the log forwarder configuration. Common actions include:
  • Edit: Modify the configuration settings of the log forwarder.
  • Delete: Remove the log forwarder configuration.
  • Start/Stop: Control the log forwarding process.

Creating a Log Forwarder

To create a new log forwarder, follow these general steps:

  • On the Log Forwarder screen, click on the “Create Log Forwarder” option. This will initiate the log forwarder creation process and display the configuration form.
  • Important Note: The specific parameters that need to be configured during the log forwarder creation process vary depending on the selected Forwarder Type (Syslog-UDP or Syslog-TCP). Therefore, the creation process is detailed separately for each type in the following sections.

Syslog – UDP

When configuring a log forwarder to use the Syslog-UDP protocol, the following parameters must be specified:

Parameters:

  • Forwarder Name: Enter a descriptive name for this log forwarder configuration.
  • Description: Provide a detailed description of this log forwarder.
  • Forwarder Type: Select “Syslog-UDP” from the available options in the dropdown menu.
  • Destination IP: Enter the IP address of the destination server or system to which the logs will be forwarded. This is the IP address of the system that will receive the Syslog messages.
  • Destination Port: Enter the port number on the destination server that is listening for Syslog-UDP traffic. The standard Syslog-UDP port is 514, but a different port may be used depending on the receiving system’s configuration.
  • Source Filter: Select an option to filter the logs based on their source. This allows you to specify which logs should be forwarded.
  • Source: Depending on the option selected in the “Source Filter” field, select the specific source of the logs from the dropdown menu. This could be a specific IP address, a host type, or a predefined group of devices.
  • Filter: Optionally, configure a filter condition to further refine the logs that are forwarded. This allows you to filter logs based on their content or other characteristics. The “Configuring Prefilters” section provides detailed information on how to configure these filters.
  • Forward Log as: Choose the format in which the logs will be forwarded. Select either “JSON” or “Raw logs” from the dropdown menu.

Testing:

  • After configuring all the necessary options, click on the “Test” button.
  • Motadata AIOps will then perform a basic ping check to verify network connectivity with the specified Destination IP. This ensures that the Motadata AIOps server can reach the destination server.
  • If the ping test is successful, the “Create Log Forwarder” button will become available, allowing you to save the configuration. If the ping test fails, you will need to verify the Destination IP and network connectivity.

Syslog – TCP

When configuring a log forwarder to use the Syslog-TCP protocol, the following parameters are required:

Parameters:

  • Forwarder Name: Enter a descriptive name for this log forwarder configuration.
  • Description: Provide a detailed description of this log forwarder.
  • Forwarder Type: Select “Syslog-TCP” from the available options in the dropdown menu.
  • Destination IP: Enter the IP address of the destination server or system.
  • Destination Port: Enter the TCP port number on the destination server that is listening for Syslog-TCP connections. The standard Syslog-TCP port is 514, but a different port may be configured.
  • Source Filter: Select an option to filter the logs based on their source.
  • Source: Depending on the “Source Filter” selection, choose the specific log source (IP, host type, group) from the dropdown.
  • Filter: Optionally, configure a filter to further refine the logs that are forwarded. See the “Configuring Prefilters” section for details.
  • Forward Log as: Select the desired log format: “JSON” or “Raw logs”.

Testing:

  • Once you have configured all the required options, click on the “Test” button.
  • Motadata AIOps will perform a test to verify the Port and IP configuration settings. This test goes beyond a simple ping and attempts to establish a TCP connection to the specified Destination IP and Port. This ensures that the destination server is listening on the specified port and that a TCP connection can be established.
  • If the test is successful, the “Create Forwarder” button will appear, enabling you to save the configuration. If the test fails, you will need to check the Destination IP, Port, and any firewall rules that may be blocking the connection.

Configuring Prefilters

Prefilters provide a powerful mechanism for filtering logs before they are forwarded to the external system. This ensures that only relevant logs are transmitted, reducing network bandwidth consumption, and minimizing the processing load on the receiving system.

Purpose: To filter logs before forwarding, ensuring that only logs that meet specific criteria are sent to the third-party system. This allows you to tailor the log stream to the exact requirements of the receiving system and your analysis needs.

Structure: Prefilters in Motadata AIOps are structured using a combination of groups and criteria:

  • Prefilters allow you to define up to 3 groups.
  • Each group can contain up to 3 individual criteria.
  • This group-and-criteria structure provides flexibility in defining both simple and complex filtering rules.

Fields: The following fields are used to configure prefilters:

  • Group(s) Matching: This option defines the logical operation between the defined groups.
  • ALL: When selected, a log must match the filtering criteria defined in all groups to be forwarded (or excluded, depending on the “Group Matching” setting). This implements an AND operation between groups.
  • ANY: When selected, a log only needs to match the filtering criteria defined in at least one group to be forwarded (or excluded). This implements an OR operation between groups.

Group Matching: This option determines whether logs that match the criteria within a single group should be included or excluded from the forwarding process.

  • Include: Logs that meet the defined criteria within the group will be forwarded.
  • Exclude: Logs that meet the defined criteria within the group will not be forwarded.

Criteria: This section defines the logical operation between multiple criteria within the same group.

  • ALL: A log must meet all of the defined criteria within the group to be considered a match. This implements an AND operation within a group.
  • ANY: A log only needs to meet one of the defined criteria within the group to be considered a match. This implements an OR operation within a group.

Counter: Select the specific metric counter that you want to use as the basis for the filter condition. The available counters in the dropdown menu may vary depending on the selected Forwarder Type (Syslog-UDP or Syslog-TCP), as different log sources provide different data. Examples of counters include timestamp, severity, hostname, message content, etc.

Select Operator: Choose the operator that will be used to compare the value of the selected counter with the specified value. Common operators include:

  • Equals (=)
  • Not equals (!=)
  • Greater than (>)
  • Less than (<)
  • Greater than or equal to (>=)
  • Less than or equal to (<=)
  • Contains
  • Does not contain

Value: Enter the value that you want to compare against the selected counter using the chosen operator. The data type of this value should match the data type of the counter (e.g., a number for a numeric counter, a string for a text counter).

Adding Groups: To add a new group to the prefilter configuration, click on the “Add New Group” option. This allows you to create more complex filtering rules with multiple branches.

Previewing Logs: Before saving the log forwarder configuration, it is highly recommended to preview the logs that will be forwarded based on the configured prefilters.

  • Click on the “Preview” option.
  • Motadata AIOps will display a sample of the logs that match the current prefilter configuration.
  • The preview will automatically adjust to reflect the log format selected in the “Forward Log as” field (JSON or Raw logs), allowing you to see exactly what will be sent to the external system.
  • By default, the sample log preview will cover a period of 30 minutes. This duration provides a representative sample of recent log activity.

Resetting Parameters: To clear all the configured parameters in the prefilter section and start the configuration from scratch, click on the “Reset” option. This will clear all group and criteria settings.

×