Log Filtering: How to Cut Log Ingest Volume Without Losing Evidence
Every log estate reaches a point where volume grows faster than the value inside it. The usual response is to find the biggest source and drop it.
Cutting volume is the easy part. Cutting the right half takes judgment. Log filtering is only one of four options for an expensive source, and the other three matter just as much.
In this blog, you will:
Compare the four options for a source that produces too much.
Rank your sources so the first rule you write is the one that pays.
Protect the categories that must never be dropped.
Test a filter before it deletes anything.
By the end you can defend every rule you write, including the ones that keep data.
Why Cutting Log Volume Goes Wrong in Both Directions
A volume cut fails in two directions. Both stay hidden until somebody needs the data.
Cut too much and the evidence is gone. The rule looked narrow when somebody wrote it. You find out during an investigation, when a search returns nothing. Nobody can then say whether the events never happened or never landed.
Cut too little and the bill does not move. Most first rules target the source that irritates somebody on a dashboard. Irritation has no relationship to volume. The estate feels tidier and costs the same.
The second failure costs little to correct. The first one is permanent, because a dropped event never comes back.
So we route or sample before dropping. We drop only what has earned it.
Filter, Retain, or Forward? Four Options for a High-Volume Log Source
Every expensive source has four possible fates. Only one of them deletes anything. Working out which column a source belongs in is the whole decision.
The table below sets the four side by side. Each row shows what that column keeps.
Option | What Happens to the Event | What You Keep | Use It When |
Drop | It never reaches the index | The derived metric, and nothing else | No query history, no rule reads it, no audit claim on it |
Sample | A defined share is kept | Rates and trends, not single events | The source repeats heavily and the value is in aggregate |
Route | It lands in slower, cheaper storage | Everything, at slower retrieval | You might need it, rarely, and you can wait for it |
Keep | It stays in the searchable tier | Everything, immediately | It feeds an alert, an investigation, or an audit answer |
1. Drop It Before the Index
Dropping removes the cost completely, and you cannot undo it afterward. Three things have to be true before you write the rule: no query history on the source, no rule reading it, and no audit claim on it.
Check all three before writing the rule. A source failing any one of them belongs in a different column.
2. Sample It and Keep the Shape
Sampling keeps a defined share of a repetitive source. Rates and trends survive while the volume falls. Access logs, health checks, and successful transactions all fit.
What sampling cannot do is answer a question about one specific event. Log sampling covers the techniques and the categories that should never be sampled at all.
3. Route It to Cheaper Storage
Routing moves the cost instead of removing it. The events land in a slower tier and retrieval takes longer. Everything is still there when somebody asks.
Teams forget this column most often. It usually suits data you might need once a year. Where those tiers sit and how long each one holds data belongs to log retention best practices.
4. Keep It in Full
Some sources earn their place at full price. Anything feeding a detection rule stays whole and searchable. So does anything an audit answer depends on.
The other three columns exist to fund this one. Cutting what nobody reads pays for keeping what matters.
Which Log Sources Should You Cut First?
Most first rules target the wrong source. The ranking therefore matters more than the rule itself. Four passes will tell you where to start.
Rank sources by volume, not by irritation: Pull the top ten by events per day. The list usually comes back shorter and more boring than anyone expects.
Cross the top ten against query hits: Most platforms can show which sources anyone has actually searched. A high-volume source with no query history makes your first candidate.
Check what has ever fired an alert: A source no rule reads is easier to cut than one feeding detection. Volume alone will not tell you which is which.
Look for repetition rather than rarity: A source repeating one line thousands of times a day suits sampling. A source producing rare and varied events usually does not.
We have yet to see a ranking where the top three sources were the ones people complained about. A source that fires constantly is an alerting problem, not a volume one. Alert noise reduction handles that case.
What Should Never Be Dropped, No Matter the Volume?
Some categories stay whole regardless of what they cost. The four below belong on that list in most estates:
Authentication and authorization events
Privilege and configuration changes
Records of who accessed or changed what
Anything a detection rule currently reads
Knowing the list is the easy half. Protection has to be structural. A rule written next quarter will not remember this conversation.
Write the keep rules first: Put an allow list in front of the drop rules. A broad condition then cannot reach a protected category by accident.
Scope every drop rule to a source: A global rule below WARN will eventually take something somebody needed. Naming the source keeps the blast radius visible in the rule itself.
Name an approver for exceptions: Somebody signs off when a protected category gets an exemption. That name belongs in the rule itself.
We keep the allow list in version control alongside the rules, so a change to either one shows up in review.
Where Should Log Filtering Run in the Pipeline?
Placement decides what a cut actually costs you. The ordering inside the pipeline matters as much as the rule.
Why Exclusion Sits After Policies and Metrics
In an observability pipeline, exclusion runs at stage 06. Detection policies sit at 04 and metric generation at 05. Every event gets evaluated and measured before anything discards it.
Front-door filtering inverts that order. You drop the event before the system has learned anything from it. You save real money and lose real visibility at the same time.
What Survives After the Event Is Dropped
The ordering explains why a drop is less final than it sounds. The derived metric survives, and so do the alert history and the counted rate.
So you can discard a million health checks and still know how many arrived. You give up the ability to open one of them. Log indexing is where that trade gets configured.
How Do You Test a Filter Before It Deletes Anything?
A drop rule is a deletion that runs continuously. It deserves the same care as any other irreversible change.
Count What the Rule Would Drop Before You Enable It
Run the condition as a search first. Count the matches across a full week. A narrow-looking rule can match far more than expected, and a weekend batch job is the usual reason.
Read a sample of what it caught, not only the total.
Keep a Register of What You Chose Not to Store
Write down every rule, the source it targets, and why the data was judged expendable. Responders and auditors ask the same question. They want to know whether an absence is a gap or a decision.
A register answers that in seconds. Without one, the answer takes a day and still sounds like a guess.
Re-Run the Rule When the Format Changes
Patterns break when a release changes a message. The rule then matches everything or nothing, and neither failure raises an alert.
Re-run the count after any source changes format. We have watched a rule quietly stop matching for a quarter. The bill was the only thing that noticed.
How to Start Cutting Log Volume Safely
Start with one source and one rule. The process is the thing you are testing first.
Pick one source from the top of the volume list: Start at the top, where a change shows up fastest.
Choose its column before you write the rule: Drop, sample, route, or keep. Deciding the option first stops the rule from deciding for you.
Dry-run it and read what it would have taken: Count across a week, then read a sample of the matches.
Enable it with the allow list in front: Protected categories then stay out of reach however the rule is written.
Record the decision and set the re-check: Tie the review to a source change or a volume threshold, never to a date.
Where the rules live decides whether any of this survives the next hundred sources. In ObserveOps Infinity, named exclusion rules carry their own source filter.
Each rule states what it applies to. Those events are dropped before they reach indexing and retention, and you configure the rule once.
One limit is worth being honest about. No platform can tell you which source you will need next year.
It can only make the rules visible and cheap to change, which is what most estates lack. We would rather see a team spend a week ranking sources than an afternoon writing rules, on whichever unified observability platform it runs.
Make Every Expensive Source an Explicit Choice
Log filtering stops being risky once every high-volume source has a column against its name. Drop, sample, route, or keep. Decided once and written down, that choice turns a recurring argument into a rule anybody can read.
The timing is the uncomfortable part. You choose before you know what the next investigation will need. Some of those choices will be wrong, and only the drops are permanent.
So keep the register current, and revisit it whenever volumes shift or a new source class arrives. The whole exercise starts at log ingestion, because a source you never collect is the one decision that costs nothing to reverse.
FAQs
Can you recover logs that a filter has dropped?
Not once the rule has run. A dropped event never reaches storage, so there is nothing to restore. Removing or widening the rule only affects events arriving afterward, which is why a drop rule earns a dry run first.
How much log volume can you safely cut?
There is no safe percentage. It depends on how many of your sources carry query history and alert coverage, which is why the ranking comes before the rule. Teams that measure first usually cut more than they expected.
Does filtering logs break your alerts?
Not when exclusion runs after detection. In a pipeline that evaluates policies before discarding events, the alert and the derived metric both survive the drop. Front-door filtering does break alerts, because nothing has read the event yet.
What should you do with logs you rarely read but might need?
Route them to slower, cheaper storage instead of dropping them. You keep the full record and pay less to hold it. The trade you accept is a longer wait when somebody finally asks for it.
Who decides which log sources get filtered?
It works best as a shared decision. The platform team brings the volume ranking, security confirms what has to stay whole, and the application owner explains what the source is actually for.
Author
Ramya Shah
Technical Writer
Ramya Shah is a technical content writer with a computer engineering background and roots in automotive journalism. He covers IT Service Management, observability, IT operations, and AI-driven automation. An early adopter of AI-assisted writing workflows, he turns complex IT processes into clear, engaging content optimized for search and answer engines (AEO), lifting content output and organic visibility.
