Sometimes due to licensing constrains and large volumes of events you may need to resort to indexing only the relevant entries from a logfile. For example cases when the DEBUG mode has been enabled and it cannot be turned off or you'd like to discard a repeated log entry from indexing.
Filtering can be achieved through an app on the indexer. Filtering works based on the sourcetype defined in inputs.conf on the universal forwarder.
The props.conf/transforms.conf combo provided below will only retain log lines containing strings Error or Warning while all other messages will be sent to nullqueue (deleted).
The two transforms used here: setnull and setparsing act together. First setnull matches all events and tags them to be sent to the nullQueue. Setparsing transform then follows and tags events that match Error or Warning to be sent to theindexQueue. The result is that the eventsthat contain strings Error or Warning get passed on, while all other events get dropped.
The props and transforms need to be placed on the indexer. If you are using a heavy forwarder to pick the logfiles or a heavy forwarder is used in the chain, the props and transforms config files need to be there instead of on the indexer.
props.conf
[your_sourcetype]
TRANSFORMS-set = setnull, setparsing
transforms.conf
# This sends all events to be ignored
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
# this says ignore all events, except the ones containing Error or Warning
[setparsing]
REGEX = Error|Warning
DEST_KEY = queue
FORMAT = indexQueue
[your_sourcetype1]
TRANSFORMS-set = setnull, setparsing
[your_souretype2]
TRANSFORMS-set = setnull, setparsing