Skip to content

Drain3 Parser

The pipeline uses Drain3 for online, streaming log-template mining. Drain3 incrementally groups similar messages into clusters (templates) without batch retraining, making it ideal for a high-throughput event stream.

How It Works

Drain parses each message into a log template (message with variable parts replaced by placeholders) and a cluster id (template_id).

"Connection timeout connecting to 10.0.1.5"  ─┐
"Connection timeout connecting to 10.0.1.9"  ─┼→  template: "Connection timeout connecting to <*>"
"Connection timeout connecting to 10.0.1.12" ─┘   cluster_id: 482

Configuration

config = TemplateMinerConfig()
config.drain_sim_th = 0.4       # similarity threshold
config.drain_depth = 4          # parse tree depth
config.drain_max_clusters = 100000
config.profiling_enabled = False
config.snapshot_interval_minutes = 1
Setting Value Effect
drain_sim_th 0.4 Messages > 40% similar merge into one cluster
drain_depth 4 Limits tree depth (speed/memory trade-off)
drain_max_clusters 100000 Upper bound on template count

Pre-Normalization

Before Drain sees a message, the parser replaces noisy tokens with a wildcard to avoid template explosion:

_TS_PATTERN      = re.compile(r"\d{4}[-/]\d{2}[-/]\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?")
_HEX_HASH_PATTERN = re.compile(r"\b[a-fA-F0-9]{8,}\b")

# "ERROR 2026-08-11T09:12:44Z hash=9f2c7d1ab3e" 
#   → "ERROR <*> hash=<*>"

Timestamps and long hex hashes are replaced with <*>, preventing each unique value from becoming its own template.

Async Persistence

Drain3 state (clusters + template texts) is persisted so the model survives restarts:

Component Behavior
AsyncFilePersistence Debounced save every 30 s + flush on shutdown
Background thread Forces a save every 10 s
shutdown() Flushes state on graceful stop

This avoids blocking the hot path while guaranteeing at-most-30s state loss on crash.

Parse Result

@dataclass
class ParseResult:
    template_id: int | None   # Drain cluster id (None = unmatched/empty)
    template_text: str        # mined template or raw message
    parameters: list[str]     # captured variable values

Failure Handling

Case Behavior
Empty/blank message Return empty ParseResult (no cluster)
Drain error Log warning, use raw message as template, template_id = None
Cluster overflow Config bounds; old clusters may be evicted

A template_id = None still flows downstream — normalization and dedup continue, but no ML classification is attempted.

Throughput

Metric Value
Events/sec ~2,000
Latency/event ~1 ms
Cluster state save 30 s debounce