Parser — Drain3

Drain3 is the log parser that clusters raw messages into templates using an online tree-based algorithm. It runs as the first processing stage in the pipeline.

---

What is Drain3?

Drain3 (Python port of IBM's Drain algorithm) parses unstructured logs into templates (patterns with variable placeholders <*>). It builds an online search tree for real-time clustering.

%%{init: {'theme':'default','themeVariables':{'primaryColor':'#1a73e8'}}}%%
flowchart TD
  MSG[Raw Message] --> TOKENS[Tokenize]
  TOKENS --> TREE[Search Tree by Length]
  TREE --> SIM[Similarity Check<br/>sim_th=0.4]
  SIM -->|Match| EXIST[Existing Cluster]
  SIM -->|No Match| NEW[New Cluster]
  EXIST --> TMPL[Template: "User <*> logged in from <*>"]
  NEW --> TMPL2[New Template]

---

Algorithm

1. Tokenize message by whitespace/special chars
2. Tree Level 1: Branch by token count (length)
3. Tree Level 2+: Branch by token at each position
4. Leaf: Compare new log to existing cluster templates using token similarity
5. Threshold (sim_th=0.4): If similarity ≥ threshold → assign to cluster; else → create new cluster

Key Parameters: | Parameter | Value | Effect | |-----------|-------|--------| | sim_th | 0.4 | Similarity threshold (lower = more aggressive clustering) | | depth | 4 | Tree depth (higher = more precise) | | max_clusters | 100,000 | Hard limit on template count | | max_children | 100 | Max branches per node |

---

Version & Config

# services/pipeline/parser.py
from drain3 import TemplateMiner
from drain3.persistence import AsyncFilePersistence

parser = LogParser(
    state_path="/app/state/drain3_state.json",
    max_clusters=100000,
)

# Internally uses:
TemplateMiner(
    persistence=AsyncFilePersistence("/app/state/drain3_state.json"),
    config=TemplateMinerConfig(
        sim_th=0.4,
        depth=4,
        max_clusters=100000,
        max_children=100,
    )
)

Version: drain3>=0.9.11 (PyPI)

---

Input / Output

Input	Output
message: str (raw log line)	template_id: str (e.g., "123") template_text: str (e.g., "User <> logged in from <>")

# Usage in pipeline
parse_result = parser.parse(raw_message)
event.template_id = parse_result.template_id
event_dict["_template_text"] = parse_result.template_text

---

Persistence

File: /app/state/drain3_state.json (mounted volume ./state:/app/state)

• AsyncFilePersistence: Writes every 30 seconds (non-blocking)
• Size: ~300 KB typical (thousands of templates)
• Survives restarts: Templates learned are retained

# View state
docker compose exec consumer-pipeline cat /app/state/drain3_state.json | head -50

---

Why Drain3?

Approach	Pros	Cons
Drain3	Online, real-time, persisted, low CPU	Fixed similarity threshold
Regex/Grok	Precise	Manual maintenance, brittle
LLM	Semantic understanding	High latency, cost, non-deterministic
Spell/Logmine	Batch, academic	Not streaming, no Python lib

Drain3 sweet spot: Real-time, zero-config, template-level ML (1000× fewer inferences).

---

Template Examples

Raw Message	Template
User 4923 logged in from 10.0.0.5	User <*> logged in from <*>
ERROR: Connection to db-primary failed (timeout=30s)	ERROR: Connection to <*> failed (timeout=<*>)
2026-01-15 10:30:00 INFO [main] Starting service on port 8080	<*> INFO [<*>] Starting service on port <*>
Memory usage: 85% (threshold 80%)	Memory usage: <*> (threshold <*>)

---

Integration with ML

Templates are the unit of classification:

1. Parser extracts template_id
2. Cache checked: template_cache.get(template_id)
3. If miss → RoBERTa classifies template_text
4. Result cached: template_cache.set(template_id, priority, confidence)

→ 1000× fewer ML inferences vs per-message classification.

---

Troubleshooting

Issue	Check
Too many templates (>50k)	Increase sim_th to 0.5, check for high-cardinality logs (request IDs)
State file growing	Normal; check max_clusters limit
Restart loses templates	Verify volume mount ./state:/app/state
High CPU	Reduce max_clusters, check log volume

# Template count
docker compose exec consumer-pipeline python -c "
from drain3 import TemplateMiner
from drain3.persistence import AsyncFilePersistence
miner = TemplateMiner(AsyncFilePersistence('/app/state/drain3_state.json'))
print(f'Clusters: {miner.cluster_count()}')
"