Feature Engineering

The Feature Engineering stage enriches events with computed numerical features for ML classification and analytics. Uses Polars for vectorized, high-performance transformations.

---

Features Produced

Feature	Type	Description	Used By
severity_rank	Int8	DEBUG=0, INFO=1, WARNING=2, ERROR=3, CRITICAL=4	ML, Analytics, Sorting
is_error_or_worse	Bool	severity_rank >= 3 (ERROR/CRITICAL)	ML, Filtering, KPIs
source_event_count	Int32	Count of events per source in batch	ML, Anomaly detection
template_event_count	Int32	Count of events per template_id in batch	ML, Template popularity

---

Implementation (Polars)

# feature_engineering.py
import polars as pl

def compute_batch_features(events: list[dict]) -> pl.DataFrame:
    df = pl.DataFrame(events)

    # severity → rank (vectorized replace_strict)
    severity_map = {
        "DEBUG": 0, "INFO": 1, "WARNING": 2, "ERROR": 3, "CRITICAL": 4
    }
    df = df.with_columns(
        pl.col("severity").replace_strict(severity_map, default=1)
        .cast(pl.Int8).alias("severity_rank")
    )

    # is_error_or_worse
    df = df.with_columns(
        (pl.col("severity_rank") >= 3).alias("is_error_or_worse")
    )

    # source counts
    source_counts = df.group_by("source").agg(
        pl.len().alias("source_event_count")
    )
    df = df.join(source_counts, on="source", how="left")

    # template counts
    template_counts = df.group_by("template_id").agg(
        pl.len().alias("template_event_count")
    )
    df = df.join(template_counts, on="template_id", how="left")

    return df

---

Why Polars?

Aspect	Polars	Pandas	Pure Python
Speed	~10× faster	Baseline	100× slower
Multithread	Yes (auto)	No (GIL)	No
Memory	Lazy, streaming	Eager	N/A
Type Safety	Strict schema	Flexible	None
Expressions	Parallel	Sequential	Manual loops

Pipeline impact: Feature engineering ~2,300 eps (not a bottleneck).

---

Feature Details

severity_rank (Int8)

# Mapping
DEBUG=0, INFO=1, WARNING=2, ERROR=3, CRITICAL=4

- Enables numeric comparison: severity_rank >= 3 → error+ - Used in ML as primary ordinal feature

is_error_or_worse (Bool)

severity_rank >= 3

- Binary flag for fast filtering - Used in KPIs (error_events count)

source_event_count (Int32)

COUNT(*) OVER (PARTITION BY source)

- Detects volume spikes per source - ML feature: sudden source activity = anomaly

template_event_count (Int32)

COUNT(*) OVER (PARTITION BY template_id)

- Template popularity in current batch - ML feature: rare templates = potential novel issues

---

Output Schema

Each event dict enriched with:

{
    ...original_fields,
    "severity_rank": 3,
    "is_error_or_worse": True,
    "source_event_count": 1250,
    "template_event_count": 42,
}

Types: Polars → Python dict conversion handles Int8/Int32/Bool automatically.

---

Performance

Metric	Value
Batch size	5,000 events
Processing time	~50 ms
Throughput	~2,300 eps
Memory (batch)	~15 MB

---

Extending Features

To add a new feature: 1. Add expression in compute_batch_features() 2. Update CanonicalEvent schema if persistent 3. Update ML training pipeline (if used by classifier)

# Example: time since template first seen
df = df.with_columns([
    (pl.col("timestamp") - pl.col("first_seen"))
    .dt.total_seconds()
    .alias("template_age_seconds")
])