ML Classifier

The ML Classifier assigns priority P1–P4 to each event using a RoBERTa model with a three-tier cache (PostgreSQL → in-memory dict → Redis) and a rule-based fallback.

---

Priority System

Priority	Severity	Color	SLA	Meaning
P1	CRITICAL	🔴 Red	Immediate	Production down, data loss, security breach
P2	ERROR	🟠 Orange	1 hour	Service degraded, failed jobs, high error rate
P3	WARNING/INFO	🟡 Yellow	4 hours	Performance degradation, non-critical issues
P4	DEBUG	🟢 Green	Next day	Debug traces, verbose logs

---

Classification Flow

flowchart TD
  EV[Enriched Event] --> TID{template_id?}
  TID -->|No| FALLBACK[Rule Fallback]
  TID -->|Yes| CACHE1[PostgreSQL<br/>template_priority]
  CACHE1 -->|Hit| USE_CACHED[Use Cached Priority]
  CACHE1 -->|Miss| CACHE2[In-Memory Dict]
  CACHE2 -->|Hit| USE_CACHED
  CACHE2 -->|Miss| CACHE3[Redis]
  CACHE3 -->|Hit| USE_CACHED
  CACHE3 -->|Miss| ROBERTA[RoBERTa Inference]
  ROBERTA -->|Confidence ≥ 0.8| STORE[Store in All Caches]
  ROBERTA -->|Confidence < 0.8| FALLBACK
  FALLBACK --> RULE[Severity→Priority Rule]
  USE_CACHED --> OUTPUT[Event with Priority]
  STORE --> OUTPUT
  RULE --> OUTPUT

---

RoBERTa Model

Property	Value
Architecture	RobertaForSequenceClassification
Base	roberta-base (125M params)
Classes	4 (P1, P2, P3, P4)
Max Length	512 tokens
Input	template_text (Drain3 template)
Output	Priority + Confidence (0–1)
Model Path	/app/model/log_priority_roberta
Version	roberta_v1.1 (folder name)

Service: Separate FastAPI service on port 8001 (ml-service) - GET /health — model loaded status - POST /predict — {text: str} → {priority, confidence}

---

Inference Pipeline

# ml_classifier.py
class MLClassifier:
    def __init__(self):
        self.model = None
        self.tokenizer = None
        self._model_path = "/app/model/log_priority_roberta"

    def load(self):
        self.tokenizer = AutoTokenizer.from_pretrained(self._model_path)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            self._model_path, device_map="cpu"
        )

    def predict(self, template_text: str) -> dict | None:
        if not self.is_loaded:
            return None

        inputs = self.tokenizer(
            template_text, truncation=True, max_length=512,
            return_tensors="pt"
        )
        with torch.no_grad():
            logits = self.model(**inputs).logits
            probs = torch.softmax(logits, dim=-1)
            conf, pred = torch.max(probs, dim=-1)

        priority = ["P1", "P2", "P3", "P4"][pred.item()]
        confidence = conf.item()

        if confidence >= 0.80:
            return {"priority": priority, "confidence": confidence}
        return None  # Trigger fallback

---

Rule Fallback

When model unavailable or confidence < 0.80:

SEVERITY_TO_PRIORITY = {
    "CRITICAL": {"priority": "P1", "confidence": 0.95},
    "ERROR":    {"priority": "P2", "confidence": 0.85},
    "WARNING":  {"priority": "P3", "confidence": 0.75},
    "INFO":     {"priority": "P3", "confidence": 0.70},
    "DEBUG":    {"priority": "P4", "confidence": 0.90},
}

Deterministic, no ML dependency.

---

Cache Tiers

Tier	Store	Lookup	Write	TTL
L1	PostgreSQL template_priority	Startup load_from_postgres()	set() → INSERT	Persistent
L2	In-process dict	get(template_id)	set() → dict	Process lifetime
L3	Redis ml:priority:{template_id}	get()	setex()	24h

Cache Key: ml:priority:{template_id}

{
  "priority": "P2",
  "confidence": 0.87,
  "model_version": "roberta_v1.1"
}

---

Event Output Fields

Field	Source	Example
priority	ML / Cache / Rule	P2
priority_confidence	Model confidence	0.87
priority_source	ml | cache | rule	ml
model_version	Model folder name	roberta_v1.1

---

Configuration

Env Var	Default	Description
ML_MIN_CONFIDENCE	0.80	Min confidence to cache/use ML result
ML_MODEL_PATH	/app/model/log_priority_roberta	Model weights directory
ML_SERVICE_URL	http://ml-service:8001	For remote inference (unused, local)

---

Metrics (Logged per Batch)

Counter	Meaning
ml_cache_hits_total	Served from any cache tier
ml_predictions_total	RoBERTa inference executed
ml_fallback_total	Rule fallback used

Target: Cache hit rate > 95%

# Check via API
curl -s http://localhost:8000/api/ai-insights/stats | jq .ml_status
# "healthy" if hit_rate >= 95% and low_conf == 0

---

Training (Offline)

# 1. Label templates via LLM
python devtools/label_with_llm.py  # Uses NVIDIA API, writes labels

# 2. Fine-tune RoBERTa
jupyter devtools/IaOps.ipynb  # Notebook: tokenize → train → export

# 3. Export model
trainer.save_model("/app/model/log_priority_roberta")

# 4. Deploy: rebuild ml-service + pipeline images
docker compose build ml-service consumer-pipeline
docker compose up -d ml-service consumer-pipeline

---

Troubleshooting

Symptom	Check
All priority_source=rule	Model not loaded (ml-service logs), confidence threshold too high
Low cache hit rate	New templates (normal), cache TTL too short, Redis unavailable
High latency	ml-service not running, model loading, CPU contention
Wrong priorities	Retrain with better labels, check label_with_llm.py thresholds

# Check ML service
curl http://localhost:8001/health
# {"status":"healthy","model_loaded":true,"model_version":"roberta_v1.1"}

# Test inference
curl -X POST http://localhost:8001/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "Database connection pool exhausted"}'