Skip to content

Labeling Pipeline

The labeling pipeline turns operator corrections and model confidence data into ground-truth training data for RoBERTa. It closes the loop: classify → surface → human correct → retrain.

The Feedback Loop

flowchart LR
  A[New template] --> B[RoBERTa predicts priority]
  B --> C[Dashboard shows prediction + confidence]
  C -->|operator disagrees| D[PATCH priority]
  D --> E[template_priority updated<br/>model_version = manual]
  E --> F[Export labeled templates]
  F --> G[Fine-tune RoBERTa]
  G --> B
  1. Model predicts a priority for each new template
  2. Dashboards surface the prediction with its confidence
  3. Operators correct wrong predictions via the priority override API
  4. Corrections are persisted as ground truth
  5. Periodically, labeled templates are exported to retrain/improve the model

Ground Truth Sources

Source Label Confidence Reliability
Human correction (manual) Operator-chosen priority 1.0 High
Rule fallback (rule) severity→priority 1.0 Medium
ML prediction (ml) Model output softmax Low–Medium
Historical incidents Incident-attached templates — High

Human corrections and incident-linked templates are the preferred training material; model outputs are used for distillation/self-training only when no better label exists.

Label Storage

Labels live in template_priority:

Column Value
template_id PK — the Drain3 template
template Template text (pattern)
priority Ground-truth priority
confidence 1.0 for manual
model_version manual or model id
trained_at When the label was set
prediction_count How many events used it

Export Format

A training export is a JSONL of {text, label} pairs:

{"text": "Connection timeout connecting to <*>", "label": "P2"}
{"text": "Disk space low on <*> partition <*>", "label": "P3"}

Only templates with high-confidence labels (manual or incident-derived) are included.

Retraining Workflow

Training data is exported from template_priority (manual + incident-derived labels only) and used to fine-tune the RoBERTa model into backend/app/Ml_Model/log_priority_roberta.

Note: No training/export scripts are shipped in the repository. The export is produced on demand (e.g. SQL SELECT template, priority FROM template_priority WHERE model_version = 'manual'), and the model artifact is mounted read-only into the pipeline and picked up on restart.

# 1. Export labels (ad-hoc, example)
psql "$DATABASE_URL" -c \
  "COPY (SELECT template, priority FROM template_priority WHERE model_version = 'manual') TO STDOUT" \
  > labels.jsonl

# 2. Fine-tune the model (GPU optional)
#    (use your preferred HuggingFace transformers training loop)

# 3. Bump version
# update DEFAULT model_version in template_cache.py / ml_classifier.py

After retraining, the model is mounted into the pipeline and picked up on restart.

Metrics to Watch

Metric Meaning
Manual correction rate % of predictions overridden — high = model weak
Confidence of accepted High-confidence predictions rarely corrected
prediction_count per template Coverage of the cache
Cache hit rate % of events bypassing the model