LogSys Documentation

AIOps AlertOps Platform — Pillar 1: Data Collection & Preparation

Complete technical documentation for the LogSys platform: a real-time streaming pipeline for collecting, normalizing, deduplicating, and analyzing logs, metrics, and traces.

Property	Value
Project	AIOps AlertOps — Pillar 1
Codename	LogSys
Architecture	Event-driven microservices (Kafka + FastAPI + React)
Data Throughput	~2,100 logs/sec sustained
Latency (E2E)	< 500ms
Deployment	Docker Compose (9 services)

---

🏗️ High-Level Architecture

%%{init: {'theme':'default','themeVariables':{'primaryColor':'#1a73e8','secondaryColor':'#e8f0fe','tertiaryColor':'#f8f9fa','lineColor':'#5f6368','fontFamily':'Inter','fontSize':'14px'}}}%%
graph TB
  subgraph INPUTS["📥 Data Sources"]
    WH[GitLab Webhook]
    UP[File Upload<br/>CSV/JSON/TXT]
    PO[Polling<br/>Jira/Prometheus/Azure]
  end

  subgraph ROUTER["🔀 HTTP → Kafka Router"]
    VEC[Vector :8686/8687]
  end

  subgraph BROKER["📨 Message Broker"]
    KAF[Apache Kafka 3.7 KRaft<br/>raw_logs, raw_metrics,<br/>raw_traces, raw_events,<br/>canonical-events, dlq]
  end

  subgraph PIPELINE["⚙️ Consumer Pipeline"]
    direction LR
    P1[1. Parse<br/>Drain3]
    P2[2. Normalize<br/>Pydantic + PII masking]
    P3[3. Dedup<br/>Redis Lua]
    P4[4. Features<br/>Polars]
    P5[5. ML Classify<br/>RoBERTa]
    P6[6. Storage<br/>PostgreSQL batch INSERT]
    P1 --> P2 --> P3 --> P4 --> P5 --> P6
  end

  subgraph API["🖥️ API Layer"]
    FAS[FastAPI<br/>35 endpoints<br/>JWT + RBAC<br/>Redis Cache]
    WS[WebSocket<br/>/ws/kpis]
  end

  subgraph UI["🎨 Frontend"]
    REA[React 19 + TypeScript<br/>TanStack Table/Query<br/>Real-time Dashboard]
  end

  INPUTS --> VEC --> KAF
  KAF --> PIPELINE
  PIPELINE --> API
  PIPELINE -.->|Cache/State| REDIS[(Redis 7)]
  API --> REA
  REA -.->|Upload| VEC
  API <--> DB[(PostgreSQL 16<br/>Partitioned)]

---

🔑 Key Capabilities

🔄 Real-time Pipeline

End-to-end streaming from source to dashboard in <500ms. Batched Kafka consumption with horizontal scaling via partitions.

🧠 ML-Powered Prioritization

RoBERTa-based classifier (P1–P4) with template-level caching. 1000× fewer inferences vs per-message classification. Fallback rule engine for reliability.

🔍 Intelligent Deduplication

SHA-256 based deduplication with Redis Lua atomic scripts. 300-second sliding window. ~76% deduplication rate on reference dataset.

📊 AI Insights Engine

Rule-based RCA (no LLM) with root cause detection, recommended actions, MTTD/MTTA/MTTR calculation, and anomaly detection across services.

🎨 Modern Frontend

React 19 + TanStack Router/Query/Table. Real-time WebSocket updates. RBAC with page/app/KPI granularity. DXC "Stitch & Glow" design system.

🛡️ Enterprise Security

JWT in HttpOnly cookies + Bearer tokens. Role-based access (superadmin/admin/manager/viewer). Rate limiting. CORS guard. PII masking at ingestion.

---

📋 Version Information

LogSys v1.0.0 Python 3.11+ FastAPI 0.115 PostgreSQL 16 Kafka 3.7 React 19 TypeScript 6