Architecture Overview¶

LogSys follows a cloud-native, event-driven architecture built on proven open-source technologies. The platform ingests heterogeneous telemetry data, normalizes it into a canonical format, enriches it with ML-based prioritization, and exposes it via a real-time API for operational dashboards.

System Context¶

%%{init: {'theme':'default','themeVariables':{'primaryColor':'#1a73e8','secondaryColor':'#e8f0fe','tertiaryColor':'#f8f9fa','lineColor':'#5f6368','fontFamily':'Inter','fontSize':'14px'}}}%%
graph TB
  subgraph EXTERNAL["🌐 External Systems"]
    GIT[GitLab]
    JIR[Jira]
    PROM[Prometheus]
    AZDO[Azure DevOps]
    FILE[Log Files]
  end

  subgraph INGEST["📥 Ingestion Layer"]
    COL[Collectors<br/>APScheduler + FastAPI]
    VEC[Vector<br/>HTTP→Kafka Router]
  end

  subgraph MESSAGE["📨 Message Layer"]
    KAF[Apache Kafka 3.7 KRaft<br/>6 Topics / 1 Partition Each]
  end

  subgraph PROCESS["⚙️ Processing Layer"]
    PIPE[Consumer Pipeline<br/>6 Stages]
    CACHE[(Redis 7<br/>Dedup + Cache)]
  end

  subgraph STORAGE["💾 Storage Layer"]
    PG[(PostgreSQL 16<br/>Partitioned Events)]
    MV[Materialized Views<br/>OLAP Aggregates]
  end

  subgraph SERVE["🌐 Serving Layer"]
    API[FastAPI<br/>REST + WebSocket]
    ML[ML Service<br/>RoBERTa Inference]
  end

  subgraph CONSUME["👁️ Consumption Layer"]
    UI[React Dashboard<br/>Real-time]
    EXT[External Consumers<br/>via API]
  end

  EXTERNAL -->|Push/Poll| INGEST
  INGEST -->|Produce| MESSAGE
  MESSAGE -->|Consume| PROCESS
  PROCESS -->|Read/Write| CACHE
  PROCESS -->|Write| STORAGE
  STORAGE -->|Read| SERVE
  SERVE -->|Query| CONSUME
  CONSUME -.->|Upload| INGEST

Core Design Principles¶

Principle	Implementation
Event-First	All data flows as immutable events through Kafka
Schema Contract	Single `CanonicalEvent` schema (Pydantic) shared by all services
Template-Level ML	Classify Drain3 templates, not individual messages (1000× efficiency)
Graceful Degradation	ML fallback → rule-based priority; dedup best-effort; DLQ for failures
Operational Simplicity	Single `docker compose` for full stack; Alembic for schema migrations
Observability by Default	Structured logging, health endpoints, OLAP views, real-time WS

Data Flow Summary¶

Stage	Technology	Throughput	Latency
Ingest	Vector / Collectors	10K+ eps	< 10ms
Queue	Kafka 3.7 KRaft	100K+ eps	< 5ms
Parse	Drain3	~2,000 eps	~1ms
Normalize	Pydantic v2	~2,200 eps	~0.5ms
Dedup	Redis Lua	~2,100 eps	< 1ms
Features	Polars	~2,300 eps	~0.2ms
ML Classify	RoBERTa (cached)	~2,050 eps	~2ms (cache hit: <1ms)
Storage	PostgreSQL batch INSERT	~2,050 eps	~5ms
API	FastAPI + Redis	10K+ rps	< 50ms
UI	React + WebSocket	Real-time	< 100ms

Service Inventory¶

Service	Replicas	Memory	CPU	Ports	Dependencies
`kafka-broker`	1	1 GB	0.5	9092	—
`postgres`	1	512 MB	0.5	5432	—
`redis`	1	200 MB	0.25	6379	—
`vector`	1	200 MB	0.25	8686, 8687	Kafka
`api`	1–3	300 MB	0.5	8000	PG, Redis, Kafka
`collectors`	1	300 MB	0.25	8080	API, Kafka, Redis
`consumer-pipeline`	1–6	2 GB	1–2	—	API, PG, Redis, Kafka
`ml-service`	1	2 GB	1–2	8001	—

Startup Sequence¶

sequenceDiagram
  participant K as Kafka Broker
  participant KI as Kafka Init
  participant PG as PostgreSQL
  participant R as Redis
  participant V as Vector
  participant A as API
  participant C as Collectors
  participant P as Pipeline
  participant M as ML Service

  K->>K: Start KRaft controller
  KI->>K: Wait healthy → Create topics
  PG->>PG: Start, healthcheck
  R->>R: Start, healthcheck
  V->>K: Connect, wait topics
  A->>PG: Connect
  A->>R: Connect
  A->>A: alembic upgrade head
  A->>A: Start FastAPI + background tasks
  C->>A: Wait service_started
  C->>KI: Wait completed
  C->>R: Wait healthy
  C->>C: Start scheduler + webhook server
  P->>A: Wait service_started
  P->>KI: Wait completed
  P->>PG: Wait healthy
  P->>R: Wait healthy
  P->>P: Load Drain3 state, RoBERTa model
  P->>P: Start consuming
  M->>M: Load RoBERTa model
  M->>M: Start inference server

Failure Modes & Resilience¶

Failure	Detection	Mitigation
Kafka broker down	Healthcheck + consumer lag	KRaft single-node (dev); multi-broker (prod)
PostgreSQL down	API `/monitoring/health`	Connection pool retry; circuit breaker
Redis down	Pipeline dedup fails open	Dedup skipped; PG unique index backstop
ML service down	Classifier returns `None`	Rule-based fallback (severity→priority)
Pipeline crash	Consumer lag alert	Graceful shutdown; offset commit; DLQ
Vector down	Ingestion stops	Collectors direct to Kafka; healthcheck
API overload	Rate limiting (slowapi)	429 responses; horizontal scaling

Security Boundaries¶

graph LR
  subgraph DMZ["DMZ / Public"]
    LB[Load Balancer<br/>TLS Termination]
    VEC[Vector :8686/8687]
  end

  subgraph PRIVATE["Private Network (aiops_net)"]
    API[API :8000]
    KAF[Kafka :19092]
    PG[Postgres :5432]
    REDIS[Redis :6379]
    COL[Collectors :8080]
    PIPE[Pipeline]
    ML[ML Service :8001]
  end

  LB -->|HTTPS| API
  LB -->|HTTPS| VEC
  VEC --> KAF
  COL --> KAF
  KAF --> PIPE
  PIPE --> PG
  PIPE --> REDIS
  API --> PG
  API --> REDIS
  API --> KAF
  ML -.->|Model inference| PIPE

Boundary	Protection
External → API	TLS, JWT, Rate Limit, CORS
External → Vector	HMAC (GitLab), JWT (uploads)
Internal → Kafka	Network isolation (`aiops_net`)
Internal → PG/Redis	Auth via env vars, no external ports
Pipeline → ML	Read-only model mount, local network

Scaling Strategy¶

Component	Horizontal Scaling	Vertical Scaling
API	Add replicas behind LB	Increase memory (cache)
Pipeline	Add replicas (matches Kafka partitions)	Increase batch size, memory
Collectors	Single active (leader election)	N/A
Kafka	Add brokers + partitions	Increase heap
PostgreSQL	Read replicas (future)	Increase CPU/RAM, tune `work_mem`
Redis	Cluster mode (future)	Increase `maxmemory`

Technology Decisions¶

Category	Choice	Rationale
Message Broker	Kafka 3.7 KRaft	High throughput, durability, replay, no ZK ops
Stream Processor	Custom Python pipeline	Fine-grained control, ML integration, Drain3 native
API Framework	FastAPI + ORJSON	Performance, OpenAPI, type safety
Database	PostgreSQL 16	JSONB, partitioning, materialized views, pg_trgm
Cache	Redis 7	Lua scripting, pub/sub, LRU eviction
ML Inference	RoBERTa + ONNX (future)	Template-level classification, CPU-friendly
Log Parser	Drain3	Online, streaming, persisted state
Frontend	React 19 + TanStack	Type-safe routing, query, table; real-time WS
Vector	Timber Vector	Zero-code HTTP→Kafka, VRL transforms
Container	Docker Compose	Single-file dev/prod parity