Skip to content

Design Principles

The architectural decisions behind LogSys, each with the concrete implementation that enforces it.

1. Event-First Architecture

Principle: All data flows as immutable events through Kafka.

Aspect Implementation
Transport 6 Kafka topics (single partition each)
Immutability Events are append-only; corrections override fields, never delete
Replay Consumer groups + offsets enable replay on failure
Decoupling Producers (Vector/collectors) and consumers (pipeline) never talk directly

2. Single Schema Contract

Principle: One CanonicalEvent shape shared by every service.

Aspect Implementation
Definition Pydantic model in services/libs/common
Validation Malformed events → DLQ (never silently dropped)
Versioning schema_version field for forward compatibility
Benefit Adding a connector touches only the connector

3. Template-Level ML

Principle: Classify log templates, not individual messages.

Aspect Implementation
Why ~1000× fewer inferences than per-message
Parser Drain3 groups messages → template_id
Cache L1/L2/L3 template priority cache
Result ~99% of events bypass the model entirely

4. Graceful Degradation

Principle: Every external dependency can fail without a total outage.

Dependency failure Behavior
ML model Rule-based severity → priority fallback
Redis down Dedup skipped; PG unique index backstop
PostgreSQL down API health shows down; writes buffer in Kafka
Kafka down Collectors buffer/batch; healthcheck alerts

5. Operational Simplicity

Principle: One command runs the entire platform.

Aspect Implementation
Full stack Single docker compose (9 services)
Migrations alembic upgrade head auto-runs at API startup
Secrets .env / .env.example; never committed
Parity Same compose file for dev and prod

6. Observability by Default

Principle: Everything emits structured signals.

Signal Where
Structured logs structlog JSON, per component
Health endpoints /api/monitoring/health, /healthz on each service
OLAP views mv_dashboard_aggregates, mv_timeseries_hourly
Real-time WebSocket /ws/kpis + Redis events:new pub/sub
Metrics Pipeline batch/processing timings logged per stage

7. Security by Design

Principle: Sensitive data and endpoints are protected by default.

Concern Implementation
Auth JWT (HttpOnly cookies, SameSite=Strict)
RBAC Roles viewer → manager → admin → superadmin
PII Stripped in the normalizer before storage
Destructive ops clear-all requires admin/superadmin
Uploads Raw file content never echoed back to client
Network Internal services isolated on aiops_net

8. Data Locality & Performance

Principle: Queries should be fast without denormalizing everything.

Technique Where
Partitioning events partitioned by month
Keyset pagination O(limit) list queries
Materialized views Precomputed KPI aggregates
Trigram GIN ILIKE full-text search
Redis caching KPI results + template priorities