Scaling¶
LogSys scales horizontally where the architecture allows and vertically where it does not. Each component has a strategy matched to its role.
Scaling Model¶
| Component | Horizontal | Vertical | Constraint |
|---|---|---|---|
| API | Add replicas behind LB | Bump memory (cache) | Stateless; Redis for shared state |
| consumer-pipeline | Add replicas | Batch size, memory | Must match Kafka partitions |
| collectors | Single active (leader) | N/A | Avoid duplicate polling |
| Kafka | Add brokers + partitions | Heap (KAFKA_HEAP_OPTS) |
KRaft single-node in dev |
| PostgreSQL | Read replicas (future) | CPU/RAM, work_mem |
Partitions + indexes |
| Redis | Cluster mode (future) | maxmemory |
Dedup windows + cache |
| ml-service | Replicas (stateless) | GPU later | Model load x memory |
The Pipeline Bottleneck¶
consumer-pipeline is the throughput-critical component. Kafka currently uses 1 partition per topic, which caps consumer parallelism at 1 per consumer group (one consumer per partition).
To scale the pipeline horizontally:
- Increase topic partitions (e.g. to 6) —
kafka-topics.sh --alter - Run multiple
consumer-pipelinereplicas - Each replica owns a subset of partitions
Until partitions are increased, adding pipeline replicas provides no parallelism gain — this is by design for dev simplicity (single-partition ordering guarantees).
API Scaling¶
The API is stateless (sessions/state live in Redis + PostgreSQL), so it scales trivially:
Put it behind a load balancer that checks /api/monitoring/health. Cached KPI responses (kpis_global:*) are shared via Redis, so replicas serve warm data immediately.
Kafka Scaling¶
| Change | Command (in container) |
|---|---|
| Add broker | New KRaft node + controller.quorum.voters |
| Increase partitions | kafka-topics.sh --alter --topic canonical-events --partitions 6 |
Higher partition counts trade ordering guarantees for parallelism.
PostgreSQL Scaling¶
| Technique | Status |
|---|---|
| Partitioning (monthly) | In production |
| Index tuning | In production (GIN trigram, composites) |
work_mem increase |
Recommended for large aggregates |
| Read replicas | Future work |
| Connection pooling | PgBouncer if replica count grows |
Redis Scaling¶
Current limits: maxmemory 150mb, policy allkeys-lru. For larger deployments:
- Increase
maxmemory(dedup windows grow with volume) - Move to cluster mode for multi-GB state
- Keep cache vs dedup keys separated if eviction policy conflicts
ML Scaling¶
ml-service is stateless (model weights read-only) — scale replicas freely:
- CPU-bound inference; add replicas for higher QPS
- The pipeline's inline classifier shares the same model and is the primary path
- GPU acceleration is a future option
Collectors¶
Poll-based collectors run as a single active process (APScheduler leader) to avoid duplicate polling. To scale, split sources across collector instances rather than replicating the same schedule.
Capacity Planning Signals¶
| Signal | Indicates | Action |
|---|---|---|
| Consumer lag rising | Pipeline is bottleneck | Increase partitions + replicas |
| API p99 > 50 ms | API overload | Scale API replicas |
Redis dbsize high |
Dedup window too large | Tune window / raise memory |
| PG slow aggregates | KPI queries heavy | Raise work_mem, check indexes |
| DLQ growing | Downstream failures | Investigate storage/validation |