Services Overview¶

LogSys consists of 9 Docker services orchestrated via Docker Compose. This section documents each service's responsibility, configuration, and operational details.

Service Catalog¶

Service	Container	Image	Ports	Memory	Replicas	Purpose
Kafka Broker	`kafka-broker`	`apache/kafka:3.7.0`	9092	1 GB	1	Message broker (KRaft)
Kafka Init	`kafka-init`	`apache/kafka:3.7.0`	—	—	1 (job)	Topic provisioning
PostgreSQL	`postgres`	`postgres:16-alpine`	5432	512 MB	1	Primary storage
Redis	`redis`	`redis:7-alpine`	6379	200 MB	1	Cache, dedup, pub/sub
Vector	`vector`	`timberio/vector:0.45.X-debian`	8686, 8687	200 MB	1	HTTP → Kafka router
API	`api`	Build: `./backend`	8000	300 MB	1–3	REST API + WS
Collectors	`collectors`	Build: `./services/collectors`	8080	300 MB	1	Polling + webhooks
Pipeline	`consumer-pipeline`	Build: `services/pipeline/Dockerfile`	—	2 GB	1–6	Parse → Store
ML Service	`ml-service`	Build: `./backend/app/Ml_Model`	8001	2 GB	1	RoBERTa inference

Network & Volumes¶

%%{init: {'theme':'default','themeVariables':{'primaryColor':'#1a73e8'}}}%%
graph LR
  NET[aiops_net<br/>bridge]

  PGD[(pg-data)]
  KD[(kafka-data)]
  RD[(redis-data)]
  PAD[(pgadmin-data)]

  KAF[kafka-broker] --- NET
  PG[postgres] --- NET
  REDIS[redis] --- NET
  VEC[vector] --- NET
  API[api] --- NET
  COL[collectors] --- NET
  PIPE[consumer-pipeline] --- NET
  ML[ml-service] --- NET

  KAF -.-> KD
  PG -.-> PGD
  REDIS -.-> RD

Volume	Service	Purpose
`kafka-data`	kafka-broker	Kafka log segments
`pg-data`	postgres	PostgreSQL data directory
`redis-data`	redis	Redis persistence (RDB/AOF)
`pgadmin-data`	pgadmin	pgAdmin config (optional)
`./backend:/app`	api	Hot-reload (dev override)
`./services/pipeline:/app`	pipeline	Hot-reload (dev override)
`./services/collectors:/app`	collectors	Hot-reload (dev override)
`./services/libs/common:/app/libs/common`	pipeline, collectors	Shared library mount
`./state:/app/state`	pipeline	Drain3 state persistence
`./backend/app/Ml_Model/log_priority_roberta:/app/model/log_priority_roberta:ro`	pipeline, ml-service	Read-only model weights

Startup Dependencies¶

%%{init: {'theme':'default','themeVariables':{'primaryColor':'#1a73e8'}}}%%
sequenceDiagram
  participant K as kafka-broker
  participant KI as kafka-init
  participant PG as postgres
  participant R as redis
  participant V as vector
  participant A as api
  participant C as collectors
  participant P as consumer-pipeline
  participant M as ml-service

  K->>K: Start KRaft
  KI->>K: Wait healthy
  KI->>KI: Create topics
  PG->>PG: Start + healthcheck
  R->>R: Start + healthcheck
  V->>K: Wait healthy
  A->>PG: Wait healthy
  A->>R: Wait healthy
  A->>A: alembic upgrade head
  A->>A: Start FastAPI
  C->>A: condition: service_started
  C->>KI: condition: service_completed_successfully
  C->>R: condition: service_healthy
  P->>A: condition: service_started
  P->>KI: condition: service_completed_successfully
  P->>PG: condition: service_healthy
  P->>R: condition: service_healthy
  M->>M: Load model, start server

Service	Waits For	Condition
`vector`	`kafka-broker`	`service_healthy`
`api`	`postgres`, `redis`	`service_healthy`
`collectors`	`api`, `kafka-init`, `redis`	`service_started`, `service_completed_successfully`, `service_healthy`
`consumer-pipeline`	`api`, `kafka-init`, `postgres`, `redis`	`service_started`, `service_completed_successfully`, `service_healthy`

Resource Limits (Production Tuning)¶

# Example production overrides
deploy:
  resources:
    limits:
      cpus: '2'
      memory: 4g
    reservations:
      cpus: '1'
      memory: 2g

Service	CPU Limit	Memory Limit	Tuning Notes
`kafka-broker`	1.0	1 GB	Increase `KAFKA_HEAP_OPTS` for higher throughput
`postgres`	1.0	1 GB	Tune `shared_buffers`, `work_mem`, `effective_cache_size`
`redis`	0.5	512 MB	Increase `maxmemory` for larger dedup windows
`api`	1.0	512 MB	Scale horizontally behind LB
`collectors`	0.5	512 MB	Single active instance (leader election needed for HA)
`consumer-pipeline`	2.0	4 GB	Scale to match Kafka partitions
`ml-service`	2.0	4 GB	GPU optional (CPU inference ~50ms)

Health Checks¶

Service	Check	Interval	Timeout	Retries
`kafka-broker`	`kafka-broker-api-versions.sh`	10s	10s	15
`postgres`	`pg_isready`	5s	5s	10
`redis`	`redis-cli ping`	5s	5s	10
`api`	`GET /health` (implicit)	—	—	—
`collectors`	`curl /health`	30s	10s	3
`vector`	Implicit (process)	—	—	—

Logs & Observability¶

# All services
docker compose logs -f

# Specific service
docker compose logs -f consumer-pipeline

# Last 100 lines
docker compose logs --tail=100 api

# Follow with timestamps
docker compose logs -f -t consumer-pipeline

# Filter by level (structlog JSON)
docker compose logs consumer-pipeline | jq 'select(.level=="error")'

Common Operations¶

Task	Command
Restart single service	`docker compose restart api`
Rebuild after code change	`docker compose up -d --build api`
Scale pipeline	`docker compose up -d --scale consumer-pipeline=3`
View resource usage	`docker stats $(docker compose ps -q)`
Exec into container	`docker compose exec api bash`
Backup PostgreSQL	`docker compose exec postgres pg_dump -U aiops aiops > backup.sql`
Reset all data	`docker compose down -v && docker compose up -d`