Skip to content

Deployment

LogSys deploys as a Docker Compose stack — 9 services that run identically in dev and production.

Prerequisites

  • Docker Engine ≥ 24
  • Docker Compose v2
  • .env file (copy from .env.example, see Configuration)

Services

Service Image/Build Ports Dependencies
kafka-broker apache/kafka:3.7.0 (KRaft) 9092 —
kafka-init same image (one-shot) — kafka healthy
postgres postgres:16-alpine 5432 —
redis redis:7-alpine 6379 —
vector timberio/vector:0.45.X 8686, 8687 kafka healthy
api backend/Dockerfile 8000 postgres + redis healthy
collectors services/collectors/Dockerfile 8080 api + kafka-init + redis
consumer-pipeline services/pipeline/Dockerfile — api + kafka-init + postgres + redis
ml-service backend/app/Ml_Model/Dockerfile 8001 —
simulator devtools/webhook_simulator.py (Python script) — demo data, not a container

Deploy

# 1. Configure environment
cp .env.example .env
# edit secrets/ports

# 2. Full stack (excludes simulator)
docker compose up -d --build

# 3. Send demo data (simulator script)
python devtools/webhook_simulator.py --file ./sample-logs.txt --rate 100

# 4. Verify
curl -s http://localhost:8000/api/monitoring/health

Startup Order

Dependencies are enforced via depends_on + healthchecks:

  1. kafka-broker healthy → kafka-init creates topics → completes
  2. postgres + redis healthy
  3. api waits for postgres/redis, runs alembic upgrade head, starts
  4. vector connects to Kafka
  5. collectors + consumer-pipeline wait for api + kafka-init + infra
  6. ml-service loads RoBERTa (independent)

Networking

All services share the aiops_net bridge network. Only the API, Vector, collectors, Kafka, Postgres, Redis, and ML ports are exposed to the host.

Persistence

Volume Backs
kafka-data Kafka log segments
pg-data PostgreSQL data
redis-data Redis RDB snapshots
pgadmin-data pgAdmin config

Volumes are not deleted on docker compose down; use down -v with caution.

Production Hardening

Concern Action
TLS Terminate at load balancer → API/Vector
Secrets POSTGRES_PASSWORD, JWT keys via secret manager
Replicas Scale api and consumer-pipeline behind LB
Backups pg_dump scheduled; test restore
Resource limits Set explicit deploy.resources.limits
Healthchecks Point LB at /api/monitoring/health
Logging docker compose logs -f or ship to aggregator

Update

git pull
docker compose up -d --build --remove-orphans

Alembic migrations apply automatically at API startup.