Troubleshooting
A runbook for the most common LogSys issues, from "container won't start" to "ML returns wrong priorities".
Services Not Starting
| Symptom |
Likely cause |
Fix |
kafka-broker restart loop |
Port 9092 busy |
ss -ltnp | grep 9092; free the port |
api unhealthy |
PostgreSQL not ready |
Wait for healthcheck; check .env credentials |
consumer-pipeline exits |
Kafka topic missing |
Check kafka-init completed; recreate topics |
ml-service OOM |
Model + container limit |
Raise memory limit in compose |
| Port conflict on 8000 |
Another API on host |
Set API_PORT=8008 in .env |
# See why a container failed
docker compose logs <service>
docker compose ps
No Events in Dashboard
| Check |
Command / Location |
| Pipeline running |
docker compose ps consumer-pipeline |
| Kafka topics exist |
kafka-topics.sh --list (in broker) |
| Producer active |
docker compose logs collectors (scheduler jobs) |
| Simulator enabled? |
Run devtools/webhook_simulator.py |
| Time window |
Timeseries defaults to last 30 days; check start_time |
If using the simulator:
python devtools/webhook_simulator.py --file ./sample-logs.txt --rate 100
Consumer Lag Growing
docker compose exec kafka-broker /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:19092 --describe --all-groups
| Cause |
Fix |
| Pipeline slow (ML) |
Check logs for processing_time_ms spikes |
| Too few partitions |
Increase partitions + pipeline replicas |
| Redis down |
Dedup skips; verify redis-cli ping |
DLQ / Failed Events
# Count
curl -s http://localhost:8000/api/kpis | jq .dlq_count
# Inspect reasons
docker compose exec postgres psql -U aiops -c \
"SELECT reason, count(*) FROM dlq_events GROUP BY reason ORDER BY 2 DESC LIMIT 10;"
Common reasons: schema validation failures, constraint violations, parse errors. Fix the connector mapping and replay from the DLQ topic.
ML Issues
| Symptom |
Fix |
| All priorities are rule-based |
Model not loaded — check /app/model mount + logs |
| Wrong priorities |
Inspect priority_source; override via API to correct |
| Cache stale after override |
Redis invalidation runs automatically; verify key gone |
| Model loading fails |
Check model directory mounted read-only exists |
docker compose logs consumer-pipeline | grep ml_model
Database Problems
| Issue |
Fix |
| Slow queries |
EXPLAIN ANALYZE; verify partition pruning + indexes |
| Disk full |
Retention: drop old partitions (DROP TABLE events_2026_01) |
| Partition missing for current month |
Run auto-partition code / migration 0013 logic |
| Lock contention |
Check pg_stat_activity; batch writes via executemany |
API Errors
| HTTP |
Meaning |
Fix |
| 401 |
Missing/expired token |
Re-login, refresh token |
| 403 |
Role insufficient |
Check user role vs required permission |
| 404 |
Resource missing |
Verify ID/name, time bounds |
| 413 |
Upload > 10 MB |
Reduce file size |
| 429 |
Rate limited |
Wait for Retry-After |
| 503 |
Dependency down |
Check /api/monitoring/health |
Reset to a Clean Slate
# Keep volumes, wipe application data
curl -X DELETE http://localhost:8000/api/monitoring/clear-all \
-H "Authorization: Bearer $TOKEN" # admin/superadmin
# Full reset including volumes
docker compose down -v
docker compose up -d --build