Skip to content

Real-time Updates

Real-time capabilities are built on WebSocket + Redis pub/sub. The backend pushes KPI snapshots; the pipeline notifies of new events.

Architecture

sequenceDiagram
  participant P as Pipeline (storage.py)
  participant R as Redis
  participant A as API (ws manager)
  participant F as React (useKpiStream)

  P->>R: PUBLISH events:new {count}
  R-->>A: forwarded to WS manager
  A-->>F: kpi:snapshot (recomputed)
  Note over F: dashboard updates live
Producer Channel Consumer
Storage stage events:new (Redis pub/sub) WS manager
WS server kpi:snapshot (WebSocket) Dashboard
Client ping / pong Keepalive

Client Hook: useKpiStream

The frontend connects via a dedicated hook:

const { snapshot, connected } = useKpiStream();

Behavior:

  1. Open WS /ws/kpis (JWT via cookie/header)
  2. Render the initial kpi:snapshot
  3. Re-render on each new snapshot
  4. On disconnect → auto-reconnect with exponential backoff
  5. Exposes connected for UI badges

WebSocket Protocol

Message Direction Meaning
kpi:snapshot server → client Full KPI state
ping client → server Keepalive
pong server → client Reply
ping server → client Server idle keepalive (30 s)

Auth fallbacks: Bearer header → sub-protocol jwt.<token> → HttpOnly cookie → ?token= query. See WebSocket API.

Push vs Poll Strategy

Channel When Why
WebSocket Live dashboard Instant updates, low overhead
REST Initial load, page navigation, reconnects Simple, cacheable
Redis pub/sub Pipeline → API notification Decouples pipeline from API

The dashboard uses push-first, poll-fallback: WS keeps data live, REST covers the gaps during reconnects.

Rate Limiting

The WS endpoint enforces a per-IP limit (ws_limiter). Clients exceeding it are closed with 1008 Policy Violation.

Operational Notes

  • Keepalive prevents idle socket expiry (server pings after 30 s idle)
  • Reconnect backoff avoids reconnect storms after broker/API restarts
  • KPI snapshots are cheap: results are cached in Redis (kpis_global:*)