Vector

Vector (Timber) is a lightweight HTTP → Kafka router that receives ephemeral uploads and webhooks, transforms them, and publishes to canonical-events.

---

Role

Input	Port	Source	Auth
Ephemeral upload	8686	API /api/sources/ephemeral	JWT
Generic webhook	8687	External systems	HMAC / None

---

Configuration (services/vector/vector.yaml)

sources:
  ephemeral_http:
    type: http
    address: 0.0.0.0:8686
    encoding: json
    strict_path: false

  webhook_http:
    type: http
    address: 0.0.0.0:8687
    encoding: json
    strict_path: false

transforms:
  ephemeral_enrich:
    type: remap
    inputs: ["ephemeral_http"]
    source: |
      .timestamp = .timestamp || now()
      .ingested_at = now()
      .source = .source || "ephemeral"
      .source_type = .source_type || "http"
      .environment = get_env_var("ENV") ?? "production"
      if is_object(.message) {
        .message = encode_json(.message)
      }

  webhook_enrich:
    type: remap
    inputs: ["webhook_http"]
    source: |
      .timestamp = .timestamp || now()
      .ingested_at = now()
      .source = .source || "webhook"
      .environment = get_env_var("ENV") ?? "production"

sinks:
  kafka:
    type: kafka
    inputs: ["ephemeral_enrich", "webhook_enrich"]
    bootstrap_servers: "${KAFKA_BROKERS:-kafka-broker:19092}"
    topic: canonical-events
    encoding:
      codec: json
    healthcheck:
      enabled: true

---

Data Flow

flowchart LR
  API[API :8000<br/>POST /sources/ephemeral] -->|Stream| VEC1[Vector :8686]
  WH[External Webhook] -->|HTTP| VEC2[Vector :8687]
  VEC1 --> REMAP[VRL Transform]
  VEC2 --> REMAP
  REMAP --> KAFKA[(Kafka<br/>canonical-events)]

---

VRL Transform Details

Two separate VRL transforms enrich each input; ingested_at is always stamped and source/environment get sensible defaults. Ephemeral uploads additionally JSON-encode object messages.

Transform	Input	VRL Highlights
ephemeral_enrich	ephemeral_http	.timestamp || now(), .source = "ephemeral", encode_json(.message)
webhook_enrich	webhook_http	.timestamp || now(), .source = "webhook", get_env_var("ENV")

Vector publishes raw JSON to canonical-events without compression; the fields are further validated by the pipeline's CanonicalEvent model.

---

Scaling

Replicas	Use Case
1	Dev / Low volume
2–3	Production (HA)
5+	High-volume uploads

Stateless — scale horizontally behind load balancer.

---

Health Check

curl http://localhost:8686/health
# {"status": "ok"}

# Metrics (Prometheus format)
curl http://localhost:8686/metrics

---

Troubleshooting

Symptom	Check
Events not in Kafka	docker compose logs vector — check remap errors
400 on upload	Payload missing message field
High latency	Vector CPU, Kafka backpressure
Schema mismatch	Verify VRL transform outputs all required CanonicalEvent fields