Webhook Receiver (GitLab)

FastAPI endpoint running inside the Collectors container (:8080) that receives GitLab webhooks and publishes to Kafka.

---

Endpoint

Method	Path	Description
POST	/webhooks/gitlab	GitLab webhook receiver
GET	/health	Health check

---

GitLab Configuration

1. In GitLab: Settings → Webhooks
2. URL: http://<host>:8080/webhooks/gitlab
3. Secret Token: Set GITLAB_WEBHOOK_SECRET in .env
4. Trigger events: Push, Issues, Merge Requests, Pipeline, Job, etc.
5. SSL verification: Disable for self-signed (dev)

---

HMAC Verification

GitLab sends X-Gitlab-Token header with HMAC-SHA256 of payload:

# Verification
expected = hmac.new(
    GITLAB_WEBHOOK_SECRET.encode(),
    await request.body(),
    hashlib.sha256
).hexdigest()

provided = request.headers.get("X-Gitlab-Token")
if not hmac.compare_digest(expected, provided):
    raise HTTPException(401, "Invalid webhook signature")

---

Supported Event Types

GitLab Event	object_kind	Mapping
Push	push	Commits → events
Issue	issue	Issue open/close/update
Merge Request	merge_request	MR open/close/merge
Pipeline	pipeline	Pipeline status
Job	build	Job status
Tag Push	tag_push	Tag events
Release	release	Release published

---

Data Mapping (Common Fields)

GitLab Payload	Canonical Event
project.name	source = gitlab.{project_path}
object_kind	tags.gitlab_event_type
event_name	tags.gitlab_event_name
object_attributes.title	message
object_attributes.description	summary
object_attributes.state	tags.state
object_attributes.url	tags.url
user.username	tags.triggered_by
repository.homepage	tags.repo_url

Fixed Fields: - type = event - environment = from GITLAB_ENV or production - severity = derived from event type: - Pipeline failed → CRITICAL - Job failed → ERROR - Issue/MR opened → INFO - Push → INFO

---

Sample Payload (Pipeline Event)

{
  "object_kind": "pipeline",
  "event_name": "pipeline",
  "object_attributes": {
    "id": 12345,
    "status": "failed",
    "ref": "main",
    "sha": "abc123...",
    "web_url": "https://gitlab.com/group/proj/-/pipelines/12345"
  },
  "project": {
    "name": "my-service",
    "path_with_namespace": "group/my-service"
  },
  "user": {
    "username": "john.doe"
  }
}

Canonical Output:

{
  "schema_version": "1.0",
  "timestamp": "2026-01-15T10:30:00Z",
  "source": "gitlab.group/my-service",
  "type": "event",
  "severity": "CRITICAL",
  "message": "Pipeline failed on main (abc123...)",
  "summary": "Pipeline #12345 failed for commit abc123...",
  "tags": {
    "gitlab_event_type": "pipeline",
    "gitlab_event_name": "pipeline",
    "pipeline_id": "12345",
    "status": "failed",
    "ref": "main",
    "sha": "abc123...",
    "triggered_by": "john.doe",
    "repo_url": "https://gitlab.com/group/my-service"
  },
  "dedup_key": "sha256:..."
}

---

Testing

Simulator

# Send test payload
python devtools/webhook_simulator.py \
  --file ./sample-gitlab-webhook.json \
  --rate 10 \
  --url http://localhost:8080/webhooks/gitlab

Manual curl

curl -X POST http://localhost:8080/webhooks/gitlab \
  -H "Content-Type: application/json" \
  -H "X-Gitlab-Token: $(echo -n '{"test":1}' | openssl dgst -sha256 -hmac "$GITLAB_WEBHOOK_SECRET")" \
  -d '{"object_kind":"push","project":{"name":"test","path_with_namespace":"group/test"}}'

---

Troubleshooting

Symptom	Check
401 Invalid signature	Verify GITLAB_WEBHOOK_SECRET matches GitLab webhook config
400 Bad Request	Payload missing required fields; check GitLab webhook test
Not received	GitLab webhook delivery log (Settings → Webhooks → Test)
Duplicate events	Pipeline dedup handles (300s window)

# View logs
docker compose logs -f collectors | grep -i gitlab

# Test HMAC locally
echo -n '{"test":1}' | openssl dgst -sha256 -hmac "$GITLAB_WEBHOOK_SECRET"