Azure DevOps Collector

Polls Azure DevOps REST API v7.1 using WIQL (Work Item Query Language) and publishes work items as raw_events to Kafka.

---

Configuration

Variable	Description	Example
AZURE_DEVOPS_ORG	Organization name	myorg
AZURE_DEVOPS_PROJECT	Project name	MyProject
AZURE_DEVOPS_PAT	Personal Access Token	abc123... (read work items scope)

Poll Interval: 600 seconds (10 min) — configurable in sources.yaml

---

Authentication

PAT as Basic Auth:

Authorization: Basic base64(:{PAT})

Create PAT at: https://dev.azure.com/{org}/_usersSettings/tokens Scope: Work Items (Read)

---

WIQL Query

Default query fetches recently changed work items:

SELECT [System.Id], [System.Title], [System.State], [System.WorkItemType],
       [System.AssignedTo], [System.ChangedDate], [System.CreatedDate],
       [Microsoft.VSTS.Common.Priority], [System.Tags]
FROM WorkItems
WHERE [System.ChangedDate] >= '@last_cursor'
ORDER BY [System.ChangedDate] DESC

Customization (in sources.yaml):

azuredevops:
  poll_interval_seconds: 600
  wiql: |
    SELECT [System.Id], [System.Title], [System.State]
    FROM WorkItems
    WHERE [System.WorkItemType] = 'Bug'
      AND [System.State] IN ('New', 'Active')
    ORDER BY [System.ChangedDate] DESC

---

Data Mapping

Azure DevOps Field	Canonical Event Field
id	tags.ado_id
fields.System.Title	message
fields.System.State	tags.state
fields.System.WorkItemType	tags.work_item_type
fields.System.AssignedTo.displayName	tags.assignee
fields.System.ChangedDate	timestamp (event time)
fields.System.CreatedDate	tags.created_date
fields.Microsoft.VSTS.Common.Priority	tags.priority (1-4)
fields.System.Tags	tags.ado_tags (semicolon-separated)
fields.System.AreaPath	tags.area_path
fields.System.IterationPath	tags.iteration_path

Fixed Fields: - source = azuredevops.{project} - type = event - environment = from AZURE_DEVOPS_ENV or production - severity = derived from priority: 1→CRITICAL, 2→ERROR, 3→WARNING, 4→INFO

---

Polling Logic

sequenceDiagram
  participant SCH as Scheduler
  participant ADO as Azure DevOps API
  participant KAF as Kafka
  participant REDIS as Redis

  SCH->>REDIS: GET polling:cursor:azuredevops
  REDIS-->>SCH: last_changed_date (or null)
  SCH->>ADO: POST /_apis/wit/wiql?api-version=7.1
  Note right of ADO: WIQL with ChangedDate filter
  ADO-->>SCH: Work item refs (ids)
  SCH->>ADO: GET /_apis/wit/workitems?ids=...&fields=...&api-version=7.1
  ADO-->>SCH: Full work items
  loop For each work item
    SCH->>SCH: Map to CanonicalEvent
    SCH->>KAF: Produce raw_events (acks=all)
  end
  KAF-->>SCH: ACK
  SCH->>REDIS: SET polling:cursor:azuredevops = max(ChangedDate)

Cursor: Uses System.ChangedDate (ISO 8601). Only items changed since last cursor are fetched.

---

Pagination

• WIQL returns IDs only (max 200 per query)
• Batch GET /wit/workitems?ids=... fetches full items (max 200 per request)
• Automatic pagination for large result sets

---

Error Handling

Error	Behavior
401 Unauthorized	PAT expired/insufficient scope — alert, stop
403 Forbidden	Project access denied — check PAT project scope
429 Rate Limited	Respect Retry-After header, exponential backoff
5xx / Timeout	Retry with backoff (max 3)
Malformed WIQL	Log error, skip cycle

---

Sample Output

{
  "schema_version": "1.0",
  "timestamp": "2026-01-15T10:30:00Z",
  "source": "azuredevops.MyProject",
  "type": "event",
  "severity": "ERROR",
  "message": "Build pipeline failing intermittently",
  "summary": "State: Active | Assigned: john.doe | Priority: 2",
  "tags": {
    "ado_id": "12345",
    "state": "Active",
    "work_item_type": "Bug",
    "assignee": "john.doe",
    "priority": "2",
    "ado_tags": "backend;pipeline;urgent",
    "area_path": "MyProject\\Backend",
    "iteration_path": "MyProject\\Sprint 12"
  },
  "dedup_key": "sha256:..."
}

---

Troubleshooting

Symptom	Check
No events	Verify PAT has Work Items (Read) scope, correct org/project
401 errors	Regenerate PAT, verify format (no trailing spaces)
Missing fields	Ensure WIQL SELECT includes desired fields
Duplicate events	Pipeline dedup handles; cursor in Redis

# View cursor
docker compose exec redis redis-cli GET "polling:cursor:azuredevops"

# Test WIQL manually
curl -u :$PAT \
  -X POST "https://dev.azure.com/{org}/{project}/_apis/wit/wiql?api-version=7.1" \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT [System.Id] FROM WorkItems WHERE [System.ChangedDate] > \"2026-01-01\" ORDER BY [System.ChangedDate] DESC"}'