Jira Collector

Polls Jira REST API v3 for issues and publishes them as raw_events to Kafka.

---

Configuration

Variable	Description	Example
JIRA_BASE_URL	Jira instance URL	https://company.atlassian.net
JIRA_EMAIL	Service account email	jira-collector@company.com
JIRA_API_TOKEN	API token (not password)	ATATT3xFfGF0...

Poll Interval: 600 seconds (10 min) — configurable in sources.yaml

---

Authentication

Basic Auth (RFC 7617):

Authorization: Basic base64(email:api_token)

Never use password. Create API token at: https://id.atlassian.com/manage-profile/security/api-tokens

---

JQL Query

Default query fetches recently updated issues:

jql = (
    "ORDER BY updated DESC "
    "AND updated >= -10m"  # Relative to poll interval
)

Customization (in sources.yaml):

jira:
  poll_interval_seconds: 600
  jql: "project = MYPROJECT AND issuetype = Bug"

---

Data Mapping

Jira Field	Canonical Event Field
key (e.g., PROJ-123)	tags.jira_key
summary	message
description	summary
status.name	tags.jira_status
priority.name	tags.jira_priority
assignee.displayName	tags.assignee
reporter.displayName	tags.reporter
created	tags.created
updated	tags.updated
project.key	source (with prefix jira.)

Fixed Fields: - source = jira.{project_key} - type = event - environment = from JIRA_ENV env var or production - severity = derived from Jira priority (Critical→CRITICAL, High→ERROR, etc.)

---

Polling Logic

sequenceDiagram
  participant SCH as Scheduler
  participant JIR as Jira API
  participant KAF as Kafka
  participant REDIS as Redis

  SCH->>SCH: Every 600s
  SCH->>REDIS: GET polling:cursor:jira
  REDIS-->>SCH: last_cursor (or null)
  SCH->>JIR: GET /rest/api/3/search?jql=...&startAt=0
  JIR-->>SCH: Issues page
  loop For each issue
    SCH->>SCH: Map to CanonicalEvent
    SCH->>KAF: Produce raw_events (acks=all)
  end
  KAF-->>SCH: ACK
  SCH->>REDIS: SET polling:cursor:jira = new_cursor

Cursor: Uses updated timestamp. Only issues updated since last cursor are fetched.

---

Error Handling

Error	Behavior
Network timeout	Retry with exponential backoff (max 3)
401 Unauthorized	Log error, alert, stop polling
429 Rate Limited	Respect Retry-After, backoff
5xx Server Error	Retry with backoff
Kafka produce fail	Retry (max 5), then log to DLQ

Tenacity Configuration:

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError))
)

---

Sample Output (Kafka raw_events)

{
  "schema_version": "1.0",
  "timestamp": "2026-01-15T10:30:00Z",
  "source": "jira.PROJ",
  "type": "event",
  "severity": "ERROR",
  "message": "Database connection pool exhausted",
  "summary": "Users unable to login - PROJ-456",
  "tags": {
    "jira_key": "PROJ-456",
    "jira_status": "In Progress",
    "jira_priority": "High",
    "assignee": "john.doe",
    "reporter": "jane.smith"
  },
  "dedup_key": "sha256:..."
}

---

Troubleshooting

Symptom	Check
No events	Verify JIRA_BASE_URL, credentials, network access
401 errors	Regenerate API token, check email matches
Duplicate events	Pipeline dedup handles; check cursor in Redis
High lag	Reduce poll interval, check Jira API rate limits

# View last cursor
docker compose exec redis redis-cli GET "polling:cursor:jira"

# View collector logs
docker compose logs -f collectors | grep -i jira