Skip to content

Template Cache

The template cache makes priority lookups nearly free by keeping every known template's priority in fast memory. It has three layers, with PostgreSQL as the source of truth.

Layer Architecture

flowchart TB
  subgraph L1["L1 — Python dict (RAM)"]
    DICT[O(0) lookup<br/>all priorities loaded at startup]
  end
  subgraph L2["L2 — Redis"]
    REDIS[ml:priority:{id}<br/>JSON payload]
  end
  subgraph L3["L3 — PostgreSQL"]
    PG[(template_priority<br/>source of truth)]
  end

  LOOKUP[Lookup] --> DICT
  DICT -->|MISS| REDIS
  REDIS -->|MISS| PG
  PG -->|loaded at startup| DICT
  PRED[RoBERTa new prediction] --> PG
  PG --> REDIS
  PG --> DICT
Layer Read Write Persistence
L1 dict O(0) in-process immediate lost on restart
L2 Redis < 1 ms immediate eviction possible
L3 PostgreSQL indexed PK via storage stage durable (source of truth)

Startup Loading

At pipeline startup, the entire template_priority table is loaded into a Python dict:

class TemplateCache:
    def load_from_postgres(self) -> int:
        conn = psycopg.connect(self._dsn, row_factory=psycopg.rows.dict_row)
        cur.execute("SELECT template_id, priority, confidence, model_version, trained_at::text FROM template_priority")
        for row in cur.fetchall():
            self._cache[str(row["template_id"])] = TemplatePrediction(...)
        return len(self._cache)

This gives O(0) lookups for every known template — the hot path never touches a database.

Cache Entry

@dataclass
class TemplatePrediction:
    priority: str            # P1–P4
    confidence: float        # model confidence or 1.0
    model_version: str       # roberta_v1.1 / manual
    trained_at: str          # ISO date

Redis key format: ml:priority:{template_id} with a JSON body:

{
  "priority": "P2",
  "confidence": 0.91,
  "model_version": "roberta_v1.1",
  "trained_at": "2026-08-11"
}

Write Path

On a cache miss, RoBERTa predicts, then:

  1. Update L1 dict (RAM, immediate)
  2. Set L2 Redis key
  3. Persist L3 PostgreSQL via the storage stage after the template row exists (FK dependency)

If Redis is down, writes are skipped silently — L1 + L3 still function.

Invalidation

Trigger Action
Human priority override invalidate_pattern("ml:priority:{template_id}") — Redis key deleted
Pipeline restart L1 reloaded from PostgreSQL (stale L2 keys simply overwritten)
Eviction Redis LRU policy; re-fetch from L3 if needed

Why Three Layers?

Concern Solved by
Latency L1 dict — sub-microsecond lookups
Multi-process consistency L2 Redis — shared across API + pipeline replicas
Durability L3 PostgreSQL — survives all restarts/evictions