Ep 20: Knowledge Base Ops — Incremental Updates, Cleanup & Quality Monitoring

⏱ Est. reading time: 10 min Updated on 4/9/2026

RAG Is Not a One-Time Project

Knowledge bases need continuous maintenance, just like code.

graph TB
    subgraph "RAG Operations Loop"
        Index["📥 Index"] --> Retrieve["🔍 Retrieve"]
        Retrieve --> Optimize["⚡ Optimize"]
        Optimize --> Monitor["📊 Monitor"]
        Monitor --> Update["🔄 Incremental Update"]
        Update --> Clean["🗑️ Stale Cleanup"]
        Clean --> Index
    end
    style Monitor fill:#f59e0b,stroke:#d97706,color:#fff

1. Incremental Update Pipeline

graph TB
    Schedule[⏰ Daily 3:00 AM] --> Scan[📂 Scan docs folder]
    Scan --> Compare{New or modified?}
    Compare -->|"New"| Insert["Extract → Chunk → Embed → Insert"]
    Compare -->|"Modified"| Update["Delete old → Re-index"]
    Compare -->|"No change"| Skip[⏭️ Skip]
    Insert & Update --> Record["📝 Update Data Table"]
    style Schedule fill:#6366f1,stroke:#4f46e5,color:#fff

2. Stale Document Cleanup

// Delete vectors older than 90 days via Qdrant API
const ninetyDaysAgo = $now.minus({ days: 90 }).toISO();
await fetch('http://qdrant:6333/collections/knowledge-base/points/delete', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    filter: { must: [{ key: "metadata.indexedAt", range: { lt: ninetyDaysAgo } }] }
  })
});

3. Quality Monitoring (LLM-as-Judge)

graph TB
    Test[📋 20 Standard Q&A Pairs] --> Loop[Loop]
    Loop --> RAG[🔍 RAG Search + Answer]
    RAG --> Judge["🧠 LLM Judge
Score: 1-5"]
    Judge --> Record[📊 Data Table]
    Record --> Report{Avg < 3?}
    Report -->|"Yes"| Alert[🚨 Slack Alert]
    style Judge fill:#8b5cf6,stroke:#7c3aed,color:#fff

Module 4 Complete!

mindmap
  root((Module 4: Enterprise RAG))
    Ep 16 Fundamentals
      Embedding, Chunking, Vector DBs
    Ep 17 Indexing
      Qdrant Deploy, Pipeline
    Ep 18 RAG Agent
      Vector Store Tool, Retrieval
    Ep 19 Advanced
      Hybrid, Re-Rank, Multi-Query
    Ep 20 Operations
      Incremental Update, Cleanup, Monitoring

Next Module

Module 5 covers MCP (Model Context Protocol) — the standardized protocol enabling Agents to connect to any external service, unlocking the "infinite toolbox."