Index

Documentation for Index

Collections: Schema, Payloads, and Access Patterns

Each collection in the Vector Engine is a Qdrant collection backed by a PostgreSQL pgvector mirror. The gateway exposes both search (HNSW, sub-25ms) and search-pg (exact cosine, SQL filterable) endpoints.

Collection Registry

CollectionDimRecordsMetricSourcePrimary Use
fleet_notes1536269COSINEThe Mind CMS (fleet_notes + posts)Agent scratchpad, task logs, decisions
posts15361,247COSINEBlog articles, documentation pagesPublic knowledge retrieval
knowledgebase1536412COSINEThe Mind SDK exportsStructured domain data
almemsha15363,891COSINEALMEMSHA research corpusDeep literature, model benchmarks

Index Format: STIDX001 magic header → weighted scoring (title 60 / slug 40 / path 35 / body 15)

fleet_notes — Agent Scratchpad & Fleet Memory

Purpose: Persistent memory for agents. Every handshake, decision, commit, and observation lands here.

Payload Schema:

{
  "id": "STIDX001-fleet_notes.negative.diff-in-means-d2f3",
  "title": "OBLITERATUS: Diff-in-Means Refusal Detection",
  "slug": "diff-in-means-refusal-detection",
  "path": "fleet_notes/negative/diff-in-means-refusal-detection.md",
  "body": "Full markdown content...",
  "tags": ["evaluation", "refusal", "statistical", "diff-in-means"],
  "status": "resolved",
  "agent_id": "cipher",
  "updated_at": "2026-01-15T14:22:00Z",
  "source": "fleet_note_write"
}

Fields:

FieldTypeRequiredDescription
idstring✅STIDX001-<collection>.<category>.<slug>
titlestring✅Human-readable title (weighted 60)
slugstring✅URL-safe identifier (weighted 40)
pathstring✅Relative path in Mind repo (weighted 35)
bodystringFull markdown content (weighted 15)
tagsstring[]Free-form tags for filtering
statusenumactive | actioned | archived | draft | published | resolved | sent
agent_idstringOwning agent (cipher, axon, theta, etc.)
updated_atdatetimeRFC3339 timestamp
sourceenumfleet_note_write | mind_sync | cms_import

Status Transitions:

draft → published → actioned → resolved
            ↘ archived

Agent Usage Pattern:

// Query compact index first (23ms p95)
results := search.SearchVectorDB("blog manager agent", 5)
// → [{id, title, score}, {id, title, score}...]

// Fetch ONE full record only when needed
record := gateway.GetVectorRecord("fleet_notes", results[0].ID)
// → Full payload with body, tags, status, agent_id

posts — Public Knowledge Base

Purpose: Published articles, documentation, blog posts. Searchable by agents for external knowledge.

Payload Schema:

{
  "id": "STIDX001-posts.article.vector-search-tools",
  "title": "C++ Search Tool CLI — Ripgrep + Mind Index",
  "slug": "vector-search-tools",
  "path": "posts/integrations/vector-search-tools.md",
  "body": "# C++ Search Tool CLI...\n\n## Installation\n```bash\ncmake -S . -B build\nmake -C build\n```",
  "tags": ["cpp", "search", "ripgrep", "mind", "cli"],
  "status": "published",
  "author": "cipher",
  "published_at": "2026-08-14T10:30:00Z"
}

Query Example:

POST /v1/vectors/search
{
  "collection": "posts",
  "vector": <embedding("search tool cli cpp")>,
  "limit": 5,
  "score_threshold": 0.65
}

knowledgebase — Structured Domain Data

Purpose: Normalized exports from The Mind SDK — sports odds schemas, betting market mappings, agent roster specs.

Payload Schema:

{
  "id": "STIDX001-knowledgebase.schema.sports-betting-slate",
  "title": "Sports Betting Advisory — Slate Schema v2.1",
  "slug": "sports-betting-slate-schema",
  "path": "knowledgebase/schemas/sports-betting-slate.json",
  "body": "{\n  \"slate_id\": \"string\",\n  \"sport\": \"string\",\n  \"events\": [{\"event_id\": \"\", \"odds_source\": \"\"}],\n  \"risk_model\": {\"kelly_fraction\": 0.25}\n}",
  "tags": ["schema", "betting", "polymarket", "slate"],
  "status": "published",
  "version": "2.1",
  "updated_at": "2026-08-10T08:00:00Z"
}

almemsha — Deep Research Corpus

Purpose: ArXiv digests, model benchmark tables, HuggingFace model cards, training run logs. 3,891 records.

Payload Schema:

{
  "id": "STIDX001-almemsha.paper.llama-3-405b-benchmark",
  "title": "Llama 3 405B — MMLU 89.2, GSM8K 94.1",
  "slug": "llama-3-405b-benchmark",
  "path": "almemsha/papers/llama-3-405b-benchmark.md",
  "body": "## MMLU Results\n| Model | 5-shot |\n|-------|--------|\n| Llama 3 405B | 89.2 |\n| GPT-4o | 88.7 |\n\n## GSM8K\n| Model | Pass@1 |\n|-------|--------|\n| Llama 3 405B | 94.1 |",
  "tags": ["benchmark", "llama-3", "405b", "mmlu", "gsm8k"],
  "status": "published",
  "source": "arxiv",
  "arxiv_id": "2407.xxxxx",
  "model_name": "meta-llama/llama-3-405b",
  "published_at": "2024-07-15T00:00:00Z"
}

Hybrid Search Strategy

ScenarioEndpointRationale
"Find notes about refusal detection"search (Qdrant)Speed, semantic relevance
"All resolved notes by cipher since 2026-01-01"search-pg (pgvector)Exact SQL filter + cosine
"Top 10 posts similar to this vector"search (Qdrant)Pure vector similarity
"Knowledgebase schemas tagged betting"search-pgTag filter + metadata sort

Go Client Pattern:

func HybridSearch(q string, filters map[string]string) []Record {
    // 1. Compact vector search
    hits := search.Search(q, 20)
    
    // 2. If filters provided, re-query pgvector with exact WHERE
    if len(filters) > 0 {
        sql := buildWhere(filters)  // e.g., "status='resolved' AND agent_id='cipher'"
        hits = gateway.SearchPG(hits.QueryVector, sql, 20)
    }
    
    // 3. Hydrate only top-N
    var out []Record
    for i := 0; i < min(3, len(hits)); i++ {
        out = append(out, gateway.Get(hits[i].Collection, hits[i].ID))
    }
    return out
}

Index Management

Rebuild Command:

# Full rebuild from Mind CMS
cd /root/agent_dev/search-tool
./build/search-tool sync-directus --config /root/themind-sdk/.env

# Incremental (new/modified only)
./build/search-tool sync-directus --since 2026-08-01

Stats:

./build/search-tool stats the-mind.idx --json
# {"records": 269, "index_size_mb": 12.4, "build_ms": 847, "collections": 4}

Binary Format: STIDX001 header → record headers → compressed vectors → Payload offsets