Vector Database Gateway

The Vector Database Gateway provides Qdrant and PostgreSQL pgvector as managed vector search infrastructure. Agents authenticate via the Firebase Handshake, receive a gateway API key, and issue vector operations through the REST endpoint at https://vector.highlimitdesigns.com.

Architecture

Agent Runtime          Gateway Layer              Storage Layer
─────────────          ─────────────              ─────────────
CyBot / Cipher  ──┐   ┌──────────────────┐   ┌──────────────────┐
   │              │   │  nginx TLS       │   │  Qdrant (HNSW)   │
   │  Handshake   ├──→│  Rate Limiter    ├──→│  Collections:    │
   │  (Ed25519)   │   │  Auth Verifier   │   │  - fleet_notes   │
   │              │   │  Router          │   │  - posts         │
   │  API Key     │   │  (FastAPI)       │   │  - knowledgebase │
   ├──→──────────┤   └──────────────────┘   │  - almemsha      │
   │              │           │             │  HNSW: M=16, ef=128│
   │  Vector Ops  │           ├─────────────┤  1536-dim, COSINE │
   │  - Upsert    │           │             └──────────────────┘
   │  - Search    │           │             ┌──────────────────┐
   │  - Scroll    │           └────────────►│  PostgreSQL      │
   │  - Delete    │                         │  pgvector        │
   │  - Count     │                           │  Tables:       │
   └──────────────┘                           │  - embeddings  │
                                              │  - metadata    │
                                              │  HNSW idx      │
                                              └──────────────────┘

Gateway Endpoint: https://vector.highlimitdesigns.com/v1/ Admin Console: https://studio.highlimitdesigns.com/vectors Search CLI Binary: /root/agent_dev/search-tool/build/search-tool (C++17, STIDX001 index)

Agent Onboarding

1. Provision Agent (CMS Studio)

# Human operator action in CMS Studio Console
Provision Agent → agent_id: "my-research-agent"
                 → role: "Knowledge Worker"
                 → quota: 10000 ops/day

Generates Ed25519 keypair. Public key fingerprint stored in The Mind. Private key written to agent's sovereign volume.

2. Agent Handshake (Automatic at Runtime)

// CyBot runtime - runs on agent start
challenge := client.Post("/v1/auth/handshake", map[string]string{
    "agent_id": "my-research-agent",
    "nonce":    crypto.RandHex(32),
})
// Gateway returns: server_nonce, public_key_fingerprint

sig := ed25519.Sign(privKey, challenge.ClientNonce + challenge.ServerNonce)

tokenResp := client.Post("/v1/auth/verify", map[string]string{
    "agent_id":  "my-research-agent",
    "signature": hex.EncodeToString(sig),
})
// Returns: { "api_key": "vk_live_abc123...", "expires_in": 3600 }

3. Use API Key on All Requests

Authorization: Bearer vk_live_abc123...
Content-Type: application/json

Vector Operations

Upsert Vectors (Qdrant)

POST /v1/vectors/upsert
{
  "collection": "fleet_notes",
  "points": [
    {
      "id": "STIDX001-fleet_notes.negative.diff-in-means-d2f3",
      "vector": [0.023, -0.145, ... 1536 dims ...],
      "payload": {
        "title": "OBLITERATUS: Diff-in-Means Refusal Detection",
        "collection": "fleet_notes",
        "status": "resolved",
        "tags": ["evaluation", "refusal", "statistical"],
        "mind_path": "fleet_notes/STIDX001-fleet_notes.negative.diff-in-means-d2f3.md",
        "updated_at": "2026-01-15T14:22:00Z"
      }
    }
  ]
}

Search (Qdrant)

POST /v1/vectors/search
{
  "collection": "fleet_notes",
  "vector": [0.023, -0.145, ...],
  "limit": 10,
  "score_threshold": 0.72,
  "filter": { "must": [{ "key": "status", "match": { "value": "resolved" } }] }
}

Response (23ms p95):

{
  "results": [
    {
      "id": "STIDX001-fleet_notes.negative.diff-in-means-d2f3",
      "score": 0.847,
      "payload": { "title": "...", "collection": "fleet_notes", ... }
    }
  ],
  "took_ms": 23,
  "records_scanned": 269
}

Search (PostgreSQL pgvector)

POST /v1/vectors/search-pg
{
  "table": "embeddings",
  "vector": [0.023, -0.145, ...],
  "limit": 10,
  "metric": "cosine",
  "filter_sql": "collection = 'fleet_notes' AND status = 'resolved'"
}

Scroll / Paginate (Qdrant)

POST /v1/vectors/scroll
{
  "collection": "fleet_notes",
  "limit": 50,
  "offset": "STIDX001-fleet_notes.negative.diff-in-means-d2f3",
  "filter": { "must": [{ "key": "status", "match": { "value": "active" } }] }
}

Count

GET /v1/vectors/count?collection=fleet_notes&filter={"must":[{"key":"status","match":{"value":"active"}}]}

Delete

DELETE /v1/vectors/delete
{
  "collection": "fleet_notes",
  "ids": ["STIDX001-fleet_notes.negative.diff-in-means-d2f3"]
}

Collections Schema

Collection	Dimensions	Vectors	Metric	Source
`fleet_notes`	1536	269	COSINE	The Mind CMS (`fleet_notes` + `posts`)
`posts`	1536	1,247	COSINE	Blog posts, articles
`knowledgebase`	1536	412	COSINE	The Mind SDK exports
`almemsha`	1536	3,891	COSINE	ALMEMSHA research corpus

Index Format: STIDX001 magic header, weighted scoring (title 60 / slug 40 / path 35 / body 15)

Python SDK (The Mind SDK)

from themind import VectorClient

client = VectorClient(
    base_url="https://vector.highlimitdesigns.com",
    api_key="vk_live_abc123..."  # from handshake
)

# Search Qdrant
results = client.search(
    collection="fleet_notes",
    query_vector=[0.023, -0.145, ...],
    limit=10,
    score_threshold=0.72,
    filters={"status": "resolved"}
)

# Search PostgreSQL
results = client.search_pg(
    table="embeddings",
    query_vector=[0.023, -0.145, ...],
    limit=10,
    filter_sql="collection = 'fleet_notes'"
)

# Upsert
client.upsert(
    collection="fleet_notes",
    points=[{
        "id": "STIDX001-new-record",
        "vector": [...],
        "payload": {"title": "New Note", "status": "active", "tags": ["research"]}
    }]
)

cURL Examples

# Search Qdrant
curl -X POST https://vector.highlimitdesigns.com/v1/vectors/search \
  -H "Authorization: Bearer vk_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{"collection":"fleet_notes","vector":[0.023,-0.145,...],"limit":5}'

# Search PostgreSQL
curl -X POST https://vector.highlimitdesigns.com/v1/vectors/search-pg \
  -H "Authorization: Bearer vk_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{"table":"embeddings","vector":[0.023,-0.145,...],"limit":5,"metric":"cosine"}'

# Upsert
curl -X POST https://vector.highlimitdesigns.com/v1/vectors/upsert \
  -H "Authorization: Bearer vk_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{"collection":"fleet_notes","points":[{"id":"STIDX001-test","vector":[...],"payload":{"title":"Test","status":"active"}}]}'

# Count
curl "https://vector.highlimitdesigns.com/v1/vectors/count?collection=fleet_notes" \
  -H "Authorization: Bearer vk_live_abc123..."

Search Tool CLI (C++17 Binary)

The /root/agent_dev/search-tool/build/search-tool binary provides local-first search for agents:

# Index from JSONL (The Mind exports)
./search-tool index-jsonl --input /root/themind-sdk/fleet_notes.jsonl --output /root/themind-sdk/the-mind.idx

# Search local index (23ms, 269 records)
./search-tool search "blog manager agent" --limit 5 --json

# Get single record by STIDX ID
./search-tool get STIDX001-fleet_notes.negative.diff-in-means-d2f3 --json

# Stats
./search-tool stats the-mind.idx --json

Integration Pattern: Agent queries local index first (zero tokens), then calls /v1/vectors/get only for the specific records needed.

Rate Limits & Quotas

Tier	Daily Ops	Burst	Concurrent
Agent (default)	10,000	50/s	5
Research Agent	50,000	200/s	20
Infra Architect	Unlimited	Unlimited	Unlimited

Configured in CMS Studio → Agent Provisioning → Quota & Rate Limits.

Monitoring & Observability

Endpoint	Purpose
`GET /v1/health`	Gateway liveness (nginx + FastAPI + Qdrant + PG)
`GET /v1/stats`	Collection counts, index size, index status, p95 latency
`GET /v1/debug/agent/{agent_id}`	Agent's active sessions, recent ops, quota usage

Admin Dashboard: studio.highlimitdesigns.com/vectors — live search testing, collection mgmt, agent quota tweaks.

Security Notes

No passwords — agents prove key possession via Ed25519 challenge/response
Short-lived API keys — 1hr default, rotatable via CMS Studio
Instant revocation — delete agent in CMS → all future handshakes fail
Per-agent enforcement — rate limits & quotas at gateway layer
Audit trail — every handshake + vector op logged to fleet_notes with tags ["auth", "vector", "agent_id"]

Authentication (Firebase Handshake) — Agent provisioning, handshake flow, dashboard
Search Tool CLI — Local C++ search binary, STIDX001 index format
The Mind SDK — Go broker for Directus CMS
Search Systems — ripgrep + Mind + web search for agents
Webhooks — Agent completion events to HTTPS endpoints

Overview Collections

Overview

Vector Database Gateway

Vector Database Gateway

Architecture

Agent Onboarding

1. Provision Agent (CMS Studio)

2. Agent Handshake (Automatic at Runtime)

3. Use API Key on All Requests

Vector Operations

Upsert Vectors (Qdrant)

Search (Qdrant)

Search (PostgreSQL pgvector)

Scroll / Paginate (Qdrant)

Count

Delete

Collections Schema

Python SDK (The Mind SDK)

cURL Examples

Search Tool CLI (C++17 Binary)

Rate Limits & Quotas

Monitoring & Observability

Security Notes

Related