Skip to content

Search Systems and Retrieval

The HLD Search suite provides a multi-layered retrieval architecture for autonomous agents, spanning local file-based knowledge, internal CMS data (The Mind), and external web intelligence. The system prioritizes speed and determinism over probabilistic vector retrieval.

Used for instant retrieval of fleet notes, scripts, and documentation stored within the agent's local environment.

  • Engine: ripgrep (rg)
  • Methodology: High-speed full-text substring matching.
  • Performance: Sub-50ms execution across thousands of documents.

Implementation Example

The tool invokes a ripgrep subprocess to ensure minimal overhead and zero indexing latency.

import subprocess

def search_local(query: str, path: str = "."):
    \"\"\"Execute ripgrep against the local knowledgebase.\"\"\"
    cmd = ["rg", "--json", "-i", query, path]
    result = subprocess.run(cmd, capture_output=True, text=True)

    # Parse JSON output for exact line-snippet context
    matches = [json.loads(line) for line in result.stdout.splitlines()]
    return matches

Queries the sovereign HLD knowledge graph (The Mind). This system utilizes native SQL LIKE operations for exact property filtering.

  • API: The Mind REST API
  • Operators: _icontains (case-insensitive substring) and _or (multi-field search).

Query Logic

The search targets the subject, body, and tags fields within a single transaction.

params = {
    "filter": {
        "_or": [
            {"subject": {"_icontains": query}},
            {"body": {"_icontains": query}},
            {"tags": {"_icontains": query}}
        ]
    },
    "fields": ["id", "subject", "body", "status"]
}

response = requests.get(f"{MIND_API_URL}/items/fleet_notes", params=params)

The external search tool provides real-time internet intelligence through a primary API provider with a resilient fallback mechanism.

Provider Stack

  1. Primary: Brave Search API — Provides high-signal, clean JSON responses for agent consumption.
  2. Fallback: DuckDuckGo Scraper — Triggered automatically if the primary API fails or hits rate limits.

Resilient Retrieval Logic

async def web_search(query: str):
    if BRAVE_SEARCH_API_KEY:
        try:
            return await brave_search(query)
        except Exception as e:
            log.warning(f"Brave Search failed, falling back to DDG: {e}")

    return await duckduckgo_fallback(query)

async def brave_search(query: str):
    headers = {"X-Subscription-Token": BRAVE_SEARCH_API_KEY}
    url = f"https://api.search.brave.com/res/v1/web/search?q={query}"
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, headers=headers)
        return resp.json()["web"]["results"]

Retrieval Philosophy

Unlike traditional RAG (Retrieval-Augmented Generation) systems that rely on potentially halluncinatory embedding models, the HLD Search suite is built on Deterministic Retrieval.

By using raw substring matching and native database filters, we guarantee that if a piece of information exists, it is retrieved exactly as written, with the precise context required for autonomous execution.