Document intelligence & RAG

Retrieval-augmented generation (RAG) lets agents answer questions from private documents instead of relying solely on model training data. Alquimia Platform treats document retrieval as a first-class part of the agent execution pipeline, with multiple access patterns and full observability.

What the use case looks like

A legal analyst asks the agent: “What are the termination clauses in the Acme contract?” The agent:

1. Receives the question and classifies intent.
2. Searches the registered contract topic for relevant sections.
3. Returns an answer with citations to the source paragraphs.
4. Records the retrieval sources in the worklog for audit.

Registry-first knowledge lifecycle

In Alquimia Platform, knowledge is curated in the registry before an agent can consume it. A TopicRegistration is a stable handle; files are ingested into topics; and the vector-store backend is a deployment detail.

# Create a topic for contract documents
alquimia registry topics add contracts \
  --severity read-only \
  --tier-grants reader,editor,operator

# Ingest a contract: chunk, embed, and add to the topic catalogue
alquimia registry topics add-file contracts ./contracts/acme-2024.pdf \
  --description "Acme master service agreement" \
  --tags legal,acme,2024 \
  --hydrate

# Rebuild the vector index after bulk ingestion or migration
alquimia registry topics rebuild contracts

The agent spec only references the topic_id:

{
  "assistant_id": "legal-researcher",
  "role": "reader",
  "response": {
    "provider_id": "alquimia",
    "profile": {
      "system_prompt": "You are a legal research assistant. Answer only from the contract topic and cite sources.",
      "knowledge_base": [
        {
          "topic_id": "contracts",
          "search_mode": "on_demand",
          "search_kwargs": {"k": 5},
          "doc_template": "Source: {{document.metadata.source}}\nContent: {{document.page_content}}"
        }
      ],
      "evaluation_strategy": {"evaluation_strategy_id": "one-shoot"}
    },
    "config": {
      "provider_id": "openai",
      "params": {
        "model": "gpt-4o",
        "temperature": 0.1,
        "api_key": {"$secretRef": "RESPONSE_PROVIDER_API_KEY"}
      }
    }
  }
}

Tip

ALQUIMIA_KNOWLEDGE_PROVIDER lets the same topic_id resolve to qdrant in production, redis in staging, or in_memory in tests without changing the spec or the topic registration.

Retrieval patterns

Pattern	When to use
Automatic injection (rag)	Every question should include relevant context, such as support FAQs.
On-demand search (on_demand)	The agent should decide when to search, such as legal research.
Direct file access (direct)	The agent needs to list and read specific files, such as code repositories.
Boltzmann Brain (brain)	Query a specialized reasoning module with fine-grained authorization.

Knowledge access policy and content shields

Topics enforce access through a TopicAccessPolicy. A reader agent cannot query a topic whose severity is mutating-recoverable, and a destructive topic always forces approval regardless of tier grants.

{
  "topic_id": "contracts",
  "agentspace_id": "legal",
  "access_policy": {
    "severity": "read-only",
    "tier_grants": ["reader", "editor", "operator"],
    "approval_required": false
  },
  "shields": {
    "confidentiality": {
      "provider_id": "shield-config",
      "connector": {
        "provider_id": "alquimia",
        "profile": {
          "system_prompt": "Check if the retrieved text contains 'Attorney-Client Privileged' or 'Confidential Work Product'. Output JSON: {\"contains_privileged\": true|false}",
          "evaluation_strategy": {
            "evaluation_strategy_id": "one-shoot",
            "structured_output": {
              "method": "json_schema",
              "json_schema": {
                "type": "object",
                "properties": {"contains_privileged": {"type": "boolean"}},
                "required": ["contains_privileged"]
              }
            }
          }
        },
        "config": {"model_ref": "gpt-4o-mini-classifier"}
      },
      "action": "block",
      "threshold": 0.8,
      "block_message": "Privileged content removed from retrieval result."
    }
  }
}

When the content shield blocks, the retrieved chunk is replaced with the block_message and a shield.blocked.v1 event is recorded.

Boltzmann Brain lifecycle

For high-stakes knowledge that must be provable and versioned, a Boltzmann Brain stores content as an immutable, content-addressed OCI artifact with a Merkle provenance ledger.

# Register the brain
alquimia registry brains add fourier-signals \
  --oci-reference ghcr.io/acme/brains/fourier-signals \
  --tag v1

# Grant access per memory module
alquimia registry brains set-policy fourier-signals semantic \
  --severity read-only --tier-grants reader,editor,operator
alquimia registry brains set-policy fourier-signals canonical \
  --severity destructive --tier-grants operator

# Ingest new evidence through the CLI gate (not by agents)
alquimia registry brains ingest fourier-signals ./evidence/lecture07.txt

# Publish a new version
alquimia registry brains publish fourier-signals --tag v2

Retirement operations include:

• drop — remove a block and rebuild the composition over the survivors. Dropping canonical evidence cascades to everything derived from it.
• supersede — mark a block as replaced by a newer block (append-only, suitable for episodic memory).
• redact — destroy bytes for legal or safety reasons while keeping the provenance ledger intact.

OCI distribution for knowledge

A complete agentspace — agents, topics, files, tool registrations, and model references — can be published as a signed OCI artifact. Pulling it into a new environment recreates the knowledge catalogue and can rehydrate vector indexes from the registered files.

curl -X PUT "http://runtime:8080/registry/publish?agentspace_id=legal&tag=v1.0.0" \
  -H "Authorization: Bearer $API_TOKEN"

curl -X PUT "http://runtime:8080/registry/pull?agentspace_id=legal&tag=v1.0.0" \
  -H "Authorization: Bearer $API_TOKEN"

The artifact excludes secret values by default. Secrets are resolved at inference time from Vault or environment variables in the target environment.

For application teams

Citable answers

Retrieved chunks can be attached to responses so users can verify claims against source documents.

Multiple sources

Combine contracts, manuals, wikis, and structured databases in a single agent configuration.

Source-aware prompts

Control how retrieved context is inserted into the system prompt and how much context is kept.

For platform / SRE teams

Pluggable vector stores

Swap Qdrant, Redis, or in-memory stores at deployment time without changing agent specs.

Usage attribution

Track retrieval volume, token consumption, and answer quality per knowledge base and agent.

Access control

Restrict which agents and users can query sensitive topics through agentspace and role scoping.

Common integration points

System	Role
Object store (S3/MinIO)	Store raw documents and extracted blobs
Vector database	Index and search document chunks
Document pipeline	Convert PDFs, Office files, and HTML into chunks
Identity provider	Enforce topic-level access controls

Example outcomes

• Faster self-service access to policies, contracts, and product documentation.
• Reduced risk of outdated or fabricated answers.
• Auditable trace from question to source document.

Related pages

• Use cases overview
• Memory & context
• Agents & configuration
• Observability