Context Engine

The Context Engine manages the full lifecycle of knowledge for production agents: ingest documents (text, files, URLs, directories), chunk and embed them, store chunks in vector and graph databases, and retrieve relevant context using multi-strategy search. Use it when you need document management, scoped access control, tag-based filtering, deduplication, and lifecycle policies — capabilities that the lower-level VectorMemory and GraphMemory primitives do not provide.

The Context Engine implements the MemoryProvider protocol, so it plugs directly into any agent via memory=engine.

Prerequisites: Memory & RAG · Next: Directives · Safety

Architecture

Documents / Text / URLs / Directories
    |
    v
IngestionPipeline
    |
    +---> Docling (PDF, DOCX, HTML, images)
    +---> tree-sitter (source code: Python, JS, Go, ...)
    +---> URL fetcher (with SSRF protection)
    |
    v
Chunking (recursive, with semantic dedup)
    |
    v
Embedding (OpenAI, configurable)
    |
    +---> Vector Store (Milvus or in-memory)
    +---> Metadata Store (Postgres or in-memory)
    +---> Graph Store (NebulaGraph, optional)
    |
    v
Multi-Strategy Retrieval
    +---> Vector search (semantic similarity)
    +---> BM25 search (keyword matching)
    +---> Graph search (relationship traversal)
    |
    v
Reciprocal Rank Fusion (RRF) merge
    |
    v
Optional cross-encoder re-ranking
    |
    v
Deduplicated, scored results

Quick Start

import asyncio
from sagewai import UniversalAgent, ContextEngine, ContextScope
from sagewai.context import InMemoryMetadataStore, InMemoryVectorStore

async def main():
    # 1. Create a context engine (in-memory for development)
    engine = ContextEngine(
        metadata_store=InMemoryMetadataStore(),
        vector_store=InMemoryVectorStore(),
        project_id="my-project",
    )

    # 2. Ingest knowledge
    await engine.ingest_text(
        text="Sagewai supports 100+ LLM providers through LiteLLM.",
        title="About Sagewai",
        scope=ContextScope.PROJECT,
        scope_id="my-project",
    )

    # 3. Create an agent with context-aware memory
    agent = UniversalAgent(
        name="assistant",
        model="gpt-4o",
        memory=engine,                # automatic RAG retrieval
        tools=engine.get_tools(),     # self-editing memory tools
    )

    response = await agent.chat("What LLM providers does Sagewai support?")
    print(response)

asyncio.run(main())

Ingestion

The Context Engine accepts content from several sources. Each source runs through the same chunking, embedding, and deduplication pipeline before storage.

Text

from sagewai.context.models import ContextSource

doc = await engine.ingest_text(
    text="Your document content here.",
    title="My Document",
    scope=ContextScope.PROJECT,
    scope_id="my-project",
    source=ContextSource.MANUAL,
    metadata={"author": "Alice", "department": "Engineering"},
)

Files

Supported formats: PDF, DOCX, HTML, Markdown, images (via Docling), and source code (via tree-sitter).

doc = await engine.ingest_file(
    file_data=open("report.pdf", "rb").read(),
    filename="report.pdf",
    scope=ContextScope.PROJECT,
    scope_id="my-project",
)

URLs

Fetches and parses web pages with built-in SSRF protection. If you re-ingest the same URL, the engine auto-supersedes the old version rather than creating a duplicate.

doc = await engine.ingest_url(
    url="https://docs.example.com/api-reference",
    scope=ContextScope.PROJECT,
    scope_id="my-project",
)

Directories

Recursively indexes a directory tree, respecting .gitignore, .claudeignore, and other ignore files. Binary files, .git, node_modules, and files over 50 MB are skipped automatically.

docs = await engine.ingest_directory(
    path="/path/to/codebase",
    scope=ContextScope.PROJECT,
    scope_id="my-project",
)

Scopes

Every document belongs to one of two access scopes. Scopes control which agents can see which content.

When an agent searches, it queries both its project scope and the organization scope. Project-scoped results take priority over org-scoped results during deduplication.

from sagewai import ContextScope

# Org-level: visible to all projects
await engine.ingest_text(
    text="Company vacation policy: 25 days per year.",
    title="HR Policy",
    scope=ContextScope.ORG,
    scope_id="acme-corp",
)

# Project-level: visible only to this project
await engine.ingest_text(
    text="Project Alpha uses React and FastAPI.",
    title="Tech Stack",
    scope=ContextScope.PROJECT,
    scope_id="project-alpha",
)

Tags

Tags are stored as a list of strings on each document. The storage uses a PostgreSQL TEXT[] column with a GIN index so tag lookups stay fast at scale.

# Ingest with tags
await engine.ingest_text(
    text="Q4 revenue was $12M, up 15% YoY.",
    title="Q4 Financial Summary",
    scope=ContextScope.PROJECT,
    scope_id="my-project",
    metadata={"tags": ["finance", "q4", "revenue"]},
)

# Search with tag filtering — only returns documents matching these tags
results = await engine.search(
    "quarterly revenue",
    tags=["finance", "q4"],
)

Tag filtering happens at the document level, before chunk retrieval. A chunk only enters search results if its parent document matches all requested tags.

Multi-Strategy Retrieval

search() runs three strategies in parallel and merges their results. Each strategy has different strengths, so the combination outperforms any single approach.

results = await engine.search(
    "How does the billing system work?",
    top_k=5,
    scopes=[ContextScope.PROJECT],
    tags=["billing"],
)

for result in results:
    print(f"Score: {result.score:.3f} | {result.content[:100]}")

Strategy Details

Results merge via Reciprocal Rank Fusion (RRF), which combines ranked lists without needing score normalization across strategies. You can add a cross-encoder re-ranker for higher precision when latency allows.

Self-Editing Memory Tools

The Context Engine exposes four tools that let agents manage their own knowledge during a conversation:

agent = UniversalAgent(
    name="analyst",
    model="gpt-4o",
    memory=engine,
    tools=engine.get_tools(),  # memory_store, memory_search, memory_forget, memory_update
)

The agent calls these tools at its own discretion — deciding what to remember, what to drop, and when to update a stale fact.

Auto-Learn

Set auto_learn=True to have the agent automatically extract and store facts from every conversation via the MemoryBridge. No explicit tool calls needed.

agent = UniversalAgent(
    name="analyst",
    model="gpt-4o",
    memory=engine,
    tools=engine.get_tools(),
    auto_learn=True,  # auto-extracts facts from conversations
)

Production Stores

The default in-memory stores need no infrastructure and are the right choice for development. For production, use Postgres for metadata and Milvus for vectors:

from sagewai.context import (
    ContextEngine,
    PostgresContextStore,
    MilvusContextVectorStore,
)
from sagewai.memory.nebula import NebulaGraphMemory

engine = ContextEngine(
    metadata_store=PostgresContextStore(database_url="postgresql://localhost/sagewai"),
    vector_store=MilvusContextVectorStore(
        uri="http://localhost:19530",
        collection="context_chunks",
    ),
    graph_store=NebulaGraphMemory(
        space="knowledge",
        hosts="127.0.0.1:9669",
    ),
    project_id="my-project",
    embedding_model="text-embedding-3-small",
)

Production stores require the sagewai[memory] and sagewai[postgres] extras.

Lifecycle Management

Documents age out automatically when you configure a LifecycleConfig. Lifecycle actions trigger based on document age and importance scores. Re-ingesting a document auto-supersedes the previous version through the conflict detection system.

from sagewai.context import LifecycleConfig

engine = ContextEngine(
    metadata_store=...,
    vector_store=...,
    lifecycle_config=LifecycleConfig(
        compress_after_days=30,
        archive_after_days=90,
        discard_after_days=365,
    ),
)

What's Next

• Directives — Use @context('query') to pull documents inline in prompts
• Memory — Lower-level vector and graph memory primitives
• Safety — HallucinationGuard validates responses against retrieved context