Skip to content

Multi-Vector Retrieval Architecture

This page systematizes Eagle-RAG's multi-vector retrieval architecture: how a single query fans out to multiple (collection, encoder) embedding spaces, how hybrid dense+sparse runs within a space, and how RRF rank-fusion merges across spaces. It is the upstream of Evidence aggregation — multi-vector is the retrieval-time multi-space fan-out, evidence aggregation is the post-retrieval consolidation.

Relationship to existing docs

This page is the concept canonical. Per-function implementation details live in Retrieval and Router engine; the architectural decision is ADR-004 Multi-Encoder RRF Fusion; how plugins register encoders/collections is in Plugin architecture.

1. Definition

Multi-vector retrieval means a single query is executed against multiple embedding spaces simultaneously — each space has its own vector dimension, encoder, and modality — and results are fused across spaces. This contrasts with naive single-encoder, single-collection, single-space RAG.

Eagle-RAG's multi-vector form, in one sentence:

A single query fans out to multiple (collection, encoder) embedding spaces → optional hybrid dense+sparse within a space → RRF (Reciprocal Rank Fusion, rank-only) across spaces, never mixing raw scores across embedding spaces.

Three key constraints (from ADR-004):

  1. Raw scores from different encoders are incomparable (different dims, distributions, normalization) → across spaces only ranks are compared, never scores.
  2. Hybrid sparse is a within-space enhancement, re-ranking dense ANN hits of one collection by lexical term overlap — not a cross-space operation.
  3. Core default routing never auto-queries specialized collections (G4) — only domain plugins explicitly route into domain-specific spaces.

2. Three layers

2.1 Space layer: multiple collections

Each plugin_namespace (= one Milvus Database) holds a set of collections, each an independent embedding space:

Collection Dim Encoder Modality Owner
eagle_text 1536 Qwen text-embedding-v4 text Core default
eagle_visual 2048 Qwen3-VL-Embedding-2B visual Core default
eagle_text_biomed 768 PubMedBERT text biomed domain
eagle_text_medcpt 768 MedCPT text biomed domain
eagle_chemical 512 MolFormer text biomed domain
eagle_medical_radiology 1024 MedImageInsight visual biomed domain
eagle_medical_pathology 1024 UNI2 visual biomed domain

Code: collection metadata is registered as CollectionProfile(dim, default_encoder, hybrid_enabled, extra_output_fields) into EncoderRegistry (eagle_rag/plugins/encoder_registry.py L32-39). Core default collections are registered in eagle_rag/plugins/core_defaults.py (with hybrid_enabled=True, L106).

2.2 Encoder layer: EncoderRegistry

EncoderRegistry is the single source of truth for the encoder↔collection dimension contract:

# eagle_rag/plugins/encoder_registry.py
class EncoderRegistry:
    def register(name, encoder, *, dim, modality="text") -> None: ...
    def register_collection(collection, *, dim, default_encoder=None,
                            hybrid_enabled=False, extra_output_fields=()) -> None: ...
    def validate_plan(self, collection: str, encoder_name: str) -> None:
        # rerank encoders skip; otherwise encoder.dim must equal collection.dim
        ...

Key contract: validate_plan(collection, encoder) enforces encoder.dim == collection.dim before writes (L111-118), raising ValueError on mismatch. This guards against silent errors like "writing a 2048-d vector into a 1536-d collection."

At query time: EncoderRegistry.collection_profile(collection) returns the space's default_encoder / hybrid_enabled / extra_output_fields, so RetrieverOrchestrator knows which encoder to encode the query with, whether to trigger hybrid, and which output fields to read.

2.3 Fusion layer: within-space hybrid + cross-space RRF

flowchart LR
    Q["query"] --> ROUTE["QueryRouteClassifier.route()"]
    ROUTE --> P1["Plan 1<br/>(eagle_text, qwen-emb-v4)"]
    ROUTE --> P2["Plan 2<br/>(eagle_text_biomed, pubmedbert)"]
    ROUTE --> P3["Plan 3<br/>(eagle_visual, qwen3-vl)"]

    P1 --> ANN1["dense ANN"]
    P2 --> ANN2["dense ANN"]
    P3 --> ANN3["dense ANN"]

    ANN1 --> HY1["hybrid_fuse_dense_sparse<br/>(alpha·dense + (1-alpha)·sparse)"]
    ANN2 --> HY2["hybrid_fuse_dense_sparse"]
    ANN3 --> HY3["(visual: no sparse)"]

    HY1 --> RRF["merge_rrf<br/>rank-only fusion"]
    HY2 --> RRF
    HY3 --> RRF
    RRF --> DEDUP["dedupe_cross_collection<br/>source_chunk_id / (doc_id,path)"]
    DEDUP --> SUPP["RRF_POST_MERGE hook<br/>candidate injection"]
    SUPP --> RR["rerank_merged<br/>RERANK_MERGED / qwen3-rerank"]
    RR --> OUT["top_n evidence"]

Within-space hybrid (eagle_rag/retrievers/hybrid_text_retriever.py L73-104):

def hybrid_fuse_dense_sparse(dense_nodes, query, *, alpha=0.6,
                             extra_sparse_terms=None, rrf_k=60):
    # 1. rank dense ANN hits by lexical term overlap (sparse)
    sparse_nodes = sparse_rank_nodes(dense_nodes, sparse_query, extra_terms=...)
    # 2. alpha-weighted fusion (alpha=1 pure dense, alpha=0 pure sparse)
    combined = alpha * dense_score + (1 - alpha) * sparse_score

Activation (retriever_orchestrator.py L427-434, L519-522): router.hybrid_text_enabled and the collection is in router.hybrid_text_collections, or CollectionProfile.hybrid_enabled, or is the default text collection. Visual collections do not run sparse (no terms).

Cross-space RRF (eagle_rag/router/rerank_fusion.py L37-62): merge_rrf fuses per-plan results by 1/(k+rank); empty result sets contribute no phantom rank (G8). This is the only legal cross-space fusion — raw scores are incomparable across encoders.

Cross-space dedupe (rerank_fusion.py L65-93): dedupe_cross_collection collapses duplicate logical chunks by source_chunk_id or (document_id, path), keeping the higher-ranked (G32), and records an rrf_dedupe audit event.

3. Query routing data structures

The multi-vector fan-out "plan" is plugin-pluggable:

# eagle_rag/plugins/routing.py
@dataclass(frozen=True)
class CollectionQueryPlan:
    collection: str
    encoder: str
    top_k: int = 5

@dataclass(frozen=True)
class QueryRouteDecision:
    plans: tuple[CollectionQueryPlan, ...]   # one query may carry many plans = multi-space fan-out
    retrieval_hints: dict[str, Any] = ...

Dense/sparse query expansion: domain plugins may subscribe to the QUERY_DENSE_EXPAND hook and return ExpandedQuery(dense_query, sparse_terms, intent) (routing.py L37-43).

  • biomed example: UMLS entity recognition rewrites the dense query (synonym expansion) and emits sparse_terms (drug names, MeSH terms) to boost lexical recall.
  • RetrieverOrchestrator._retrieve_generic_milvus (L464-479) calls the hook first to get expanded, then encodes dense_query and passes sparse_terms through to hybrid_fuse_dense_sparse.

4. Key design contracts

Contract Source Meaning
RRF is the only legal cross-space fusion ADR-004 G8 Raw scores from different encoders are incomparable; across spaces only ranks
Hybrid sparse is a within-space enhancement hybrid_text_retriever.py Lexical overlap re-ranks one collection's dense ANN hits, not cross-space
Core never auto-queries specialized collections ADR-004 G4 Core QueryRouteClassifier never fans out to domain collections, only eagle_text(+eagle_visual)
validate_plan guards dim consistency encoder_registry.py L111-118 Enforces encoder.dim == collection.dim before writes
Single-plan failure is best-effort skipped ADR-004 G14 A space raising an exception returns [], not failing the whole query

Relation to single-domain deployment: cross-industry isolation uses multiple instances (one Milvus Database per plugin_namespace); within one DB a single query may span multiple collections (ADR-002, Multi-tenancy). Multi-vector is "multiple spaces within one domain," not "cross-domain fan-out."

5. Relationship to multimodal fusion

Multimodal fusion is the multi-vector architecture specialized to the modality dimension: eagle_text (1536) and eagle_visual (2048) are two embedding spaces of different dimensions; text nodes and visual tiles enter their own collections via their own encoders, queries fan out across both spaces, RRF fuses, and the VLM consumes both in one prompt. The multi-vector architecture generalizes this pattern to "any number of domain-specific encoder spaces."

6. Roadmap (not implemented)

The following are unimplemented design gaps, not current capabilities.

  • Per-collection / per-query adaptive hybrid alpha: today router.hybrid_alpha is a global value (the alpha of hybrid_fuse_dense_sparse); the ideal weight differs by collection and by query type (entity recall vs semantic generalization).
  • Late-interaction multi-vector (ColBERT-style token-level vectors): today each chunk has one dense vector; no token-level late interaction.
  • Cross-space score-normalization comparison visualization: RRF hides each space's raw score; debugging lacks a "score distribution per space" tool.
  • Multi-vector ablation eval scaffolding: no systematic "turn off one space, measure QA delta" entry (partially achievable via the suppress_collections intent, but no unified harness).

Potential extension points: add a hybrid_alpha field to CollectionProfile; have ExpandedQuery.intent carry an alpha hint; add a MULTI_VECTOR_ABLATION diagnostic hook.

7. References