DocumentationArchitecture & Pipelines

Search & Retrieval Architecture

Providing researchers with highly relevant, cross-corpus results that combine classical exact-match precision with modern semantic understanding.

Holistic View

Results from Quran, Hadith, and scholarly Books in a single, unified interface.

Contextual Relevance

Finding sentences that explain concepts, rather than merely containing keywords.

Traceability

Every result is rigidly linked back to its original source and parent node.

Data Provenance

Executed cross-corpus on SurrealDB indices generated from the compiled Quranic Verses, Hadith narration texts, and shamela book pages.

1. Lexical Plane (BM25)Full-Text Search index for exact matches, rare terminology, and specific names, using Arabic normalization.
2. Semantic Plane (HNSW)1024-dim Vector index for synonyms and conceptually related verses, running locally via Ollama.

OpenBayan utilizes a Hybrid Search strategy implemented securely within SurrealDB.

The query is cleaned and normalized. Simultaneously, it is sent to Ollama to generate a 1024-dimension vector.

FTS retrieves top matches based on keyword frequency, while the Vector Query retrieves matches based on cosine similarity.

Results from both planes are merged. A combined score prioritizes results appearing in both planes.

The system FETCHes metadata and entities for interactive tooltips, returning a sorted, fused result set.

Planned enhancements to the search infrastructure.

Implementing a secondary Cross-Encoder model to fine-tune the top 10 results.

Allowing users to filter by Taxonomy (e.g., only Fiqh books) or Topic.

Ability to search for entities and find all sentences where they interact with another specific entity.