RAG & Hybrid Search Product
Making complex document collections easier to search—and building retrieval quality that can be measured.
Work on ESS Explorer through the Cadmus GenAI Lab, developed toward a client-facing proof of concept. This was professional-role work, not a Kogan AI Consulting client engagement.
Project at a glance
| ~10,000 | 3 | 1 selected |
|---|---|---|
| Documents in the corpus | Embedding models benchmarked | Model chosen after relevance and cost comparisons |
The selected model was all-mpnet-base-v2. These figures describe project scale and selection rigor—not business ROI or a measured retrieval-accuracy percentage.
The challenge
Large libraries of PDFs and reports often contain valuable information that’s difficult to find. Keyword search can miss relevant passages written with different terminology; semantic search can miss exact terms, document IDs or specialized phrases.
For a useful AI search product, the problem is not merely producing a fluent answer. The system must find relevant source material reliably and make that material traceable.
My contribution
I worked on the retrieval foundation: document preparation and chunking, vector embeddings, evaluation of candidate embedding models, source metadata, and the hybrid search that combined keyword and semantic retrieval.
Three sentence-transformer models were assessed against representative search needs. all-mpnet-base-v2 was selected for stronger relevance than the tested MiniLM alternatives while retaining an acceptable latency and operating-cost profile.
Hybrid search combined BM25 keyword retrieval with semantic vector search, then applied a reranker to refine the combined results and surface the most relevant documents.
What the work delivered
The work delivered a hybrid retrieval system for a corpus of roughly 10,000 documents: an evaluated embedding model, BM25 keyword retrieval, semantic vector search, and a reranker over the combined results, with passages returned alongside their source context.
No independently verified user-time savings or numerical retrieval uplift is claimed here; the available benchmark materials establish relative comparisons rather than public quantitative gains.
Why it matters
Reliable RAG begins with reliable retrieval. This work demonstrates how to evaluate search quality, make cost-versus-relevance decisions, and build toward AI tools that use verifiable source material rather than relying on a black-box demonstration.
Explore the technical approach · View services · Discuss a similar project