Introduce Hybrid Search API using SQLite FTS5 + Vector search

[varshaprasad96]

🚀 Describe the new functionality needed

Currently, Llama-Stack supports optimized chunked writes (PR #1094) for efficient SQLite-based storage. However, there is no built-in Hybrid Search API that combines FTS5 and sqlite-vss to enable semantic and lexical retrieval.

This issue proposes the addition of a Hybrid Search API that allows users to:

1. Store text documents with both full-text and vector embeddings.
2. Perform hybrid search that ranks results by combining BM25-based text relevance and vector similarity.
3. Utilize chunked writes (from PR feat: Chunk sqlite-vec writes #1094) to optimize insertions for large datasets.

Ref: https://github.com/liamca/sqlite-hybrid-search/tree/main - The idea would be take Reciprocal Rank Fusion between FTS5 and vector-based search results to ensure that highly ranked documents across multiple lists are prioritized.

💡 Why is this needed? What if we don't build it?

Building Hybrid Search with RRF will ensure better accuracy, more relevant results inside Llama-Stack's current sqlite vector DB implementation.

Other thoughts

No response

[varshaprasad96]

cc: @franciscojavierarceo

[franciscojavierarceo]

We could probably make the choice between these configurable.

[varshaprasad96]

/assign @varshaprasad96

[franciscojavierarceo]

@varshaprasad96 I've implemented an MVP of full text search with BM2 in @feast-dev that can be useful in understanding what an implementation could look like (see here: feast-dev/feast#5082).

There will be obvious differences and it'll require expanding the API to allow for passing an input of the raw string/query along with the embedding.

I'd also recommend with adding full text search first alongside vector search and then exposing hybrid, as they all have nuances.

Given the variance in how other providers will handle their implementation, it may make sense to share a short RFC outlining the approach.

[varshaprasad96]

Thanks @franciscojavierarceo! I've started on implementing FTS in sqlite in here.

+1, makes sense. I'll modify the implementation and create a PR to support FTS and Vector search in parallel. Will then create a RFC to introduce changes to API such that it's easier to implement hybrid search with other DBs as well.