Inside VOICE-RAG: From Speech to Grounded Answers

VOICE-RAG is a voice-first retrieval-augmented generation system built for HH Goa 2026. It handles spoken questions in four languages and limits answers to a curated corpus of 40,000 passages. Its central idea is simple: evidence matters more than fluent completion. If the corpus cannot support an answer, the system refuses visibly.

A user holds to talk, producing a 16 kHz mono WAV recording. Sarvam saaras:v3 converts it to text. The system embeds the query, searches multilingual E5 vectors, checks the top score against a language-specific floor, and then verifies grounding. By default, it extracts the best-supported sentence from the top passage.

The core pipeline has a measured 56 ms latency around the voice step. For example, “Who is the Prime Minister of India?” is refused because its score is 0.779, below the 0.840 floor. Generative answers are optional, slower, and must cite supplied passages in strict JSON.

VOICE-RAG uses one multilingual retrieval system for four languages: English, Hindi, Gujarati, and Telugu. This matters because users can speak naturally in different scripts while the system keeps one evidence-focused workflow. The corpus is curated, so the system is bounded by what those passages contain rather than by general world knowledge.

The corpus is MSMARCO-XI, with 10,000 passages per language. That produces 40,000 passages in total. Each passage is represented using 384-dimensional multilingual E5 embeddings, allowing the system to search across the language collection.

Language affects more than transcription. Unicode script detection identifies Telugu, Gujarati, and Devanagari text, while other text defaults to English. Auto-detected queries receive a soft 0.03 rerank bonus for same-language passages, using a three-times-larger candidate pool. The selected query language also determines the relevance floor, helping prevent cross-language matches from passing under the wrong threshold.

When VOICE-RAG cannot find sufficiently relevant evidence, it refuses to answer. This is a deliberate design choice. The system is corpus-bounded, so a question with no supporting passage must not be answered from a language model’s parametric memory. Refusal makes missing evidence visible instead of hiding it behind confident prose.

The relevance guard compares the top retrieved cosine score with a floor selected for the query’s language. For example, “Who is the Prime Minister of India?” scores 0.779 against a 0.840 floor and is refused. “Who won the FIFA World Cup 2022?” is also refused at 0.796. In contrast, an Xbox 360 price question passes with confidence 0.933.

The system exposes this decision in its interface. Each refusal states the score and floor, while the evidence drawer and latency trace show the decision chain. This gives users a concrete reason for the refusal and helps distinguish missing evidence from a failed answer.

The guards form a strict sequence around the retrieval core. The input stage receives spoken audio and converts it to text. The system then embeds the query and searches the vector index. Next, the relevance guard decides whether the strongest passage clears the floor for the query’s language. This prevents weak matches from reaching the answering stage.

If retrieval passes, the system produces an answer, usually by selecting the best-supported sentence from the top passage. The grounding guard then checks whether that answer is traceable to retrieved text. Token overlap of at least 0.231 passes immediately. Otherwise, an embedding-similarity fallback must reach 0.794. Failed generations are replaced with an extractive answer or refused.

The interface records every decision and latency in a visible “why this answer” chain. Generative mode adds strict JSON, citations limited to supplied passages, one retry for malformed output, and citation filtering. No ungrounded generation can reach the user.

VOICE-RAG uses separate relevance floors because retrieval scores behave differently across its language setting. A universal threshold could be too strict for one language and too permissive for another. More importantly, cross-language junk might pass by borrowing a threshold that was calibrated for a different query population.

The floors come from measurements, not guesses. Known-relevant queries clustered around approximately 0.84 to 0.86 at P25–P50, while off-corpus junk clustered around approximately 0.79 to 0.85. The system placed each floor near the separation point. The query’s language chooses the floor, not the language of the top retrieved passage.

Language is detected from Unicode script: Telugu maps to te, Gujarati to gu, Devanagari to hi, and other text to en. For auto-detected input, same-language passages also receive a soft 0.03 rerank bonus across a three-times-larger candidate pool. These choices make language handling part of refusal safety.

VOICE-RAG prefers extractive answers because they are fast and directly tied to retrieved evidence. The default is to select the best-supported sentence from the top passage. This avoids asking a language model to rewrite information and possibly add unsupported details. The approach costs only 0.09 ms and is grounded by construction.

Generative mode remains available when users want more flexible phrasing. It sends the task through OpenRouter, where answers take multiple seconds. The LLM must return strict JSON containing an answer, citations, and confidence. Citations may refer only to supplied passages. Malformed output receives exactly one retry, while invented or out-of-range citations are removed.

Generation still cannot bypass the grounding guard. If the answer has insufficient token overlap or embedding similarity, the system falls back to the extractive answer or refuses. This keeps generative flair optional while preserving the evidence contract and the latency-first default.

Retrieval-augmented generation combines search with language generation. Instead of relying only on a model’s stored patterns, it retrieves passages and asks the generator to answer from them. The goal is to connect an answer to external evidence. VOICE-RAG applies this idea within a fixed 40,000-passage corpus.

The danger is that retrieval can return text that sounds related without actually answering the question. A generator may then smooth over the gap with confident, fluent language. The article gives the example of demos answering “Who is the Prime Minister of India?” from whichever passage seems semantically close, even when no supporting passage exists.

VOICE-RAG makes this failure visible and blocks it. A relevance floor rejects weak top matches, and a grounding guard checks the produced answer against retrieved text. If grounding fails, the system uses an extractive answer or refuses. Thus, fluent wording cannot substitute for evidence.