SproutRAG: New AI Framework Improves Long-Document Retrieval Without Extra Language Model Calls

Researchers have introduced SproutRAG, an attention-guided hierarchical framework designed to enhance retrieval-augmented generation (RAG) systems for processing long documents. The framework, submitted to arXiv's Computer Science > Computation and Language category on June 16, 2026, addresses a fundamental challenge in RAG systems: balancing the granularity of document retrieval with the need to maintain contextual coherence.

SproutRAG improves upon existing approaches by organizing sentence-level text chunks into progressively larger but semantically coherent units using learned inter-sentence attention patterns. Rather than relying on costly language model calls during indexing or retrieval—or employing lossy summarization techniques—the framework learns which attention heads and layers best capture the semantic structure of documents. This enables multi-granularity retrieval capabilities without requiring additional large language model computations or compressed summaries.

The system constructs a binary chunking tree that organizes documents hierarchically. During retrieval, SproutRAG employs hierarchical beam search to identify relevant candidates at multiple levels of granularity, capturing relationships between multiple sentences that flat retrieval methods might miss. The entire framework is trained end-to-end using a joint objective that simultaneously improves both the embedding representations and the tree structure itself.

Experimental evaluation across four benchmarks—spanning scientific papers, legal documents, and open-domain text—demonstrates the framework's effectiveness. SproutRAG achieved a 6.1 percent average improvement in information efficiency compared to the strongest existing baseline methods. The research was submitted by Amirhossein Abaskohi, and implementation code is made available publicly for other researchers to use and build upon.