“J’accuse! The Unjust Demise of RAG in Favor of Long-Context LLMs: A Rebuttal”

🌈 Abstract

The article discusses the limitations of large language models (LLMs) and the emergence of a new paradigm called retrieval augmented generation (RAG). It examines the debate around whether long-context LLMs (LC-LLMs) can replace RAG, and presents research findings that challenge the perceived superiority of LC-LLMs over RAG.

🙋 Q&A

[01] Limitations of LLMs and the Emergence of RAG

1. What are the key limitations of LLMs mentioned in the article?

• LLMs can produce hallucinations and contain outdated content
• Previous LLMs had a limited context length, which made it difficult to provide sufficient context for the model

2. What is the new paradigm called retrieval augmented generation (RAG)?

• RAG was developed to address the limitations of LLMs, particularly the issue of limited context length
• RAG involves retrieving relevant information from external sources and incorporating it into the model's input

3. What are some of the challenges associated with RAG?

• Chunking the input data and choosing an appropriate chunking strategy can be a challenge
• The need to optimize the context length and find the appropriate context for the prompt can be time-consuming and laborious

[02] The Debate Around LC-LLMs and RAG

1. How have the context lengths of LLMs evolved over time?

• Today's LLMs have much larger context lengths, with some models having over 1 million tokens
• This has raised the question of whether RAG is still necessary, as the large context length could potentially allow for direct insertion of documents into the prompt

2. What does the research say about the efficiency of LC-LLMs in using their context length?

• A recent paper states that LC-LLMs do not efficiently use their context length, and that the position of relevant information in the input context can significantly affect model performance

3. What are the different perspectives on whether LC-LLMs are better than RAG?

• One article claimed that LC-LLMs consistently outperform RAG, suggesting the superiority of LC-LLMs
• However, this claim was not convincing to the community, and another article questioned the results, suggesting that RAG may actually be more efficient and have better performance than LC-LLMs

4. What are the key findings from the research that challenges the perceived superiority of LC-LLMs over RAG?

• Preserving the order of chunks in the original document is important, as losing the sequentiality of information can lead to confusion in the model
• LLMs struggle when there is too much irrelevant information, and RAG can reduce the presence of irrelevant documents in the context, improving performance
• The comparison between naive RAG and LC-LLM is flawed, as almost no one uses a naive RAG today