LLMs and the Harry Potter problem

🌈 Abstract

The article discusses the "Harry Potter problem" - the inability of large language models (LLMs) to effectively utilize and recall information from long contexts, even when they have large context windows. It highlights the practical implications of this issue, particularly in high-value use cases like analyzing insurance policies. The article also evaluates the potential solutions of using retrieval-augmented generation (RAG), fine-tuning, and agents, and concludes that the best approach is to develop an opinionated view of the document structure and information hierarchy, which requires significant domain-specific effort.

🙋 Q&A

[01] The "Harry Potter Problem"

1. What is the "Harry Potter problem" that the article discusses?

• The "Harry Potter problem" refers to the inability of large language models (LLMs) to effectively utilize and recall information from long contexts, even when they have large context windows.
• This is demonstrated by the models' poor performance in tasks like counting the number of times the word "wizard" is mentioned in a chapter of Harry Potter, despite having access to the full chapter.

2. What are the key statistics provided in the article that illustrate this problem?

• GPT4 Turbo: 55% accuracy on documents over >=64k tokens
• Claude 3 Opus: 65% accuracy on documents over >=64k tokens
• Mixtral 8x7b Instruct: 17.5% accuracy on documents over >=64k tokens
• Gemini 1.5 Pro: 45% accuracy on documents over >=64k tokens

3. Why should we care about this problem?

• The "Harry Potter problem" can significantly impact the accuracy of LLMs in high-value use cases, such as analyzing insurance policies, reviewing lengthy legal cases, understanding codebases, and reviewing medical records.
• The issue is that the failure to properly utilize the full context can lead to incorrect answers, and the models can also confidently provide incorrect information.

[02] Potential Solutions

1. Why doesn't Retrieval-Augmented Generation (RAG) solve this problem?

• Traditional RAG does not take into account the structure and informational hierarchy of a document, so the retrieved chunks and prompts may ignore relevant information located elsewhere in the document.
• Metadata filtering is a step closer but still does not completely solve the problem, as it limits the retrieval to an arbitrary number of chunks and may miss important context.

2. How do fine-tuning methods only partially solve the problem?

• Quick fine-tuning methods like LoRA do not fix the fundamental issue with how LLMs tend to digest context, as they have been shown to pay more attention to the beginnings and ends of documents than to information located near the middle.
• Carefully planned full fine-tuning in a certain order can help improve long-context performance, but this is costly and limited by the scarcity of long, industry-specific texts.

3. Why don't agents (as of now) fully solve the problem?

• While agents have the potential to solve the "Harry Potter problem," the current state-of-the-art has not yet yielded satisfactory results.
• To fully solve the problem, an agent would need to autonomously digest the entire document, develop an ontology that addresses the specific use case, and figure out a way to parse the document with all its complexities.

[03] The Authors' Approach

1. What is the authors' recommended approach to solving the "Harry Potter problem"?

• The authors have found that the best way to solve this problem is to have an opinionated view of what each long document should look like, the information it should contain, and how the information within the document is interconnected.
• This involves developing an ontology for the specific document type (e.g., insurance policies) and building an ingestion and retrieval pipeline around it, even if it means sacrificing the ability to understand other types of documents.

2. What are some specific techniques the authors mention for this approach?

• Using knowledge graphs to model the relationships between different pieces of information in the document
• Treating the document like an encyclopedia, with a table of contents, glossary, and list of citations that the LLM can consult before retrieving relevant chunks
• Experimenting with various chunking techniques to improve the model's ability to understand the document structure

3. What are the challenges and limitations of this approach?

• It is a difficult and time-consuming undertaking, as it requires deep domain-specific knowledge and extensive experimentation with different document types.
• It does not generalize well, as the process needs to be repeated for each new category of document the user wants to understand.
• The authors acknowledge that this approach is not a universal solution and that a more generalizable solution would be preferable, if one can be found.