Long-Context LLMs Meet RAG

Oct 18, 2024 · 15m 35s
Long-Context LLMs Meet RAG
Description

📈 Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG This paper explores the challenges and opportunities of using long-context language models (LLMs) in retrieval-augmented generation (RAG) systems....

show more
📈 Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

This paper explores the challenges and opportunities of using long-context language models (LLMs) in retrieval-augmented generation (RAG) systems. While increasing the number of retrieved passages initially improves performance, the authors find that it eventually degrades due to the introduction of irrelevant information, or "hard negatives." To address this, the paper proposes three methods for enhancing the robustness of RAG with long-context LLMs: retrieval reordering, RAG-specific implicit LLM fine-tuning, and RAG-oriented LLM fine-tuning with intermediate reasoning. The paper also investigates the impact of various factors related to data distribution, retriever selection, and training context length on the effectiveness of RAG-specific tuning.

📎 Link to paper
show less
Information
Author Shahriar Shariati
Organization Shahriar Shariati
Website -
Tags

Looks like you don't have any active episode

Browse Spreaker Catalogue to discover great new content

Current

Podcast Cover

Looks like you don't have any episodes in your queue

Browse Spreaker Catalogue to discover great new content

Next Up

Episode Cover Episode Cover

It's so quiet here...

Time to discover new episodes!

Discover
Your Library
Search