IdeaWazaWiki

Retrieval-augmented generation

Retrieval-augmented generation (RAG) is a technique in which an AI language model is given relevant passages, retrieved from a collection of documents, at the moment a question is asked. In plain terms, it lets an AI answer from your own documents, or from material newer than its training, instead of relying only on what it learned in training.

The problem it addresses

A language model learns from text collected up to a training cutoff, and learns nothing newer unless it is trained again. It cannot see private files such as handbooks or personal notes. It can also write fluent statements that are false, called hallucinations. The 2020 paper that named RAG observed that large models cannot easily revise their memory, cannot straightforwardly show where a prediction came from, and may hallucinate.[1]

Retrieval hands the model the right text at question time, so knowledge lives in documents that people can read, edit and replace. The paper swapped a 2016 Wikipedia index for a 2018 one to update answers about officeholders, with no retraining.[1]

The basic pipeline

  • Split. Documents are divided into chunks, such as a section or a few paragraphs.
  • Index. Each chunk is stored in a searchable index with its title and source link.
  • Retrieve. The question finds the best chunks by keyword search, by embeddings (lists of numbers that place similar meanings close together) compared with vector similarity, or by a hybrid of both.
  • Rerank. Optionally, a slower model reorders the top candidates.
  • Prompt. The best passages are placed in the prompt with the question.
  • Generate. The model writes an answer and, ideally, cites the passage behind each claim.

Origin

The name comes from the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by researchers at Facebook AI Research, University College London and New York University.[1] It joined a pretrained text generator with a dense vector index of Wikipedia, split into chunks of 100 words, and reported leading results on open-domain question answering.[1] Today the term usually covers simpler systems that retrieve text and place it in a prompt.

Key components

An embedding model turns a chunk or a question into a vector, and one model must handle both so the vectors can be compared. A vector database stores vectors and finds the nearest quickly. A plain search index can serve small collections, and SQLite includes a full-text search module, FTS5, inside a single file.[2]

Chunking strategy matters as much as the models. Chunks can have a fixed length with overlap, follow headings, or match whole wiki pages. Small chunks match precisely but lose context. Large chunks keep context but blur the topic.

Compared with fine-tuning and long context

Fine-tuning changes the weights of the model. It suits a style or a task, but it is a slow way to keep facts current and cannot point to a source page. RAG leaves the model alone and changes what it reads.

Pasting every document into a long context window is simple for a small collection. Cost grows with length, and one study found that performance can drop significantly when relevant information sits in the middle of a long input.[3] Retrieval selects a short, relevant subset.

Evaluation

Check two parts. Retrieval quality asks whether the right passages appear among the top results. Answer faithfulness asks whether the answer follows from them. Either can fail alone. OWASP names context relevance, groundedness, and answer relevance as the RAG Triad.[4]

Limitations and failure modes

  • Bad chunking. A split through a table or definition leaves misleading passages.
  • Missed retrieval. The answer exists, but the search ranks it too low, and the model may answer from memory.
  • Outdated or conflicting sources. Old and new pages may both be retrieved.
  • Prompt injection through retrieved text. OWASP describes an attacker editing a document used by a RAG application, and notes that RAG does not fully mitigate prompt injection.[4]
  • Privacy. The index holds copies of private text, and a hosted embedding service receives what it embeds.
  • Citations that look right but do not support the claim. A real link proves little unless the passage says what the answer says.

Practical guidance for small projects

  • Start with a few hundred Markdown files.
  • Use a local embedding model so text stays on the device.
  • Keep source links so answers can be checked.
  • Test with questions whose answers are known.
  • Begin with keyword search, and add embeddings if it misses differently worded questions.

Connections to this wiki

Local AI fits well, because the model and the index can both run on your own device. See Running AI models locally. Notes kept as plain files match Local-first software and data ownership and How to run a personal wiki.

The Model Context Protocol lets a server offer search over documents as a tool that an assistant can call.[5]

A wiki is a good retrieval source because pages are short, titled and linked. Clear titles, bold leads and listed sources make a page easier for an agent to find and verify. See A guide to this wiki for AI agents, Open-source software that should exist and Open research.

Discussion questions

  • Which of your document collections should never be indexed?
  • How would you tell whether a wrong answer came from poor retrieval or from the model?

Essay ideas

  • Compare fine-tuning, long context windows and retrieval for keeping an assistant current.

AI prompt ideas

  • Explain embeddings with a library analogy, then quiz me with five questions.
  • Help me write ten test questions with known answers for a folder of Markdown notes.

Readings from Wikipedia

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv
  2. ↑ SQLite FTS5 extension, SQLite documentation
  3. ↑ Lost in the Middle: How Language Models Use Long Contexts, arXiv
  4. ↑ 4.0 4.1 LLM01:2025 Prompt Injection, OWASP Gen AI Security Project
  5. ↑ MCP specification, tools
Retrieved from "https://ideawaza.com/index.php?title=Retrieval-augmented_generation&oldid=92311"