Andrej Karpathy's LLM Wiki: An Alternative to Traditional RAG

The idea in one paragraph

Andrej Karpathy sketched an alternative to traditional RAG that he calls an LLM Wiki — an "idea file" rather than a rigid specification (original gist). Instead of keeping raw documents as the only knowledge available at query time, an LLM continuously reads source material and compiles it into a persistent, Markdown-based knowledge base: summaries, entity pages, concept pages, comparisons, and synthesis — all cross-linked. Queries then consult already-compiled knowledge instead of rediscovering everything from raw chunks on every question.

If you're new to retrieval pipelines, read our RAG Engineering guide first. This post assumes you know basic RAG and want to understand what the wiki pattern changes — and what it doesn't.

1. The problem: how traditional RAG works

Normal RAG follows this flow:

User Query
↓
Embedding
↓
Vector Database
↓
Similarity Search
↓
Top K Chunks
↓
LLM
↓
Answer

Concretely: documents are loaded, split into chunks, embedded into vectors, and stored. At query time your question is embedded, the nearest chunks are retrieved, and the LLM generates an answer from them. See RAG and chunking for the mechanics.

The key limitation: the system reconstructs the answer from raw pieces on every question. Imagine 15 documents where 8 discuss transformers from different angles. Ask about attention today, ask about positional encoding tomorrow — each time the pipeline re-retrieves overlapping chunks and re-synthesizes shared background from scratch. Nothing learned yesterday persists; nothing compounds.

2. What "vector-less" RAG actually means

Not every retrieval system needs embeddings. Alternatives include keyword search, BM25, full-text search, SQL queries, metadata filtering, file-system search, hierarchical navigation, LLM-based selection, structured knowledge bases, and graph traversal. See vector-less RAG and the 14 types of RAG.

The crucial point: "vector-less RAG" is a broad category, not one architecture. Swapping a vector DB for BM25 is vector-less. So is SQL lookup. So is an LLM browsing folders. Don't let anyone sell you "vector-less" as a single competing system — and especially don't confuse the whole category with Karpathy's specific pattern, which is about persistent compiled knowledge, not merely about skipping embeddings.

Three distinct things, kept separate:

1. Traditional RAG — embed, retrieve chunks, generate per query.
2. Vector-less / non-vector retrieval — any retrieval without embeddings (many unrelated techniques).
3. Karpathy's LLM Wiki — a persistent, LLM-maintained knowledge base queried instead of (or alongside) raw chunks.

3. The LLM Wiki idea

The central move: compile knowledge once, reuse it many times.

Raw Sources
↓
LLM Ingestion
↓
Structured Markdown Wiki
↓
Cross References / Index
↓
Query the Wiki
↓
LLM Answer

An LLM reads your PDFs, articles, and notes — not to answer one question, but to write the reference future questions will use. Overlapping documents get synthesized into single coherent pages. Relationships get recorded as links. The wiki becomes the durable asset; raw sources remain untouched as ground truth.

Karpathy frames this as an idea file, not a spec: the exact page types, tooling, and query mechanics are yours to design. What matters is the shift from "retrieve raw pieces per question" to "maintain compiled knowledge continuously".

4. The three layers

Layer 1 — Raw sources. PDFs, articles, papers, web pages, images, data files, notes. Immutable; the source of truth you can always audit against.

Layer 2 — The wiki. LLM-authored Markdown: overview pages, summaries, entity pages (people, orgs, technologies), concept pages, comparison pages, synthesis pages — interconnected through links. These are compiled knowledge artifacts, not converted copies of the originals.

Layer 3 — Schema / instructions. A file like AGENTS.md or CLAUDE.md defines the rules of the system:

• File organization and naming conventions
• Page types and linking rules
• Ingestion workflow (what happens when a document arrives)
• Query workflow (how questions navigate the wiki)
• Maintenance rules (contradictions, staleness, orphans)

Raw Sources
↓
Schema / Rules
↓
LLM
↓
Wiki

The schema is what keeps hundreds of LLM-written pages coherent instead of drifting into chaos.

5. Ingesting a document, step by step

Say you add attention-is-all-you-need.pdf:

Step 1 — Collect. Store the original file unchanged. Sources are never edited.

Step 2 — Normalize. Convert to Markdown or another consistent text representation so the LLM processes everything uniformly. (Normalization aid, not a requirement that every source literally become .md.)

Step 3 — Extract. The LLM identifies main concepts, people, organizations, technologies, claims, definitions, relationships, and findings.

Step 4 — Create or update pages. An example layout (yours will differ):

wiki/
├── index.md
├── concepts/
│   ├── attention.md
│   ├── transformers.md
│   └── self-attention.md
├── entities/
│   ├── vaswani-et-al.md
│   └── google-research.md
├── sources/
│   └── attention-is-all-you-need.md
└── synthesis/
    └── transformer-evolution.md

Step 5 — Cross-reference. Attention → Self-Attention → Transformer → Positional Encoding. Links are the connective tissue.

Step 6 — Update the index. The catalog future queries navigate by.

Step 7 — Flag contradictions. New source disagrees with an existing claim? Record both, with provenance — never silently overwrite.

Step 8 — Compound. Each ingestion leaves the base richer: more pages, more links, better synthesis. This is the whole game (see section 14).

6. Why Markdown?

Markdown is human-readable and LLM-readable, portable, git-friendly, versionable, editable, inspectable, linkable — with zero vendor lock-in. You can open the knowledge base in any editor, diff it in git, or visualize it in tools like Obsidian. When the LLM makes a mistake, you read the page and fix a sentence; try that with a vector index.

Again: the files matter not because they're Markdown, but because they're compiled, reviewed, persistent knowledge rather than opaque embeddings.

7. Is it a tree or a graph? (Important)

Common misunderstanding, corrected: the directory layout can look like a tree, but cross-links make the structure a graph:

                AI
                |
        ┌───────┴───────┐
        |               |
       RAG            Agents
        |               |
    Retrieval        Tool Use
        |
   Vector Search

But — and this is the part people get wrong — nothing mandates graph traversal per question. The tree-ish folders aid human browsing, the links form a graph you can walk or visualize, index.md aids navigation, and the LLM reads whichever pages matter. Optional search tools (keyword, vector, or both) can sit on top. Graph visualization is a bonus, not the engine. Markdown + links form the interconnected structure; no single traversal algorithm is required.

8. Querying the wiki

Example: "How does self-attention improve transformer architectures?"

User Query
↓
Read Wiki Index
↓
Identify relevant concept pages
↓
Follow relevant links
↓
Read relevant Markdown pages
↓
Collect supporting information
↓
LLM synthesizes answer
↓
Citations / source references
↓
Answer

The exact mechanism is implementation-dependent — index lookup, link-following, keyword search, or even vector search over the wiki pages. The architectural difference is what gets consulted: a compiled knowledge layer, not the raw corpus. Citations should trace back through wiki pages to original sources, preserving auditability.

9. Traditional RAG vs LLM Wiki

Hedges are deliberate: each row depends on implementation.

10. Cost: vector RAG vs LLM Wiki

Two dimensions, different winners.

Ingestion. Traditional RAG pays for parsing, chunking, embeddings, and storage — cheap per document. The wiki pays for LLM reading, summarization, entity extraction, cross-linking, and maintenance — often more computation per document.

Query. Traditional RAG pays embedding + retrieval + generation per question. The wiki may skip embedding/retrieval, but the LLM often reads several structured pages — context isn't free either.

No universal "cheaper" winner. A moderate, evolving personal base can favor the wiki (synthesize once, reuse); a million-chunk corpus with heavy semantic querying favors vector/hybrid retrieval. Cost follows corpus size, query volume, model choice, and context size — measure your own shape.

11. Quality: which answers better?

Neither, universally. Vector RAG is often strong with large corpora, buried information, semantic queries, high volume, and scale demands. The wiki is often strong when knowledge accumulates over time, cross-document relationships matter, the same concepts get queried repeatedly, and durable human-readable summaries with explicit references are valuable.

Quality ultimately depends on source quality, ingestion quality, model quality, wiki structure, navigation quality, context window, query complexity, and — non-negotiable — evaluation methodology (RAG eval, grounding checks). Anyone claiming universal superiority without evals is selling something.

12. Hybrid: wiki as the knowledge layer, search as navigation

It's not wiki or vectors. A mature system keeps the wiki persistent and adds retrieval as tooling:

Raw Sources
↓
LLM Wiki
↓
Markdown Knowledge Base
↓
┌───────────────┬────────────────┐
│               │                │
Keyword Search  Vector Search    Graph/Link Navigation
│               │                │
└───────────────┴────────────────┘
↓
Reranker
↓
LLM
↓
Answer

Keyword search for exact terms, vector search over wiki pages for semantic navigation, link-walking for connected concepts — merged, reranked, generated. The wiki is the asset; search mechanisms are how you move through it. This mirrors our advanced RAG and hybrid retrieval guidance.

13. Wiki maintenance

A knowledge base rots without upkeep. The LLM should periodically lint for broken links, missing and orphan pages, contradictions, stale claims, missing cross-references, data gaps, and duplicate concepts. This is the persistence bargain: durable knowledge costs durable maintenance. Schedule it like backups — boring, automatic, non-negotiable.

14. Compounding knowledge

The conceptual payoff, stated carefully:

Source 1 → Knowledge Base
Source 2 → Knowledge Base → existing knowledge updated, new connections
Source 3 → Knowledge Base → more connections, better synthesis

The knowledge base compounds over time — each source doesn't just add pages, it upgrades existing ones. Contrast with raw-chunk RAG, where source 100 is as disconnected from source 1 as on day one. Treat compounding as the benefit to engineer toward (via synthesis pages, contradiction handling, maintenance), not an automatic property. Unevaluated, unmaintained wikis compound confusion just as efficiently.

15. Example: learning AI/RAG

You add five articles: RAG, vector DBs, CRAG, agentic AI, prompt engineering. The wiki might grow:

AI
├── RAG
│   ├── Vector Search
│   ├── Hybrid Retrieval
│   └── CRAG
├── Agents
│   ├── Tool Calling
│   └── Planning
└── Prompting
    ├── Query Rewriting
    └── Step-back Prompting

Cross-links graph it further: RAG ↔ Vector Search ↔ Hybrid Retrieval ↔ CRAG ↔ Query Rewriting. Now ask "When should I use CRAG instead of plain RAG?" — the synthesis page on retrieval correction likely answers directly, citing the CRAG and RAG pages, which cite sources. No chunk-hunting across five PDFs; the connections were compiled at ingestion.

16. When to use which

Traditional vector/hybrid RAG: very large corpora, semantic retrieval central, high query volume, low-latency needs, frequently changing data, scalable retrieval required.

LLM Wiki: personal knowledge bases, research and course notes, long-term learning, project knowledge, internal docs, cross-document synthesis, human-readable evolving knowledge.

Both: persistent compiled knowledge plus scalable semantic search — the wiki as asset, vectors as navigation.

17. Mental models

Traditional RAG:

Documents → Chunks → Embeddings → Vector Search → Context → LLM → Answer

LLM Wiki:

Raw Sources → LLM Ingestion → Structured Markdown Wiki
→ Cross-links + Index + Synthesis → Query / Navigation → LLM → Answer

Hybrid:

Raw Sources → LLM Wiki → Markdown Knowledge
→ Search / Vector / Graph Navigation → Reranking → LLM → Answer

FAQ

Is the LLM Wiki just RAG without a vector database? No — that's the oversimplification to avoid. It's a persistent, LLM-maintained knowledge base where synthesis happens at ingestion and compounds. Vector search may still be added as a navigation layer; the difference is compiled knowledge versus per-query reconstruction.

Does Markdown itself create the knowledge graph? No. Markdown files plus explicit cross-links form an interconnected structure you can view as a graph. The format enables it; the links and synthesis create it.

Must every question trigger graph traversal? No. Directories look tree-like, links form a traversable graph, and queries may use index lookup, link-following, keyword, or vector search. No single traversal mechanism is mandatory.

Is the wiki always cheaper or better than vector RAG? Neither universally. Ingestion often costs more (LLM reading, synthesis, maintenance); queries may cost less (or not). Quality depends on sources, ingestion, structure, and evals — measure, don't assume.

Is this Karpathy's official specification or a community implementation? It's Karpathy's concept — an "idea file" from his gist — not a rigid spec. Directory layouts and workflows here are illustrative examples, not mandated designs.

The two questions

Traditional RAG asks: what pieces of my documents are relevant to this question?

The LLM Wiki asks: what knowledge have I already compiled, connected, and maintained that can answer it?

The interesting future probably isn't one replacing the other — it's systems where persistent knowledge, structured Markdown, semantic search, graph navigation, and LLM reasoning work together.