RAG Tutorial in Python: Build Retrieval Without an API Key

Dileep Solanki

 
By Dileep Solanki · Published and last updated: 15 November 2026 · Code run and results recorded by the author before publication.

Most RAG tutorials jump straight to a framework and an API key. That hides the part that decides whether your system works: retrieval, meaning finding the right passage before any AI model sees it. In this tutorial you will build that half yourself in about 90 lines of Python, measure how well it works, and see exactly where it fails. Everything runs on a normal laptop, with no GPU and no paid service.

What RAG is, and which half we are building

Retrieval-augmented generation (RAG) comes from a 2020 research paper by Lewis and colleagues. It describes models that combine pre-trained parametric memory with non-parametric memory, in the original work a dense vector index of Wikipedia accessed with a neural retriever. The paper also points out that the non-parametric memory can be replaced to update the model’s knowledge as the world changes, which is why companies use RAG for handbooks, manuals and policies that change often. 

Two steps matter in practice: first find the relevant passages, then give them to a language model with instructions to answer from them. This tutorial builds the first step, plus the prompt-assembly step, and stops before the model call. That is deliberate. If retrieval returns the wrong passage, no model can recover. Testing retrieval separately tells you where the problem is.

What you need

  • Python: I ran this on Python 3.12.3. Other versions are likely fine but untested.
  • Libraries: scikit-learn 1.9.1 and NumPy 2.5.4 were installed when I ran it. Install with pip install scikit-learn.
  • Hardware: any modern laptop. There is no GPU or API key requirement.

The code

Save this as mini_rag.py. The handbook text is invented for this tutorial, so there are no copyright issues and you can edit it freely.

python
"""Tiny retrieval pipeline (the 'R' in RAG) with an evaluation loop.
No API key needed. Requires: scikit-learn
"""
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# 1) A small, made-up handbook (each entry = one document)
DOCS = {
    "library-loans": "Students may borrow up to five books at a time for a period of 14 days. "
                     "Loans can be renewed twice unless another student has reserved the book.",
    "library-fines": "A late return attracts a fine of Rs 5 per day per book. The fine is capped "
                     "at the price of the book. Borrowing is blocked until fines are cleared.",
    "exam-reval":    "Students can apply for re-evaluation of an answer script within 10 days of "
                     "result declaration by paying the prescribed fee at the examination office.",
    "attendance":    "A minimum of 75 percent attendance is required to sit for end-semester "
                     "examinations. Medical leave with a certificate counts as present up to 10 percent.",
    "hostel":        "Hostel gates close at 10 pm. Late entry requires written permission from the "
                     "warden, which must be requested at least one day in advance.",
    "scholarship":   "The merit scholarship covers 50 percent of tuition for students who score "
                     "above 85 percent in the previous year and have no disciplinary record.",
    "wifi":          "Campus wifi is available in all academic blocks and the library. Each student "
                     "receives 20 GB of data per month; speeds are reduced after the limit.",
    "placement":     "The placement cell opens registration in August. Students must upload an "
                     "updated resume and keep at least 70 percent attendance to be eligible.",
}

# 2) Chunking: split long text into overlapping word windows
def chunk_words(text, size=40, overlap=10):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

chunks, chunk_ids = [], []
for doc_id, text in DOCS.items():
    for n, piece in enumerate(chunk_words(text)):
        chunks.append(piece)
        chunk_ids.append(doc_id)

# 3) Retriever
def build_retriever(texts):
    vec = TfidfVectorizer(stop_words="english", sublinear_tf=True)
    matrix = vec.fit_transform(texts)
    def search(query, k=3):
        scores = cosine_similarity(vec.transform([query]), matrix).ravel()
        order = scores.argsort()[::-1][:k]
        return [(chunk_ids[i], float(scores[i])) for i in order]
    return search

# 4) Prompt builder (this is what you would send to an LLM)
def build_prompt(question, hits, texts_by_id):
    context = "\n".join(f"[{doc_id}] {texts_by_id[doc_id]}" for doc_id, _ in hits)
    return ("Answer using ONLY the context below. If the answer is not there, say you don't know.\n\n"
            f"Context:\n{context}\n\nQuestion: {question}\nAnswer:")

# 5) Evaluation set: question -> the document that should be retrieved
EVAL = [
    ("How many books can I borrow at once?",              "library-loans"),
    ("What is the penalty if I return a book late?",      "library-fines"),
    ("How do I get my exam paper rechecked?",             "exam-reval"),
    ("What attendance do I need to write semester exams?", "attendance"),
    ("What time does the hostel close?",                  "hostel"),
    ("Can I get a scholarship with 90 percent marks?",    "scholarship"),
    ("How much internet data do I get each month?",       "wifi"),
    ("When can I register for campus jobs?",              "placement"),
]

def evaluate(search, label):
    top1 = top3 = 0
    misses = []
    for q, gold in EVAL:
        hits = [d for d, _ in search(q, k=3)]
        top1 += hits[0] == gold
        top3 += gold in hits
        if hits[0] != gold:
            misses.append((q, gold, hits))
    print(f"{label}: hit@1 = {top1}/{len(EVAL)}, hit@3 = {top3}/{len(EVAL)}")
    for q, gold, hits in misses:
        print(f"   MISS  q={q!r}  expected={gold}  got={hits}")

if __name__ == "__main__":
    print(f"{len(DOCS)} documents -> {len(chunks)} chunks")
    long_text = " ".join(["word%d" % i for i in range(100)])
    print("Chunk demo on a 100-word text:", [len(c.split()) for c in chunk_words(long_text)], "words per chunk")
    print()
    baseline = build_retriever(chunks)
    evaluate(baseline, "Baseline TF-IDF")
    print()
    q = "What is the penalty if I return a book late?"
    hits = baseline(q, k=2)
    print("Top hits:", [(d, round(s, 3)) for d, s in hits])
    print()
    print(build_prompt(q, hits, DOCS))

Run it with python mini_rag.py.

How each part works

Chunking. Long documents are cut into overlapping windows of words (here 40 words with a 10-word overlap) so that no passage is too long and ideas aren’t cut in half at a boundary. In this tiny handbook every entry is shorter than 40 words, so each document becomes one chunk. The 100-word demo line in the output shows what happens with longer text: three windows of 40 words each.

Retrieval. Scikit-learn’s TfidfVectorizer is described in its documentation as equivalent to CountVectorizer followed by TfidfTransformer. TF-IDF gives more weight to words that are frequent in one passage but rare across all passages. We turn on sublinear_tf, which replaces the raw term frequency with 1 + log(tf) so a word repeated ten times doesn’t count ten times as much. Because the vectors are normalised, cosine similarity between two vectors is simply their dot product. One caution from the same documentation: the built-in “english” stop-word list has several known issues, and the docs suggest considering an alternative. It is fine for a demo, but check it for a real project.

Prompt assembly. build_prompt places the retrieved passages into an instruction that tells the model to answer only from the context and to say so when the answer isn’t there. In a full system, this string goes to an LLM.

Evaluation. The most important habit in the script. Each test question has a known correct document. We measure hit@1 (was the right document first?) and hit@3 (was it in the top three?).

What I got when I ran it

8 documents -> 8 chunks
Chunk demo on a 100-word text: [40, 40, 40] words per chunk

Baseline TF-IDF: hit@1 = 6/8, hit@3 = 7/8
   MISS  q='How do I get my exam paper rechecked?'  expected=exam-reval  got=['placement', 'wifi', 'scholarship']
   MISS  q='When can I register for campus jobs?'  expected=placement  got=['wifi', 'placement', 'scholarship']

Top hits: [('library-fines', 0.477), ('library-loans', 0.128)]

Answer using ONLY the context below. If the answer is not there, say you don't know.

Context:
[library-fines] A late return attracts a fine of Rs 5 per day per book. The fine is capped at the price of the book. Borrowing is blocked until fines are cleared.
[library-loans] Students may borrow up to five books at a time for a period of 14 days. Loans can be renewed twice unless another student has reserved the book.

Question: What is the penalty if I return a book late?
Answer:

Reading the results honestly

The baseline found the right document first in 6 of 8 questions, and within the top three in 7 of 8. The interesting part is the two misses.

  • “How do I get my exam paper rechecked?” The handbook says “re-evaluation of an answer script”. The question shares no useful words with it, so the passage never appears in the top three.
  • “When can I register for campus jobs?” The handbook says “placement” and “registration”. The question says “jobs” and “register”. The correct document appears second.

This is the classic vocabulary mismatch problem. Researchers note that bag-of-words methods such as TF-IDF and BM25 rely on exact matches, and that dense semantic representations can tackle vocabulary and semantic mismatches that lexical matching like BM25 cannot. That is the reason the original RAG paper used a neural retriever and not keyword matching.

A score table you can learn from

I also printed the top similarity score for each question, plus two questions that are not in the handbook at all:

QuestionTop scoreDocument returnedCorrect?
How many books can I borrow at once?0.400library-loansYes
Penalty if I return a book late?0.477library-finesYes
How do I get my exam paper rechecked?0.000placementNo
Attendance needed for semester exams?0.340attendanceYes
What time does the hostel close?0.321hostelYes
Scholarship with 90 percent marks?0.425scholarshipYes
How much internet data per month?0.369wifiYes
Register for campus jobs?0.261wifiNo
Who won the cricket world cup? (not in handbook)0.000placementn/a
What is the capital of France? (not in handbook)0.000placementn/a

Three lessons stand out:

  1. A score of 0.000 means nothing matched. The “placement” result for those rows is just the first item in a tie, not a real answer. Always check the score, and return “I don’t know” or fall back to something else when it’s zero.
  2. A threshold helps, but only partly. A cutoff just above zero rejects both out-of-scope questions. The wrong “campus jobs” answer scored 0.261, though, which sits close to the correct scores of 0.321 to 0.477. With only ten data points I can’t recommend a number. Calibrate on your own evaluation set.
  3. Don’t trust the demo. If you had tested only the easy questions, you’d have concluded the system works perfectly.

Common errors

  • ModuleNotFoundError: No module named 'sklearn': install with pip install scikit-learn. The package name differs from the import name.
  • ValueError: empty vocabulary; perhaps the documents only contain stop words: I reproduced this by fitting the vectorizer on text made only of stop words such as “the and of”. It means your documents contained nothing the vectorizer kept, so check that your text actually loaded.

Limitations

  • The handbook is eight invented documents and the test set has eight questions. This shows how the method behaves, but it is not a benchmark and the 6/8 figure shouldn’t be generalised.
  • TF-IDF is a baseline. It is cheap, fast and easy to explain, but it cannot connect “jobs” with “placement”.
  • I did not test embeddings, BM25, hybrid search or a language model call. Suggestions below are directions, not verified results.

Where to go next

  1. Swap in embeddings, which are numerical representations that place similar meanings close together. This is the natural fix for the two misses. Keep the evaluation loop the same and compare hit@1 and hit@3 before and after.
  2. Try hybrid search. An IBM Research paper notes that hybrid retrieval combining lexical BM25 and dense semantic search is now the default for retrieval-augmented workloads, because each recovers what the other misses. alphaxiv
  3. Grow your evaluation set to 50 or more real questions from real users before you trust any number.
  4. Add the language model call only after retrieval is measured, and test that the model refuses when the context lacks the answer.

Final assessment

A retrieval pipeline you can run, measure and explain is more useful than a framework you can’t see inside. The 90 lines above teach the habit that matters most: build an evaluation set first, read the failures, and let them tell you what to improve.

How this article was made

This article was researched and drafted with AI assistance and edited against the sources below. The code was run before publication on Python 3.12.3 with scikit-learn 1.9.1 and NumPy 2.5.4, and the output shown is the real output, including the two errors reproduced above. I did not test any other library versions, embedding models or LLM calls. It contains no sponsored placements or affiliate links. This is educational content, not production advice.

Sources and Further Reading

  • Newer

    RAG Tutorial in Python: Build Retrieval Without an API Key

3/related/default