🔍 I've watched a GitHub's Universe'24 session about how they implemented RAG (Retrieval-Augmented Generation) in their Copilot Chat for the JetBrains IDE.
🤔 What did I find interesting?
- The steps they've implemented (listed further down) leverage more than just vector representations.
- The choice to use BM25, an increasingly popular algorithm from the 80s that is also used by Perplexity.
- The reason for using GPT 3.5 Turbo instead of GPT 4 mini.
- Byte Pair Encoding, which started as a compression algorithm in the 90s but now being used for tokenization.
Here are the steps:
1️⃣ During startup, project files are tokenized using byte pair encoding and a local index built with 500 tokens per file.
2️⃣ When the user asks a question, it asks GPT 3.5 turbo to return a list of keywords related to the question plus synonyms and variations. Then it locates similar snippets in the local index by applying BM25 ranking and selecting the top 50 candidates.
3️⃣ The top snippets plus the user question are sent to a text embedding model to receive a vector representation. Then these vectors are compared using cosine similarity to extract the top 5 snippets.
4️⃣ The selected snippets and the prompt is sent to the LLM to get an answer.
Why BM25?
Best Match 25 is formula that ranks documents depending on: how frequent a keyword is in a document, the document size, the proportion of documents that contain the keyword relative to the total number of documents. It is a relatively complex formula that even uses logarithms and yet is faster than the alternatives.
Why they used GPT 3.5 Turbo to get keywords? Wouldn't GPT 4 mini be better? Surprisingly, they experimented and got better results with 3.5 than with 4!
Why only 5 top snippets are sent with the prompt?
Because research shows that LLMs tend to get "lost" if given too much context.
And why this hybrid RAG?
Because vector search on its own tends to be very poor at matching the meaning of the words.
So there are quite a number of things about RAG that are not immediately obvious. Pure embeddings and vector databases are not enough and some traditional keyword-based retrieval is necessary.
Thoughts? Seen any interesting RAG systems? Hybrid or otherwise? 🤓
#MachineLearning #RAG #AI #SoftwareEngineering
