🔇 "How to Make LLMs Shut Up"
That is the title of this very interesting article by Daksh Gupta, CEO of Greptile.
He describes a challenge of implementing an AI code review bot: in the initial release the bot was posting too many comments.
They analyzed and found out 19% of the comments were good, 2% were inaccurate and 79% were technically accurate or style-related but irrelevant to the developers.
So they tried fixing that via prompting but that didn't work. They tried having the LLM flag its own comment as "nitpick" when it was less relevant and it started incorrectly labelling critical issues as nitpicks. Having the LLM output as JSON in the format { comment: string, severity: string } resulted in a small improvement.
Then attempted giving examples of good and bad comments and it also didn't work.
Then changed the flow to call the LLM to generate a comment, then call again to judge the comment (kind of an agentic flow) but got nearly random results in addition to slowing down the process.
They considered fine-tuning but discarded the idea due to the cost, speed, and missing the flexibility of using any LLM.
🤔 What did work?
They started converting past comments into embeddings and storing them separated in clusters in a vector database with their respective upvote/downvote scores.
Then the next generated comments would also be converted to vector embedding and compared with the earlier downvoted comment vectors using cosine similarity.
If the comment is similar to 3 downvoted comments, discard it. Approve the ones with 3 upvoted comments. If no matches, approve it and let users upvote/downvote it.
🔗 LINK: https://lnkd.in/ghfuGE5w
