I've been exploring something that sounded improbable a few years ago - running a real LLM directly in the browser.
No server. No API keys. No backend at all.
This is entirely possible using ONNX Runtime Web + Transformers.js.
The results are interesting enough. At least you can see some LLM hallucinations without OpenAI. 😆
I put together a small demo repo showing how this works with Qwen, Llama, and other lightweight ONNX models.
The browser downloads the model at runtime, initializes it inside a Javascript Web Worker, and runs inference using WebGPU (that is, it uses your local GPU) and streams tokens back to the UI. Everything happens locally.
The demo is intentionally minimal: one HTML page, a worker, and a bundled Transformers.JS IIFE. Just enough to prove the point - you can run ONNX language models fully on the client with no external compute.
These models do hallucinate a lot because they are tiny (fewer than one billion parameters) and use q4 quantization instead of full 32-bit precision.
That size is just enough to produce grammatically coherent English, follow short instructions, rephrase and summarize within a narrow context and only stay on-track when you add RAG with explicit facts.
But that is the only way to fit in browser memory and load quickly enough to be interactive and run on WebGPU in real time.
If you're curious about local LLMs, Transformers.js, WebGPU, or building offline-first AI tools, the repo might be a useful reference.
Enjoy running ONNX models in the browser!
Repo available at my personal GitHub Page and live demo at https://onnxtransformersjs.pages.dev/
