I've checked out DeepSeek's latest AI, both the app and the 4bit quantized versions that you can run on a laptop.
At least for the questions I've asked, their local quantized model is not that great. It sure is way more verbose and goes in all directions, not always the correct one.
It reminded me exactly of when I was getting my drivers license decades ago. 😂
I passed the driving test and was waiting for the picture to be taken, then the DMV clerk said I had put the wrong last name on the form.
I told her it was Fernandez, then that person went into a monologue saying "since you came from a Hispanic country", the last name should not be my father's but my mother's surname.
I tried to clarify that Brazil is not a "Hispanic country" and my last name is actually Fernandez, how it is in my passport, but the tokens were coming out too fast from that model. 😂
So even with the open sourced DeepSeek model, you will still need either a beefy array of GPUs and a high-end machine.
Matthew Carrigan at Hugging Face came up with one example quoted at $6000 without GPUs that would run Deepseek-R1 at 8bit quantization and generate up to 8 tokens per second.
