Dense models are not well suited to this. Depends on your system, but most inference engines let you put the KV cache on the host. They should let you use "Q8" KV cache format as well which takes half the storage of bf16, the default format. You essentially are stuck at having to fit the entire dense model into VRAM because it must compute across the entire model every token. Without it being in VRAM it is slow. MoE models such as Qwen 35B with 3B active parameters are much friendlier to limited VRAM systems. This is why you see systems with 16GB of VRAM running something like DeepSeek Flash if they have a bit of spare system RAM. Usually not at any good token rate, but tolerable.

Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.

I've benchmarked quite a few models on my 9070XT, and I basically gave up hope of using models locally: https://claude.ai/artifact/77pHgucqQNsmic8ZEnqdBA

I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.

I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.

Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.

Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.

The easiest thing to do (takes 5 minutes) is to use LMStudio and then search for the model inside LMStudios UI.

It will show you on the model selection IF a model fits and if you use a "Qwen 3.8-35B-A3B" type of model, it should just work out of the box. A3B stands for Active which means you need a lot less fast ram.

You will run into issues fast though: If you have big tasks for it to do, your pc/mac will get warm very fast and noisy and you will cosume relevant more energy.

And as you mention: yeah small context!

Its still not that practical in my opinion.

If you have enough system RAM (64GB), you can run Qwen 3.8 Flash Next with a GPU as small as 12GB, this repo is pretty amazing:

https://github.com/Niko1221/Strata/

My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).

I get around 40-60 tok/s depending on the quantization level I choose.

As you can see, the CPU is nothing special, don’t even need DDR5.

AI harnesses like Claude will set it all up for you, you can just say "I want to run a localllm on this machine" and it will find the best one for your hardware and even help you fine tune it etc