Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.

I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.

I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.

Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.

Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.

It will show you on the model selection IF a model fits and if you use a "Qwen 3.8-35B-A3B" type of model, it should just work out of the box. A3B stands for Active which means you need a lot less fast ram.

You will run into issues fast though: If you have big tasks for it to do, your pc/mac will get warm very fast and noisy and you will cosume relevant more energy.

And as you mention: yeah small context!

Its still not that practical in my opinion.

https://github.com/Niko1221/Strata/

My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).

I get around 40-60 tok/s depending on the quantization level I choose.

As you can see, the CPU is nothing special, don’t even need DDR5.

Definitely grabbing 128 GB + for the next machine.

Yes the latest chip is better but the rumor is that the M7 has either more/better parts for LLM math OR that it has significiant changes to memory bandwidth too.

But well, they'll survive :) Their M-series was a stroke of luck considering AI.

The fabs are terrified to overscale in case the market collapses before the builds are complete.

> AI datacenter provisioning will go down

Depends on political opposition at both local and national levels. Hard to predict the future, which feeds into the previous point.

Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.

And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...

Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.

And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.

Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.

And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.

Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090.

The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism.

So in terms of money/usefulness, Mac always loses.