Hacker News

points by cbeach 2 days ago | 0 comments

My 5090 has a bandwidth of 1,792 GB/s, and that is just about good enough for basic (i.e. 27B models) local LLM inference.

The fastest M6 MacBook Pro (24/32GB config) has a memory bandwidth of 170 GB/s. And the M5 Max is 614 GB/s.

Am I missing some nuance of the M6 MBP here, or am I right to assume it won't set the local LLM world on fire?

[–] tpmoney 2 days ago

Arguably the thing you’re missing is the 5090 is $1999 MSRP and more like $5000+ retail currently, and then you still need to buy the rest of the computer. This on the other hand is a whole system for $2400 (for the 32gb config). Maybe won’t set the “local LLM” world on fire, but likely to do a lot more for it than the 5090s will for the moment.

[–] moosedev 2 days ago

You're correct, the base (non-Pro, non-Max) chips are still not great for local LLMs. M5 and M6 both improved prefill speed/time-to-first-token over prior gens, but bw is still a bottleneck.

There's rumored to be no M6 Pro or M6 Max coming, i.e. the higher bandwidth chips will skip a gen. M7 is also rumored to have better memory bandwidth across the range.