Hacker News

points by brandall10 2 days ago | 0 comments

That's only part of the issue - prefill is horrible prior to M5 class chips.

So it depends on what you're doing. Using it for a fun oracle without feeding it too much context per turn? Sure, an older Max chip is great.

As a coding agent doing meaningful work on a sizable project? Compared to a cheap as chips cloud option, it will take an order of magnitude more time to complete.

[–] seanmcdirmid 2 days ago

Prefill isnt a huge problem with an MoE, but ya, dense models are a non starter before the M5. You are still better off using DeepSeek than a local MoE, but locally running something gives you possibilities if you are on an airplane without internet, or if you want to jump guard rails, or if you want to run a home device server, etc…

[–] labcomputer 2 days ago

SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.