My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
[–] jacquesm 4 days ago
[–] bitexploder 4 days ago
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
[–] jacquesm 4 days ago
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.