Hacker News
points by jacquesm 11 hours ago | 0 comments
If Unsloth, DavidAU or mradermacher releases a quant that fits my hardware I'm definitely going to give this a shot.
[–] prophesi 10 hours ago
They were given day zero access to release their quantizations[0]. The lowest quant will still require 256GB VRAM to run inferences, but as always I'm amazed by the Unsloth team by how much they can reduce the requirements for these powerful LLM's, and the easy-to-follow guides to get started with using them.

[0] https://unsloth.ai/docs/models/mistral-large-4

[–] jacquesm 10 hours ago
Ugh, ok that's out of my league :) Almost though, another 32GB and it would work.

Thanks for the link, appreciated!

[–] sullivanmatt 10 hours ago
https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF

I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.

[–] jacquesm 10 hours ago
Yes, you are right, and I was too optimistic. I'd have to level up.

This is the stuff I've put together and it is clearly too small for this:

https://jacquesmattheij.com/local-inference-with-large-langu...