Hacker News

points by strictnein 13 hours ago | 0 comments

Kind of buried down at the end:

https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B

https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...

[–] dang 8 hours ago

(I've adjusted the formatting of your post so the links get linky. I hope that's ok!)

[–] jacquesm 11 hours ago

If Unsloth, DavidAU or mradermacher releases a quant that fits my hardware I'm definitely going to give this a shot.

[–] prophesi 10 hours ago

They were given day zero access to release their quantizations[0]. The lowest quant will still require 256GB VRAM to run inferences, but as always I'm amazed by the Unsloth team by how much they can reduce the requirements for these powerful LLM's, and the easy-to-follow guides to get started with using them.

[0] https://unsloth.ai/docs/models/mistral-large-4

[–] jacquesm 10 hours ago

Ugh, ok that's out of my league :) Almost though, another 32GB and it would work.

Thanks for the link, appreciated!

[–] sullivanmatt 10 hours ago

https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF

I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.

[–] jacquesm 10 hours ago

Yes, you are right, and I was too optimistic. I'd have to level up.

This is the stuff I've put together and it is clearly too small for this:

https://jacquesmattheij.com/local-inference-with-large-langu...

[–] downback 12 hours ago

Thanks