Hacker News

points by jamesblonde 15 hours ago | 0 comments

I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).

[–] monster_truck 15 hours ago

One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.

4 bit quants can be fine, try it and see.

[–] arjie 10 hours ago

You're just going to have to try it. It won't take that long to quantize. Unless the server is currently serving traffic you should be able to get it going in under a day with a frontier model doing the driving for everything - quant, implementation, etc.

[–] dada216 9 hours ago

I have similar hardware and requirements. Spend the time to write a test process to try multiple quantization configs/multiple inference engine and throw at it a bunch of request you can replay and evaluate the results of. We got it down to a process and everytime there's a new model/options/inference engine version we just run the internal bench/eval suite and make a choice.

[–] segmondy 14 hours ago

4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...

[–] himata4113 15 hours ago

use MoE offloading

[–] espadrine 9 hours ago

That negatively impacts reading speed in my experience :(

[–] echelon 15 hours ago

> we have restrictions that we can't use a chinese model!

Please expand on this if you can. Why can't you? Who mandated this?

[–] derektank 14 hours ago

Last I heard, US defense contractors had been directed to not use Deepseek in their operations late last year[0], and most have taken that as an indicator that they shouldn’t be using chinese models generally.

[0] https://digitalpolicyalert.org/change/20659-physical-and-cyb...

[–] a012 15 hours ago

US politics. Lot of US corps now ban Chinese models, don’t ask me why, the people in charge are just dumb fucks

[–] alembic_fumes 14 hours ago

Seems like a reasonable policy to me, depending on the sector at the least? Even with open weight / self-hosted models, there's no way to know if the model has been fine-tuned to contain malicious behaviour, triggered only within very specific conditions.

In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.

[–] zardo 14 hours ago

Are there actually any laws to break?

Alibaba (Qwen) at least has plenty of people in US offices to go arrest.

[–] mapontosevenths 15 hours ago

Executives live in a weird bubble driven largely by consulting firms. Right now those firms are spreading FUD about Chinese models.

[–] 50or05 10 hours ago

Recently it was found that Qwen was dialing back home once in a while. It cannot be proven that it does not take commands from home if it can establish connection.

[–] mapontosevenths 5 hours ago

Did you read the details of that incident? It was a nothing sandwich.

I think the root of the difference between us is that you are scared of their fascists, and I am scared of ours. Both are reasonable things, but only one of us can actually do anything about it.

With local models you decide how the firewall works. If you don't want it to internet, don't let it.

With US models you get whatever they say you do, and pay whatever they demand.

[–] cyanydeez 15 hours ago

Theyre driven bootlickers because you dont get hired if not boot shaped.