Hacker News
points by jillesvangurp 4 days ago | 0 comments
It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.

In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.

If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.

At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.

I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).

[–] aleph_minus_one 4 days ago
> It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.

It's also about "insurance protection" against the AI company raising the prices or changing their terms (e.g. becoming more frugal with respect to token drops).

[–] kevmo314 4 days ago
If they raise their prices you can switch pretty quickly, insurance isn’t worth much.
[–] jillesvangurp 3 days ago
The speed at which you can switch is probably proportional to your dependence. But the thing with insuring against risk is that you need to cover all of them and not just cherry pick them.

If you are running your own models on your own hardware you take a bunch of risk as well. Is your hardware capable enough to run your models? Do you have enough of it and access to more when you need to? And is the stuff that you picked still going to be good enough in six months? And can you trust those model providers that you picked to keep up with the rest of the industry or are they going to fall behind and leave you stuck with obsolete but still very expensive technology.

Of course, this is an article about IBM who are in the business of selling consulting and services at a premium. They would probably love companies to self host their models on infrastructure that IBM supplies and consults on (for obvious reasons). The question is, should you buy what they are selling or can you do better? My simple observation here is that if companies like IBM are getting excited it's probably because they are looking to separate a lot of companies from some pretty significant cash. So, I'd be wary to buy what they are selling. But that's just me.

In a rapidly evolving market with extreme levels of uncertainty around tech and what will happen next, being reluctant to get pinned down on the wrong solutions is probably a good instinct. The biggest risk here is making the wrong bets. And you need to look at what it's going to cost you if you get it wrong and what it's going to cost you to fix it.

[–] aleph_minus_one 4 days ago
This depends a lot on how deeply integrated you and your workflows are into the idiosyncrasies of some specific LLM or class of LLMs.
[–] awesan 4 days ago
In this context yes, but for example at my job we are using LLMs to evaluate some outputs of cheaper/simpler models and to add context to them. This gets expensive very quickly with per-token pricing and would pin our entire business on the whims of one or two providers. With self-hosted open-weight LLMs we can just operate like we always have, it's a different kind of processing but otherwise we are free to do as we please wrt compute.

So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.