Hacker News

points by jacquesm 4 days ago | 0 comments

It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.

[–] gorgoiler 4 days ago

Gaming used to be done in arcades for 25c a play*.

Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.

Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.

⸻

* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.

** The model, not the video game.

[–] asdf88990 4 days ago

If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.

[–] jacquesm 4 days ago

Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.

[–] dekhn 3 days ago

I have Qwen 3.8 running under ninfer or Strata sized to fill my 5090.

From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.

My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.

[–] spwa4 4 days ago

How do you run GLM 5.3?

[–] morgoo 4 days ago

Their parent comment claims they have 8 GPUs and 2 DGX sparks.

[–] spwa4 4 days ago

Even 2 DGX sparks are limited to Q2 at best. As for GPUs, it of course depends, but anything under 50k just can't run it, not even Q2.

And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)

[–] cyanydeez 4 days ago

Halogen flash server for strix halo runs qwen3.8 flash next in 40gb

[–] wjnc 4 days ago

This being /u/jacquesm would imply it is kinder to assume him having that setup than claiming to have that setup.

[–] snarfy 4 days ago

I like this analogy a lot.

It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.

It will be very nice when AI gets to the emulator stage.

[–] doubled112 3 days ago

> It will be very nice when AI gets to the emulator stage.

Do you think we'll be longing for the nostalgia hit of LLM models from 2025 in 2050?

Maybe we will collectively miss being completely right and inappropriate em—dashes.

[–] JacobLeeCadLaw 2 days ago

I'm already nostalgic for very early Dall-e LSD-like art. Ref: "Sleeping Horses - Nov 2021."

Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.

[–] pipes 4 days ago

I'd say Nintendo gave you 10x the fun of neo geo. Yeah it looked cool, but mario and Zelda on the nes played like nothing else. And then we got street fighter 2 on snes which is better than any snk fighter I've played :)

(I'll run for cover now as I'm sure neogeo fans will disagree!)

[–] ASalazarMX 3 days ago

I'll agree, since arcades could never give the experience a JRPG could. No shop would allow a player to occupy a machine all day unless they spent a lot of quarters.

[–] MrBuddyCasino 4 days ago

2D action games peaked with Metal Slug.

[–] jonhohle 3 days ago

/SOtN entered the chat.

(I also adore MS)

[–] echelon 4 days ago

I'm stealing this analogy. This is good.

But also look at live service games and the shit modern game studios are doing to try to go back to the old world of rent seeking.

[–] senordevnyc 4 days ago

Daily reminder that this isn’t what rent seeking means.

[–] doctorwho42 3 days ago

> Rent-seeking is the act of growing one's existing wealth by manipulating public policy or economic conditions without creating new wealth.

Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).

Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.

So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.

[–] senordevnyc 3 days ago

Changing how you sell your own products is not "manipulating economic conditions" whatsoever. If it was, that would mean that every business in the history of commerce has engaged in it.

As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.

[–] doctorwho42 2 days ago

Modern gaming companies can be considered rent seekers not because they charge money for their products, but because many increasingly rely on extracting additional revenue from consumers by exploiting market power, consumer lock-in, and artificial scarcity rather than creating proportionately greater value. In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity. Developing a great game is productive; leveraging an established franchise or player community to extract additional money without providing equivalent value is rent seeking.

Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.

This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.

[–] senordevnyc 2 days ago

In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity

No, it’s not, you just made that up. Why doesn’t this match your previous definition?

The rest of your long AI-written comment has nothing to do with the principle of economic rent seeking.

Not sure why I’ve been wasting my time, but I’m out. Peace!

[–] zahlman 3 days ago

Going to the arcade was about a lot more than just the games, though.

[–] behringer 3 days ago

You've stumbled on exactly why the big providers are lobbying for regulation and big claims that Ai is unsafe. They don't want you running these at home.

[–] fsckboy 3 days ago

>Gaming used to be done in arcades for 25c a play.

in the early days, 2 plays for a quarter was not unusual (but you couldn't buy one)

[–] LogicFailsMe 3 days ago

Arcades mostly failed to adapt to the advances of home consoles to provide experiences you couldn't get at home. That's on the dimwitted game suits of the 1990s who cared little about novel gameplay, networked gameplay, or better NPC AI and physics. Then mobile happened and there was a brief glorious Cambrian explosion of new ideas until the Zynga playbook and DLC. Evidence: lived through it as a small game studio developer trying to do original things. The fun experiment now and then is to break out your 1980s video game consoles and play through their libraries. Most of the games are objectively terrible but gems like Yar's Revenge are still gems today IMO.

[–] InsidiousWombat 4 days ago

It will take a while till prices drop to a level where affording 256GB card (even if it would be shared between few coworkers) will have any sense over buying tokens.

And the AI debt bubble price spike will probably stay there for a while.

[–] jauntywundrkind 3 days ago

Intel Crescent Lake is dropping. 1.4TB/s of LPDDR5X, up to 480GB. It's not going to be cheap but it's much better than trying to buy gddr based cards.

Still (as bit over) 1/2 the ram bus width of a single HBM4 stack though! (1280b vs 2048b).

[–] jacquesm 3 days ago

That's going to be pretty pricey though. And I really want to see some independent benchmarks, that all looks very good on paper but there are some complications (notably: chip count, run length of the traces).

[–] cyanydeez 4 days ago

Well, if thw ai bubble implodes, youll sudden be exposed to real costs. Thats when a real opinion will be formed.

Uber took a decade but eventually, the real cost of a taxi service went up for many use cases.

Your argument is only solid if we are headed to the singularity.

[–] doctorwho42 3 days ago

> Your argument is only solid if we are headed to the singularity.

Perfect encapsulation of the problem. I may steal the wording you used for future conversations.

[–] otterley 4 days ago

How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?

[–] jacquesm 4 days ago

Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.

[–] glassresearch 4 days ago

I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.

[–] segmondy 4 days ago

My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...

[–] Gareth321 4 days ago

Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.

Now I want a local LLM server :( $25k is just so much money.

[–] ThunderSizzle 4 days ago

If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.

Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.

[–] Gareth321 4 days ago

Thanks for the tip! I've already got a home server set up. Maybe this is my next AI purchase. Still, US$2,600 here in Europe is not cheap :-/

[–] Jnr 4 days ago

No political censorhip? Try asking Chinese models about Taiwan.

Some censorhip and bias towards certain ideology is baked into those models.

While it is not a problem for programming tasks, I would never use it for certain other tasks.

[–] Gareth321 4 days ago

> No political censorhip? Try asking Chinese models about Taiwan.

Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.

Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)

More citations:

https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)

https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...

https://ieeexplore.ieee.org/document/10817610

https://www.washingtonpost.com/technology/interactive/2026/0...

https://link.springer.com/article/10.1007/s11127-023-01097-2

[–] deepersprout 3 days ago

> No political censorhip? Try asking Chinese models about Taiwan.

In open models you can "uncensor" them: https://arxiv.org/abs/2406.11717

[–] _ache_ 4 days ago

I'm building a local LLM server myself. For about 8k€ you can have 170tps, ~6000 prefill.

A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.

[–] Gareth321 4 days ago

I would love to hear more about your setup. Which hardware and models? How much context can it handle? How does it compare to frontier models?

[–] ali_chherawalla 1 day ago

You may not need $25k. Off Grid AI runs open models on the desktop/laptop you already have and picks ones that fit its memory, free and offline. Bigger ones run on your desktop and you can use them from your phone without any cloud servers in between. either your wifi or a tailnet like tailscale

[–] cft 4 days ago

MacBooks with large RAM and optimized for local models should be Apple's medium term goal.

[–] Gareth321 4 days ago

It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.

Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.

[–] charcircuit 4 days ago

>and it doesn’t report you to the police when it suspects you did something illegal

This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.

[–] Gareth321 4 days ago

It will not do that unless you specifically teach it to do that, or you use an Anthropic model. The issue in this case was Anthropic's safety monitoring built into the model. They record everything and flag potential illegal activity for human review. This occurs even over the API. I do not know if OpenAI does this, but I would not be surprised. I think the Chinese models are far less likely to do this (unless one resides in China).

[–] InsidiousWombat 4 days ago

renting GPU off cloud is also an option

[–] jacquesm 4 days ago

Exactly, that's the spirit. I pool the resources with some friends as well.

[–] Forgeties79 4 days ago

I think both of y’all glossed over the $25k of it all pretty quickly. This isn’t remotely an option for anyone.

[–] bityard 4 days ago

You see it as money spent, others might see it as an investment. OP doesn't necessarily say exactly what they use it for, but if education is the goal, $25k and spending your evenings learning about AI theory and tooling is way cheaper than a university degree and can easily result in millions more dollars in lifetime earnings given the current early state of the technology.

Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.

[–] Forgeties79 3 days ago

I can’t “invest” $25k into an LLM. The vast majority of people can’t. And I’m not spending $5k to timeshare it with 4 other people or whatever lol I currently make do with a 9070 + 32gb of ram.

[–] solyin 4 days ago

AnyONE. But if you got some friends, and they got friends, then that $25k becomes smaller and stomachable. My favorite rumor out of LA is groups of friends with a Lamborghini Urus that they timeshare. No idea if it's true, but if car people can get together to buy/lease a car that's MSRP is $345k, I think $25k should be an option for anyone with friends. (Hypothetically. I've read a lot about this friend thing on the Internet!)

[–] Forgeties79 4 days ago

If 10 people are sharing it the speed advantages suddenly drop drastically. That’s $2500 a head. 5 would be $5k. Then all the rules and such you need in place

This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.

[–] sailfast 4 days ago

But I don’t need my LLMs part time? And we all live in the same time zone.

And what happens when the girl at the bar figures out that you TIMESHARE that Lambo?

[–] segfaltnh 4 days ago

Its a drop in the bucket for many businesses though.

[–] dekhn 3 days ago

Anyone? I know many people in Silicon Valley, Boston, Seattle, New York, and elsewhere who would consider that a reasonable expense if it kept them viable in the job market for a few more years.

My entire career has been based on purchasing the most hardware I could afford at home and using it near its peak. I've given up other life pleasures (dinners out, nicer car/housing) several times to max out my computational capacity.

Also hardware like this, if chosen carefully, often has a viable lifetime over 5+ years

[–] Forgeties79 3 days ago

I’m not spending $25k to “stay in the job market.” I come from film production, I’m no stranger to owning my own hardware. But that’s ridiculous. And at least people rented my gear.

[–] elevation 3 days ago

> My local model never says no

In case a future user wants to know what a 2026 model would refuse to do, I needed to rotate dozens of passwords last night both on the server and on the clients. When it was time to verify that the rotated credentials worked, I asked for a script that would rate limit the credential checks. Claude/opus5 (opus!!) refused and stopped in its tracks.

In another instance, Claude was making great headway bringing a legacy Windows app up in WINE until it’s saw “auth” somewhere and shut down. In the shutdown state, Claude asked me to reword the prompt to see if I could get it to act on less dangerous instructions. But when I reworded my request, narrowed the scope to be clearly unthreatening, or even redirected the model, it’s still rejected me. I suspect Anthropic uses the less suspicious rephrasing as training material to increase their rejection rate.

[–] latentsea 4 days ago

> My local model never says no. I never get banned.

So much this. Plus everything else you mentioned. The freedom and control is real.

[–] dekhn 3 days ago

I agree the local models have many nice properties, but I would not be surprised if people start to recognize that Gemini pointing you at medical services instead of attempting to diagnose you is actually a good thing in many cases (and Gemini often does do some diagnosis anyway).

[–] ceejayoz 4 days ago

> My local model never says no.

> I don't get to hear that it's not a doctor and I should call 911.

https://en.wikipedia.org/wiki/Darwin_Awards

[–] mik1998 4 days ago

It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.

Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.

[–] ceejayoz 3 days ago

> It's pretty easy for a reasonable person to determine when to call 911…

Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.

A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.

[–] segmondy 3 days ago

Have you heard the cost of health insurance and seeing a doctor? Some people don't see going to hospital as an option. Better for them to have a local model than no options. But let me tell you how I use it for health, I take all my health records, lab results, every thing, I throw it in there for a second opinion, it explains things to me, it suggests questions I should ask on follow up, possible options, etc.

[–] ceejayoz 3 days ago

> Have you heard the cost of health insurance and seeing a doctor?

Yes. I've also heard the cost (death) of people falling into pseudosciency rabbit holes.

[–] JTbane 3 days ago

The security research is probably the biggest reason to use local models as it seems like the big service providers will ban you for life if you even hint at doing that.

[–] charcircuit 4 days ago

>I don't get to hear that it's not a doctor and I should call 911.

Local models can do this and with tool use they can even call emergency services for you on your behalf.

[–] sharmajai 4 days ago

A cloud model can also get neutered anytime and/or start using more tokens while charging you the same or more per token.

[–] jack_pp 4 days ago

isn't it the same using the same model you host locally but through a cloud provider? does the $ math favour a local setup then?

[–] segmondy 3 days ago

Cloud models sometimes get degraded. You can trust your local model and know what you are running.

[–] jack_pp 3 days ago

seems like a stretch, especially when you could choose from 10 of them if u feel they're dropping off.

Also, it's not like you're running any full models on local without shelling out 20k+

[–] jacquesm 4 days ago

If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.

[–] Asyakguen 4 days ago

That NDA case is the real argument, more than price. Token prices keep dropping anyway, but "this data can never leave the building" doesn't change with a price cut ;)

[–] greesil 4 days ago

Okay man I've got to ask, what kind of data are we talking about? Legal? Medical?

I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.

[–] jacquesm 4 days ago

http://www.modularcompany.com/

Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.

It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.

[–] robotmay 4 days ago

On a tangent: this is a really good website. I understood what you do and how you do it by reading down one page. There's a lot of projects/companies I immediately tune out from because I can't actually figure out what they do amongst all the buzzwords, animations, and excess scrolling in every dimension.

[–] greesil 4 days ago

Holy fuck you have a lot of karma

[–] tjwebbnorfolk 4 days ago

The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.

They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.

I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.

[–] lelanthran 4 days ago

> The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.

But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.

Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.

[–] tjwebbnorfolk 3 days ago

Well there are risks to being part of a crowd that's all compelled to make the same decision at the same time. Certain risks can only be transferred, not reduced in absolute terms.

It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.

[–] calmingsolitude 4 days ago

LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade.

[–] distances 4 days ago

Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.

[–] Gareth321 4 days ago

The inference is profitable. It’s the training which is really expensive and not being reflected in the subscription prices.

[–] latentsea 4 days ago

Doesn't deepseek make $38 per GPU-hour? As in they can actually serve profitably?

[–] petu 4 days ago

Deepseek doesn't have subscriptions, you can only pay per token / API prices.

[–] pimeys 4 days ago

Yes but the API prices are very low. So much that it is hard to spend over 20 dollars per month as an individual no matter how much you use it.

[–] latentsea 4 days ago

How are their API prices compared to others?

[–] MarcusE1W 4 days ago

Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.

(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)

[–] murderfs 4 days ago

People keep repeating this, but it is only remotely close to true for the literal free plans and the small subset of consumers maxing out the pro plans.

At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.

That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.

[–] roca 4 days ago

The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.

[–] ThunderSizzle 4 days ago

Put the liability on the AI companies, and they'll slow down.

However, that also means AI will be even more lobotomized since the cloud providers will panic over being liable

[–] root_axis 4 days ago

> inference is profitable

People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.

Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...

[–] m4rtink 4 days ago

But what if you become unable to pay for it ? What if:

- they raise the prices 10x

- go bankrupt

- ban you or your country

- decide not to help you with what you want to do for opaque reasons

- you are in jail because the model halucinated something and the company promptly reported you to the police

Better avoid all off these and go local IMHO.

[–] pixl97 3 days ago

>you are in jail because the model halucinated something and the company promptly reported you to the police

Just note that your local model can 100% do this itself.

[–] ffsm8 4 days ago

If you compare to API prices the value difference is not as big.

The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.

But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.

[–] ffsm8 4 days ago

If you compare to API prices the value difference is not as big.

The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.

But yeah, with current sub prices/budget... It'd stick with it too

[–] reacharavindh 4 days ago

I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it? I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a day…

It doesn’t have to be black or white…

For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.

[–] thaanpaa 4 days ago

From this article alone it should be obvious that the long-repeated myth about frontier inference being subsidized is just that: a myth.

It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.

[–] pimeys 4 days ago

A few days ago we got the news that DeepSeek makes 38 dollars profit per card with their off peak pricing (in the inference server release discussion).

If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.

[–] bityard 4 days ago

There are lots of reasons Chinese AI companies will always be cheaper than Western AI companies, but chief among them is that China is pouring ludicrous amounts of money into new energy generation projects.

Hydro, wind, solar, nuclear, coal, they are building it all, as we speak, at least 10x more than the US. They are aiming to not just be energy independent, but energy dominant.

You can't now (and certainly won't in the future) be able to run an AI provider in the West as profitably as you can in China.

[–] GeekyBear 4 days ago

DeepSeek designs models that require less hardware resources for inference.

[–] tourmalinetaco 4 days ago

Yeah, the way I see it short term is take the free/subsidized tokens while we can and the further out we go the cheaper compute will get for local setups.

[–] cgio 4 days ago

Do you take into account the token allowance? For a heavy user this would be the equivalent of many subscriptions and they add up.

[–] windexh8er 4 days ago

This trope minimizes so many downsides of "frontier" models. When you can't do what you need to with a model it really isn't valuable - which is why subscriptions still exist. As soon as you're forced to pay actual token costs people who keep saying this will likely wish they had invested in something else earlier.

It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.

[–] benoau 4 days ago

If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.

[–] dylan604 4 days ago

Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.

[–] jacquesm 4 days ago

The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.

[–] jacquesm 4 days ago

Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last.

[–] gr_norm 4 days ago

The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.

Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.

[–] selcuka 4 days ago

> It is not cheaper than using a frontier lab

Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.

[–] jacquesm 4 days ago

True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally.

[–] pllbnk 4 days ago

Would you use hosted models if they promised anonymity and no log retention and servers were physically located inside jurisdictions that respect privacy (Switzerland, Norway or similar, I'm not into law) if that meant better price/performance ratio than your current rig?

[–] e12e 3 days ago

Thank you so much for sharing your insight - it's not much serious discussion out there on sane reasons for self hosting (that I've seen).

You say the big rig is 25k - and then the two sparks are about 1k each?

Electricity is for "just" the big rig?

[–] DanielHB 4 days ago

Did you get insurance on all this stuff? I imagine home insurance would not qualify for a bespoke home-datacenter.

[–] jacquesm 1 day ago

It's up:

https://jacquesmattheij.com/local-inference-with-large-langu...

[–] bitexploder 4 days ago

I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.

My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.

You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.

I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.

[–] jacquesm 4 days ago

V100S is a very good card for some workloads. And they were cheap as well.

I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.

[–] bitexploder 4 days ago

I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.

Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.

And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)

[–] jacquesm 4 days ago

You got lucky :)

I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.

GLM 5.3 runs pretty comfortably on this rig, even with large context.

[–] tepmoc 4 days ago

Not OP. But I’ve done some calculation here https://localaicost.com/

[–] hotelsacher 4 days ago

I envy you. If I only had the money for that sweet hardware. And the electricity bill.

[–] bradknowles 3 days ago

You mean, the Chinese models, right? Like the ones that stole all the weights from ChatGPT and Claude, just like those companies stole all the IP from the rest of the world, right?

I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?

[–] bathtub365 3 days ago

They didn’t steal them, they copied them, similar to how Anthropic and OpenAI copied the materials they used to train their models to begin with.

[–] raven12345 3 days ago

No need to send data away. Local open models keep everything on your device.

[–] jamesfinlayson 4 days ago

At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.

[–] jacquesm 4 days ago

Smart move to get them bought ahead of time.

[–] jamesfinlayson 3 days ago

Might have been bought 12 months ago? Not sure when exactly but the impression I got was that they weren't too old.

[–] Betelbuddy 4 days ago

Would it make more sense to just rent the infra from a Cloud provider just for amortization purposes? While...and this depending on the the Cloud provider...getting the same guarantees of governance and personal sovereignty?

[–] Shorel 4 days ago

You may get the same guarantees on paper, but at this point there's absolutely no reason to trust any AI provider.

At best they get a fine, and it will not make a dent in their cash reserves.

[–] Betelbuddy 4 days ago

Bedrock guarantees neither your inputs are shared with the LLM providers, neither your data will be used to train the next model. I dont think AWS loves AI that much...to break their business model with a violation of terms and conditions.

But I was not even suggesting that, just run your local LLM on Cloud infra. Do you really believe they are ready to risk their Cloud business just to spy on your prompts?

[–] dmantis 4 days ago

Antrophic works with an ex-deputy of the CIA (David S. Cohen). OpenAI has an ex-director of the NSA (Paul Nakasone) in the board. Literally.

[–] Betelbuddy 3 days ago

The Bedrock models are hosted at AWS.

[–] dmantis 3 days ago

I'm sure these gentlemen will figure it out.

[–] tannertech 4 days ago

8 gpus is cheap now?

[–] jacquesm 4 days ago

Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.

Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.

I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.

[–] Eridrus 3 days ago

Whether this idea is economically largely depends on how much utilization you can get and how much you can get GPUs for.

[–] larodi 4 days ago

What he does not say, but imply is that IBM's open models are also good. And they are.

[–] AndrewKemendo 4 days ago

Yup

I’m running local models more and more and starting to move to just tailscaling into everything

[–] lp92 3 days ago

It really depends on usecase i guess. I've tried a few open models, they don't hold a candle Argon/Opus/Sonnet when it comes to my current major use case which revolves around CAD and aero simulation development work.

What's your use case where the open models are working well?

[–] sjdh76 4 days ago

Depends on context of what people build.

I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.

[–] jacquesm 4 days ago

IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.

[–] sjdh76 4 days ago

They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.

[–] amelius 3 days ago

Another benefit is that you're not helping the AI labs get better at what you do.

[–] dansquizsoft 4 days ago

> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)

I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...

[–] sp527 4 days ago

You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.