Hacker News
points by trollbridge 4 days ago | 0 comments

  While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.

  “The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
[–] liorn 4 days ago
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
[–] trollbridge 4 days ago
That sounds like a task that can be automated.
[–] liorn 4 days ago
It's not that easy.

The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.

Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...

[–] trollbridge 4 days ago
That sounds like a task LLMs will soon be able to take over.
[–] LoganDark 4 days ago
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
[–] trollbridge 4 days ago
The whole point of this was for internal usage.
[–] LoganDark 3 days ago
It can be very dangerous to let people loose with such advice, FWIW. They don't necessarily know what "internal usage" actually means, or how important it is to keep it that way as long as you don't have a real expert to at least help direct the LLMs.
[–] Buttons840 4 days ago
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.