> "What kind of hardware... "

At home, I mostly use openrouter.

A few bucks will realistically last you forever. I also bought a DGX Spark at launch. I've learned a ton. but it's not usually my daily driver. I do run overnight tasks on it, like "read this book and write up notes on each chapter..."

In retrospect, given the dirt cheap prices of apis and or renting hardware, buying hardware for inference was likely a mistake (but not enough of a mistake to sell it now).

Take a look at the docs for https://github.com/niko1221/strata for some ideas, particularly https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#... .

I think building a OpenAI compatible RAG-pipeline that also perhaps includes the anthropic sdk let's you have the best of both worlds. You can build it in a way that a local agent can escalate to a frontier model when needed and also substitute prompts data that you want to hide from being "public". For example. I think that is meant by harness here.

And with some smart feedback loops, you will see that for home assistant type tasks local AI can pretty quickly fulfil all requirements. For quick responses or voice analysis you might want to start with an external provider.

Otherwise, I think for complex home automation around 24-32gb vram are a realistic house number, but more experienced users will probably need less here.

People say that tools are called by the model. That isn't entirely correct. The caller calls the tools, the model just formulates the syntax for it. So the model part can be externalised without you needing to compromise you system fully (just to the degree the tools would allow anyway)

Schaefer says they write their own harness, not necessarily training or running a local model.