I think building a OpenAI compatible RAG-pipeline that also perhaps includes the anthropic sdk let's you have the best of both worlds. You can build it in a way that a local agent can escalate to a frontier model when needed and also substitute prompts data that you want to hide from being "public". For example. I think that is meant by harness here.

And with some smart feedback loops, you will see that for home assistant type tasks local AI can pretty quickly fulfil all requirements. For quick responses or voice analysis you might want to start with an external provider.

Otherwise, I think for complex home automation around 24-32gb vram are a realistic house number, but more experienced users will probably need less here.

People say that tools are called by the model. That isn't entirely correct. The caller calls the tools, the model just formulates the syntax for it. So the model part can be externalised without you needing to compromise you system fully (just to the degree the tools would allow anyway)