Hacker News
points by LoganDark 1 day ago | 0 comments
How do you ensure that subset of the weights will continue to be accessible through the others?
[–] hnewsnemo 19 hours ago
Not a holistic answer but one example: There are specific fine-tuning techniques like Orthogonal Subspace Finetuning (https://huggingface.co/docs/peft/package_reference/osf) that target 'unused' parts of the weights so you can keep highly informative parts of the weights as-is.
[–] im3w1l 1 day ago
I don't understand your question.
[–] LoganDark 21 hours ago
Assuming for simplicity that you're talking about locking layers. Each layer influences the input to the next layer, so locking a subset of layers doesn't necessarily mean that you'll continue to see the same results in the output. Not sure if you meant something more granular or otherwise, but I don't see how it's possible to simply freeze a part of a model in order to retain a particular capability.
[–] im3w1l 15 hours ago
I meant taking all the weights that are part of the cpu and locking those, and only those.
[–] LoganDark 15 hours ago
Ah wait, you mean you'd intend the other layers to be able to evolve to use the CPU internally, so all you need is to keep the CPU functional somewhere in the model? That may work, dunno.