I’ve done it by hardcoding a knowledge graph nn and adding a few “query” layers. You can freeze the graph and swap it out and update it during training so you can update the knowledge at will during inference. You just need to keep the format the same; the model learns an arbitrary graph query language. If you register a buffer it can update you can teach it to keep scratchpad memory very quickly.

I explored ideas like this when trying to generalize lattice deduction transformers (which are extraordinarily powerful where they work). https://github.com/emberian/clairnets.

Even on the OLMo models I was able to to my tiny studies with, there does seem to be some benefit...