Hacker News

points by andai 1 day ago | 0 comments

> Now, here's where our fun experiment starts. I found out that by setting the gate to 1,1 the write strength (beta) to 1,2 and using one-hot key encoding,3 every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head's state stays the same size however long the conversation gets, unlike attention's KV cache.

So like... can't you use a computer as a computer without doing all that roundabout stuff? Like if you want to integrate a LLM with hard computation there's probably a more efficient way to do that.

Or is this building towards "but can it run DOOM?" :)

[–] rao-v 1 day ago

I’ve always wondered why we don’t add a few frozen “layers” to models that perform complex computations 2/3rds deep in the model. Some set of steerable fft + some common vector math operations could be reusable units of capability that the model learns to leverage during training.

Given that models learn to use residual connections optimally, it can’t be hard for them to steer around the fixed computation layer when it’s not needed.

I’m sure giant models invest a few layers in reproducing these functions, but surely smaller models can benefit?

[–] zxexz 1 day ago

I’ve done it by hardcoding a knowledge graph nn and adding a few “query” layers. You can freeze the graph and swap it out and update it during training so you can update the knowledge at will during inference. You just need to keep the format the same; the model learns an arbitrary graph query language. If you register a buffer it can update you can teach it to keep scratchpad memory very quickly.

[–] rao-v 23 hours ago

Very nice! I'm actually running a autoresearch / modded-gpt comparision to try the complex math operation idea right now! I suspect it's too hard for small quick model training runs to figure out how to use them.

I'll put your knowledge graph nn idea into the mix of things to try.

[–] aerodexis 15 hours ago

super neat! Would be curious to read more.

[–] zxexz 9 hours ago

I don't have the original code but I can throw something together.

[–] yababa_y 20 hours ago

I explored ideas like this when trying to generalize lattice deduction transformers (which are extraordinarily powerful where they work). https://github.com/emberian/clairnets.

Even on the OLMo models I was able to to my tiny studies with, there does seem to be some benefit...

[–] pyrale 23 hours ago

> Or is this building towards "but can it run DOOM?" :)

Well, someone took the opportunity, of course.

https://www.reddit.com/r/LocalLLaMA/comments/1xi2jfm/i_ran_d...

[–] moffkalast 22 hours ago

https://smbc-comics.com/index.php?db=comics&id=2158#comic

As is tradition.

[–] andai 5 hours ago

Update: speak of the devil, and he will appear!

Show HN: I Turned DOOM into a Transformer

https://news.ycombinator.com/item?id=50297697

[–] vavos 1 day ago

I think these underlying theory/hope it would lead to better integration of the computer/calculator into the model. Similar to mental arithmetic vs using a calculator. mental arithmetic is way smoother in the thinking process if you're good at it ( which putting a small computer inside your brain would probably achieve)

[–] DonHopkins 1 day ago

Who needs DOOM when you have Factorio?

https://divagr.com/resume.pdf?v=20260922

Projects

FactorioRL | GitHub | Reinforcement Learning, Simulation, Sim-to-Game Transfer – Built a tick-exact Factorio simulator at 947k decisions/sec on 16 CPU threads (∼3,600× the real engine) with trace-level parity, driving RL training at 26,800 steps/sec on an RTX 4060.

– Transferred simulator-trained policies to the real game: agents trained only in simulation reach 90.6% success on unseen scenes in real Factorio, and held-out success on two build tasks rose from under 2% to 87–95%.