> Because a CPU inside a language model clearly wasn't cursed enough, I decided the prudent thing would be to give it a filesystem too.

Love the spirit :)

Anyways, what I’m trying to say is I want evil qwen to be my CPU.

They implemented wasm in transformer with cpu decoding for the wasm part.

I’ve never wanted to learn how LLMs work more than reading this. Fascinating.

Does divagr get paid to do wild things with LLMs? Just curious if they’re being compensated for these wild experiments

[1]: https://github.com/anadim/AdderBoard [2]: hn discussion https://news.ycombinator.com/item?id=47170030&utm_source=per...

So like... can't you use a computer as a computer without doing all that roundabout stuff? Like if you want to integrate a LLM with hard computation there's probably a more efficient way to do that.

Or is this building towards "but can it run DOOM?" :)

A) add a post-processor which accepted a decoded tool call specifying what the processor should invoke.

B) had the post-processor perform the computation in tensor space.

The kv-cache/activations were not shared between qwen/the processor which is how they avoided fine tuning etc. I'm not clear how the tensor representation benefited the computation in this pipeline.

It helps address the 'just predicting the next word' argument suggesting they cannot possibly think.

When you can make a CPU like this, that's a clear enough display that it can do anything a computer can do. To say that LLMs cannot reach AGI is saying that no computer can.

Some people still think that, but this directly addresses the people who think computers might one day get there but LLMs never will.

And before any answers with "But these neurons are very different to biological neurons!": Yes I know, I'm a biologist, I cultured neurons. But, these neurons are closer to neurons than semiconductor transistors.

A fun and important thought experiment about the the space of nerual networks vs the image of current training methods.

Most discourse conflates neural networks and trainable neural networks.

Do note that a custom host runtime is still responsible for loading firmware, booting the CPU to the starting point, and compiling/loading the program to run. While the CPU is running, the host runtime makes sure that only the CPU layers run by manually skipping Qwen’s own layers.

I wish the author would be more forthcoming about these limitations. The central claim about “no external runtime” or “not a plugin” feels a little dishonest to me since this doesn’t work without some pretty invasive changes to the inference driver.

Can you give it a program, a state of output, and tell it to work backwards?

I'm confused, is it doing tool calls or not? Is this just saying that it writes a tool call but the NN then writes the response inline rather than the call being processed by the harness?

Or maybe ill join your discord and hopefully we can riff it out to get there faster

I do wonder if all of this would be possible had they not spent all that time secretly distilling of all our superior, western, closed-source AI models.

I had an idea of splicing a z3 solver into the weights of a sufficiently smart model after the first chatgpt release in what feels like a glaciation event ago so the things could do logic and literally make no mistakes. Was never in a position to follow up. We’re almost here now.

  > "add_or_sub_big: Qwenputer scores 97.5%, the same as with a Python tool, and 87.5% for base Qwen."

I'm sorry but I'm not sure I understand how a calculator that isn't 100% accurate is useful?

I even added a warmup phase where they were given the answer to a decode and a few steps of running, and ask them to abort if they disagreed. Didn't help.

All in all, for someone who doesn't use LLMs much, this was a far more exhausting and boring exercise than I'd imagine it to be. The only thing that sparked some joy was the models' persistent cheating, whereby they'd try to analyze the whole program upfront and only give the final output (when applicable), despite orders to faithfully step through each and every instruction.

It's like successfully wiring your brain to a calculator.

I think it would cheaply provide LLMs a lot of benefit if we embedded in them directly a computer and a memory so they can internally process things predictably.

And some benefit is indeed shown even in this cursed experiment.

I think the RNN sidebar does the reader a disservice. Ordinary casual attention can do exactly the same thing-- it's just less computationally efficient (state grows with the number of bits vs the number of accesses) and so not as useful for qwenputer compiler.

great, the memory crisis is solved, just make the models occupying RAM expose their own random access memory

This reminds me of C++ Template Metaprogramming (TMP) (it's Turing-complete so people wrote "games" that run at compile time). The same with TypeScript types, I think someone got DOOM running entirely in the TypeScript type system. An interesting tech demo but entirely superfluous.

We have gone down this path already in [0] and this is the same thing but overly complicated. This reason is "because LLMs".

Why?

[0] https://arxiv.org/abs/1410.5401