Hacker News
points by anon291 2 days ago | 0 comments
of course it has a read write head. In fact it has N read / write heads equal to the size of the context since each latent state is updated with the results of the cross attention mechanism at each step.

Alternatively, you can view the '<unknown>' token that is dynamically driven to the embedding of the output token to be the write head. Either way, the simple truth is that you can train a transformer system (i.e., an iteratively applied transformer) to simulate an arbitrary turing machine.

[–] geraneum 2 days ago
There’s no such thing as a read/write head in transformers. You can use it as a metaphor but that’s a leaky metaphor, hence all these misunderstandings.

> In fact it has N read / write heads equal to the size of the context

And what is the asymptotic notation when computation scales linearly with the size of context?

> Either way, the simple truth is that you can train a transformer system

Which still doesn’t mean they have a read/write head or have O(1) complexity for token computation with “infinite context”.

[–] anon291 1 day ago
> There’s no such thing as a read/write head in transformers. You can use it as a metaphor but that’s a leaky metaphor, hence all these misunderstandings.

The input to a transformer is a list of embeddings. The output is a list of embeddings of the same length. At the end of the day, you can represent the entirety of the transformer's matrix multiplies and such as a DFA (IEEE 754 float multiplication and addition is a simple DFA, and you're just scaling this up that many times). Thus, you take a list of embeddings (i.e., the tape) and output another, modified, list of embeddings. You have successfully read (ingested) and written (the transformer output) N positions on a tape.

[–] anon291 1 day ago
While we were having this discussion, someone went ahead and did this:

https://divagr.com/blog/qwenputer