Hacker News

points by oofbey 4 days ago | 0 comments

They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.

Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.

[–] HexPhantom 4 days ago

I think there's a middle ground between "attention is obsolete" and "this proves nothing"

[–] oofbey 3 days ago

You’re right I’m overly dismissive. But there are dozens of papers like this showing some more efficient alternative to attention. There is plenty of evidence pointing to this paper going nowhere.

Efficient alternatives have been coming out for 10 years now. Flash attention is the only one that has really moved the needle - because it’s not an algorithmic compromise. It’s just a smarter way to put the parentheses around the linear algebra operations.