You’re right I’m overly dismissive. But there are dozens of papers like this showing some more efficient alternative to attention. There is plenty of evidence pointing to this paper going nowhere.

Efficient alternatives have been coming out for 10 years now. Flash attention is the only one that has really moved the needle - because it’s not an algorithmic compromise. It’s just a smarter way to put the parentheses around the linear algebra operations.