For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
[–] mapontosevenths 4 days ago
[–] magicalhippo 4 days ago
I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.
Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).
Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.
[1]: https://github.com/Infatoshi/grassmann-flows/
[2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog....
[–] mapontosevenths 3 days ago
I still don't think it will work, but I'm also fairly confident that nobody has actually reproduced it well enough to know that for sure.
[–] imtringued 4 days ago
Not sure what the point of the paper's title is if the resulting model does not match the transformer.
[–] magicalhippo 4 days ago
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.
[–] imtringued 4 days ago
I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.
If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.
I think you get what I'm saying, but when they compute the gate they do:
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.