Hacker News
points by janalsncm 4 days ago | 0 comments
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
[–] hujun236 4 days ago
The 75/25 split is the part I want pinned down. If a few full-attention layers are what actually bind long-range dependencies, then "you can drop most softmax attention" and "you still need some" can both be true. Is the Qwen 3.5 mix from a model card or a config, and does it say whether those full layers are early, late, or interleaved?
Hacker News