Hacker News
points by hujun236 4 days ago | 0 comments
The 75/25 split is the part I want pinned down. If a few full-attention layers are what actually bind long-range dependencies, then "you can drop most softmax attention" and "you still need some" can both be true. Is the Qwen 3.5 mix from a model card or a config, and does it say whether those full layers are early, late, or interleaved?
Aún no hay comentarios.
Hacker News