> So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).

Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.