Look up what some Chinese LLMs are doing, e.g. mixing some attention mechanism with linear attention layers. Mamba2 layers, Gated Delta Net (2), Kimi Delta Attention and recent Raon Delta Attention (Gated Delta Net 2 + improvement from Kimi Delta Attention)

You are likely to find kernels for Gated Delta Net 2 as Nvidia had a paper on that.

Once you understand what attention actually is, its pretty easy to adapt it how you need it.

https://news.ycombinator.com/threads?id=ActorNightly&next=49...

TLDR its not so much about the transformer, as much as expanding dimentionality. You can chose any which way to do so.