Look up what some Chinese LLMs are doing, e.g. mixing some attention mechanism with linear attention layers. Mamba2 layers, Gated Delta Net (2), Kimi Delta Attention and recent Raon Delta Attention (Gated Delta Net 2 + improvement from Kimi Delta Attention)

You are likely to find kernels for Gated Delta Net 2 as Nvidia had a paper on that.