Once you understand what attention actually is, its pretty easy to adapt it how you need it.

https://news.ycombinator.com/threads?id=ActorNightly&next=49...

TLDR its not so much about the transformer, as much as expanding dimentionality. You can chose any which way to do so.