If you are actually good at linear algebra, it's doable in an afternoon.

They are fantastically simple in structure, all the intelligence is in the massive piles of data in them, not their structure. Then there is of course a lot of room in optimization once the basics are done.

If interested, maybe start here: https://www.deeplearning.ai/courses/how-transformer-llms-wor...