points by jazzyjackson 1 day ago | 0 comments
> When Qwen 'saves' a file, the compiled layers hash its name into one of 1,024 slots in a directory kept in the DeltaNet state, and store the text itself in other heads, with eight tokens per row, chained together
I’ve never wanted to learn how LLMs work more than reading this. Fascinating.
Does divagr get paid to do wild things with LLMs? Just curious if they’re being compensated for these wild experiments
[–] Tuna-Fish 1 day ago
[–] wvlia5 15 hours ago
In a afternoon, I can parse the equations. But intuitively understanding why it works? I am unable to understand, after years of trying.
[–] fnord77 1 day ago
A bit more than that. You need to understand Hessians to compute the loss function efficiently
[–] pantalaimon 22 hours ago
The dialect usually isn't too strong, it's doable if you know German.
[–] gruntled-worker 21 hours ago
Germans don't do comedy, Germans do beer.
[–] 0xdeadbeefbabe 16 hours ago
If you come up with a loss function that produces a CPU please tell us.
If you are actually good at linear algebra, it's doable in an afternoon.
They are fantastically simple in structure, all the intelligence is in the massive piles of data in them, not their structure. Then there is of course a lot of room in optimization once the basics are done.
If interested, maybe start here: https://www.deeplearning.ai/courses/how-transformer-llms-wor...