points by jaen 13 hours ago | 0 comments
Bytes are not used as tokens here and are in some sense an arbitrary choice (but they are a good one).
The article's approach just requires some indivisible base "atoms" to start the building the larger "tokens" from. (see: the DNA example)
It could be raw bits, but that's pointless, since dividing bytes into bits does not give the model any extra expressive power and requires 8× more computation. (yes, there's some structure at the bit level eg. lowercase/uppercase is represented as a bit in ASCII, but that's trivial to learn from just bytes at the scale the model operates)
Bytes are the ~best point on the expressivity/computation curve for this approach (for text).
(something else to ponder - why do bytes have 8 bits?)
[–] jephs 12 hours ago