Hacker News

points by jephs 17 hours ago | 0 comments

So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.

There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...

I don't think a thorough survey exists.

[–] jaen 13 hours ago

Bytes are not used as tokens here and are in some sense an arbitrary choice (but they are a good one).

The article's approach just requires some indivisible base "atoms" to start the building the larger "tokens" from. (see: the DNA example)

It could be raw bits, but that's pointless, since dividing bytes into bits does not give the model any extra expressive power and requires 8× more computation. (yes, there's some structure at the bit level eg. lowercase/uppercase is represented as a bit in ASCII, but that's trivial to learn from just bytes at the scale the model operates)

Bytes are the ~best point on the expressivity/computation curve for this approach (for text).

(something else to ponder - why do bytes have 8 bits?)

[–] jephs 12 hours ago

Unicode bytes are totally bizarro unless you happen to be writing English or some other language with a Latin script. They are absolutely not the best choice.

[–] jadermcs 9 hours ago

Current tokens are very problematic, they are constructed by frequency, but frequency doesn't capture morphology and syntax that well. If the model learn how to tokenize internally it's way better then forcing an incorrect tokenization.

[–] vatsachak 16 hours ago

They are though. Bytes are raw data and the architecture chooses the tokens

[–] jephs 12 hours ago

Bytes are not raw data.

Vocalizations, hand writing, and keys strokes are raw data.

Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.

A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.