points by kavalg 13 hours ago | 0 comments
Their premises are:
- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.
- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade
- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer
Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.
For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them
So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.