Hacker News
points by augment_me 1 day ago | 0 comments
Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.

Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.

Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.

[–] yorwba 21 hours ago
The whole point of this architecture is that it can also make use of multi-char patterns, but unlike BPE, they aren't hard-coded. And you can still use speculative decoding. Whether it works better or worse is something that you have to actually measure.
[–] nivter 22 hours ago
> you will need a very domain-specific use for this to be useful imo

The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.

A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.

[–] kavalg 14 hours ago
Their premises are:

- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.

- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade

- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer

Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.

[–] augment_me 8 hours ago
But you lose the semantics you choose to not care about. BPE is equally equipped to not fuse single chars if they have downstream value. Most modern tokenizers are already trained with the target text corpus in mind.

For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them

So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.