points by jadermcs 9 hours ago | 0 comments
Current tokens are very problematic, they are constructed by frequency, but frequency doesn't capture morphology and syntax that well. If the model learn how to tokenize internally it's way better then forcing an incorrect tokenization.
Aún no hay comentarios.