points by _aavaa_ 4 days ago | 0 comments
I believe that’s post-training, not pre-training.
[–] overfeed 4 days ago
[–] verdverm 3 days ago
most training data is not of that nature, that's "pretraining" input, it's now much more comprised of agent traces in mid/late phases, which are expanded each generation as we push what we do today with the last generation into the model itself, expect more team training in the next round
[–] verdverm 4 days ago
yes, just post training, which is the most expensive phase now as I understand it
Aren't AI companies already butting against the limits of usable training data?
Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.