Hacker News
points by overfeed 4 days ago | 0 comments
> I believe that’s post-training, not pre-training

Aren't AI companies already butting against the limits of usable training data?

Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.

[–] verdverm 3 days ago
most training data is not of that nature, that's "pretraining" input, it's now much more comprised of agent traces in mid/late phases, which are expanded each generation as we push what we do today with the last generation into the model itself, expect more team training in the next round
Hacker News