points by overfeed 4 days ago | 0 comments
> I believe that’s post-training, not pre-training
Aren't AI companies already butting against the limits of usable training data?
Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.
[–] verdverm 3 days ago