Hacker News

points by card_zero 2 days ago | 0 comments

So, I'll pick up on the weak points:

> As long as we have questions to ask, they are not theoretically constrained

> some infinities are bigger than others

Those are both limitations. They need conjectures and they need training data. Can they produce their own conjectures and training data, and feed off it forever, while staying meaningful and not being stuck in a loop (a very large set of repeating cliches) or descending into confused noise?

[–] bbor 2 days ago

Hmm, not sure how to respond, sorry — that’s the point of the entire comment. No these systems cannot think every possible thought, but neither can humans. We have no reason to think their cognition is unsustainable in a way that ours isn’t.

The end of your response seems to be about fears of “model collapse” from a few years ago, but I am convinced those are strongly oversold. For one thing, there’s no need to gather new training data ever again; the training data for Opus is clearly sufficient for instilling reason, and that’s a sufficient condition for AGI -> RSI -> ASI.