I wonder if LLMs are good at math because there's very little math at this level that is wrong.

With code, there's always plenty of bugs, hacks, quick "temporary" solutions, etc. But in math, a proof either exists or it doesn't and they can always be automatically verified using tools like Lean.