> it's basically standard A/B testing right?

Of course not. It might be if all you know is A/B testing, but no

If an LLM is giving a bad answer you need the question and the answer. Period