It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
[–] diqi 1 day ago