Hacker News

points by WarmWash 4 days ago | 0 comments

The information to get tea was in the weights (and a mistake, it should have pulled lemonade).

Once pulled it added "iced tea" to its to-do list, which gets put into context. The model then couldn't escape the idea.

[–] zby 4 days ago

"iced tea" in context was a symptom - it might influence the downstream - but it is hard to establish that. The root problem was in weights - naming it 'context poisoning' is poisoning our reasoning context :)

[–] WarmWash 4 days ago

You are confusing two different things.

It hallucinated that it needed tea.

Once that hallucination got into context, the context was poisoned.

If you don't want to believe that, then I cannot force you to, you can believe whatever you want.

But I (and the author) am explaining an otherwise well known problem with hallucinations getting into context.

[–] zby 4 days ago

My point is that calling it context poisoning is confusing because of two things. First - it is hard to establish how causal it is for the downstream. Second - cleaning the context would not fix it - because it would be poisoned on next round - you need active measures to fight it.

It is also a bit tongue in cheek - if you did not notice - because of course all of this can be explained if needed even if the name alone suggests something else.

[–] WarmWash 3 days ago

You are still confused here.

On the next round, it's likely to correctly determine lemonade.

Hallucinations are usually outliers. So 1 in 10 times it gets confused and picks tea. Those 1 in 10 times end up with poisoned context.

The models are not deterministic by design. Identical inputs produce different outputs. So if you roll a bad output, it can poison your context. Flushing the context can easily fix the problem (unless like the in the pokemon harness, you roll over a brief of the previous context)

[–] podocarp 3 days ago

But the original claim was tea was in its weights. So more like 9 out of 10 times it will still say tea and get stuck.

[–] zby 3 hours ago

Yeah - it depends on the frequency. I think the other guy is right that when it happens with low frequency then calling it 'context poisoning' makes sense - because cleaning the context mostly fixes it. I still would be very careful about using it - because it might be interpreted as if the poisoned context was the root cause - it might be a cause for the downstream processing - but it itself is caused by something else - so it will recur.