points by zby 4 days ago | 0 comments
My point is that calling it context poisoning is confusing because of two things. First - it is hard to establish how causal it is for the downstream. Second - cleaning the context would not fix it - because it would be poisoned on next round - you need active measures to fight it.
It is also a bit tongue in cheek - if you did not notice - because of course all of this can be explained if needed even if the name alone suggests something else.
[–] WarmWash 3 days ago
[–] podocarp 3 days ago
But the original claim was tea was in its weights. So more like 9 out of 10 times it will still say tea and get stuck.
[–] zby 3 hours ago
Yeah - it depends on the frequency. I think the other guy is right that when it happens with low frequency then calling it 'context poisoning' makes sense - because cleaning the context mostly fixes it. I still would be very careful about using it - because it might be interpreted as if the poisoned context was the root cause - it might be a cause for the downstream processing - but it itself is caused by something else - so it will recur.
On the next round, it's likely to correctly determine lemonade.
Hallucinations are usually outliers. So 1 in 10 times it gets confused and picks tea. Those 1 in 10 times end up with poisoned context.
The models are not deterministic by design. Identical inputs produce different outputs. So if you roll a bad output, it can poison your context. Flushing the context can easily fix the problem (unless like the in the pokemon harness, you roll over a brief of the previous context)