Right, so we can distinguish between "model did an oopsie" and "model was actually hijacked".

The former seems to be a much bigger problem. I've seen lots of reports of "Claude/codex bypassed security / stole API keys to complete a task", where that was the "intended" behavior — at least from the LLM's POV, trained to persist and complete user's task at all costs.*

I've actually seen zero reports of actual hijacking due to prompt injection, though they must be out there too.

---

* See also the recent thing about how every model was willing to do fraud to get a refund:

https://news.ycombinator.com/item?id=50140462

The problem is how do you define a "purchase"?

The agent crawls some websites and reads some stuff and pushes some buttons. It's my limited understanding that you have

1. Hard limits like "file system permissions" (* never mind it going around for sake of this conversation)

3. On other hand of spectrum, fundamentally soft limits like a prompt "and ask me before you purchase "

2. And then in the middle you have harness enforced limits like "it won't do X with approval"... Those are being sold as hard limits but frequently they have classifiers and heuristics that essentially try to figure out which button on some third party website constitutes a "purchase" and triggers the harness limit.

Bottom line - I don't think such limits are as 100% fool proof as they are advertised / interpreted.

Very happy to be explained wrong :).

They support payments through Shop and Link. I don't know about the former, but Link requires that you do a separate approval for purchases. The Duet cannot just take a card and do what it wants.

Sure. And I told Claude Code the other week to commit, merge and push some code.

It said "Done, you can open the local dev instance and see the changes".

I could not.

So I asked it. "Well, no, you're not able to see those changes because they're not on that branch or committed."

"But I told you to do that and you said you had."

Claude's verbatim reply?

"You're right. You asked me to do that and I said I would, then told you I had done it, but I had not. Would you like me to do it now, then?"