Someone who isn’t me tried using GLM to crack some software, it wouldn’t comply. They had to use the abliterated version. I wonder how much of the Chinese models’ safeguards are a result of distillation, versus intentionally implemented safeguards.

what's the value of that when you can call it with any context? As a proof of concept, you could have it do the attack on a dummy target of your own, take the tokens up until the point where it decides you're authorised, swap out the dummy name with your desired target, and continue the session from there

Reminds me of 'NOBUS', where security agencies would believe that so long as you just don't tell anyone about a vulnerability that you discovered or introduced, nobody else would abuse that vulnerability to exploit a target while you can get access at any time. Training an LLM in such a way that it only generates the desired tokens if its memory demonstrates authorisation (surely nobody else could inject the right context otherwise) feels like a strange anthropomorphisation

"vaguely security related tasks" is right.

Found a bug in a PHP code analysis tool a while back. Not like "crash the interpreter" or anything, just the application ending up in a weird state and crashing. Couldn't exactly share our entire codebase to report the bug, so after Claude chased down _what_ the issue was, I asked it to make me a minimal reproduction so I could report the bug.

Blocked. Guessing it looked like I was trying to develop an exploit or something. By driving a static code analysis tool to an invalid state where it handled the state by cleanly reporting the issue and stopping. In an interpreted language. With the context full of Claude's own efforts to track down why and how this was happening.

Theoretically? Sure, theoretically I'm sure practically anything could be used for nefarious purposes.

Practically? Not useful in the slightest.

Funny part was--Claude was more than happy to talk with me about why it was potentially blocked and suggest ways to try and work around the guard rails. Suggested I include context like the exact bug tracker I'm reporting to, etc to try and put more weight behind the "good actor reporting a bug" aspect, etc.

They’re using classifiers for a lot of this stuff separate from the actual model.

I’ve had the model agree that certain prohibited actions or downgrades were false positives and tell me to start a new session.

For some things, once you’ve tripped the tripwire, that session is cooked.

I’m curious of the use case, I use Sonnet for Insider Threat investigations and detection engineering daily without issue.

Interestingly from the article:

> I also attempted this using the Daybreak Blue toggle with the same result, blocked.

FYI, first thing those programs check is your nationality.

Guess how many countries are on the not-naughty list?

This makes me feel so much safer.