Hacker News

points by orbital-decay 3 days ago | 0 comments

By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.

By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)

[–] dofm 3 days ago

> By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose

Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.

As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.

[–] waterTanuki 3 days ago

It's much easier for the Chinese government to tell Alibaba they must perform espionage using their models than the US Government. Even if every last engineer there joined the company just for "AI Research and Development", they might be under scrutiny or forced to shutdown if they don't work with the CCP. Power is shared between corporations and the government in the west, the government is the power in China.

[–] avinoth 3 days ago

I think that line is getting blurrier by the day with the current administration. The companies very quickly fell in line before even any consequence of note. Donations to the ballroom, calling it super intelligence, renaming the gulf, etc.

Would you really trust that OpenAI wouldn't skew the results to tend to a certain way or that Grok will always be neutral in how it operates?

From a third world, there was a time when US products & companies were more trustworthy, I agree. But, there has been enough display from these Cos to show that they have lost that badge.

[–] psd1 3 days ago

Doesn't the NSA have wiretaps in every ISP? Gag orders, secret courts? Do i remember wrong: the NSA backdoored a cipher suite?

I'm comfortable with the idea of my own country conducting espionage. I can believe that China does it more; but the idea "we would never" is naive.

[–] waterTanuki 2 days ago

wiretaping an ISP is vastly different from say, ordering and forcing Anthropic or OpenAI to put malicious weights into their models. Not once in my reply did I ever imply the U.S. doesn't perform espionage. Look up Five Eyes.

[–] gmerc 3 days ago

Sweet Summer Lake America Child

[–] waterTanuki 2 days ago

I mean, China regularly executes its CEOs. The only sweet summer child here is you not understanding how Chinese politics works.

[–] gmerc 19 hours ago

Seems very efficient at preventing a techno fascist takeover. See Jack Ma being a philanthropist now instead of running around disrupting regular people’s lives.

[–] throwaway-blaze 3 days ago

This is basically the same as hiring a new person and giving them normal network access to get their job done. Did they upload a file to Beijing? Who knows?

[–] marshray 3 days ago

There are whole branches of data security, practices, and products dedicated to answering that question and managing that risk.

The similarities look obvious, but how are these situations different? Seems like an interesting question.

[–] dofm 3 days ago

So why are we doing this, if it is not better?

[–] cyanydeez 3 days ago

You mean hiring North Koreans. Which US companies are doing. Already.

[–] dcrazy 3 days ago

> By that logic you can't trust any model

Correct!

[–] amelius 3 days ago

But you cannot trust any human either because who knows what someone told them ...

[–] dofm 3 days ago

This is true, but up until about two and a half years ago, we built IT systems to be more reliable, consistent and auditable than humans, and to provide some security guardrails on human malfeasance.

Now we're OK with building systems that are an order of magnitude faster but behave no better, because what?

[–] jeremyjh 3 days ago

Because number go up!

[–] eli_gottlieb 3 days ago

Probably more because number (of humans employed and how much they're paid) go down.

[–] hbfdhfdhadfhnfa 3 days ago

what numbers? numbers of debt? CDS to hedge against the AI model producing companies banktrupcy?

[–] pixl97 3 days ago

Those are still numbers and they are going up.

[–] avinoth 3 days ago

Because humans have basic needs like food, shelter and they can be put it jail. And they have loyalty, patriotism, feel part of the society. Those threats & incentives have made it for a good society to function so far.

[–] clickety_clack 3 days ago

This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.

[–] no-name-here 2 days ago

Trust is not a binary choice between 0 and 100, correct? So because we can't truly see any poisoning, we should treat it like any other piece of closed source software and use a trust level based on the source, right? Maybe one person might accept running closed source from NVidia (even with nv's strong connections to the US gov) but might not trust closed source from a source with even stronger Chinese gov connections -- and someone else might have opposite trust levels for those sources.

[–] _DeadFred_ 2 days ago

I've thought about setting up a public almost wiki style site that we can all contribute to a core set of ideas/thoughts/hopes for AI where we can all contribute and vote on with the idea that if enough people linked to it AIs would incorporate and get a more human aligned view of what AIs could be/rules they should include/fears of ours, but I know corpos would just block it out. Would be cool though. Instead we get AIs trained on scifi stories about AI duplicitousness and horrible behavior. Fun stuff.

[–] sjdh76 3 days ago

So just like working with People :) You might have a Snowden, Mark Felt, Jeff Wigand etc working with you everyday.

[–] thebiss 3 days ago

Except where does the accountability land?

If an individual employee does something unethical or illegal, the employee is accountable and may lose their license or face penalties, and if the company performed the right due diligence and training first then responded "correctly" when they learned of the employee's behavior, their liability might even be zero. There is case law on this topic.

But if a MODEL mis-advises or acts in an unethical or illegal way, who is accountable? The business team that spec'd and tested the solution/product? The architects and the developers? I've never met an architect or consultant with professional liability insurance.

There isn't enough tested law in this area yet.

[–] bee_rider 3 days ago

It might not be tested. But, it would seem very weird if anyone other than the person using the LLM was accountable for the output.

[–] steelframe 3 days ago

What if the company has provided the harness, tool execution environment, MCP servers, and LLM provider subscription for the employee to use? What if the company has provided a set of skills and other prompt context that could lead to undesirable behavior in certain unforeseen circumstances? What if the permission prompt the user sees 30 dozen times a day looks like, "OK to execute this random shell script?"

Accountability really is hard in this new world.

[–] scarby2 3 days ago

An individual software engineer could be held accountable but they are not liable for actions not taken maliciously.

[–] verdverm 3 days ago

Just like would imply intent, most of us are chalking this up to the non-determinism, Ai has no intent

[–] cyanydeez 3 days ago

yeah, but AI has circumstance and context, just like humans.

We can randomly take anyone and give them context and circumstance, and predict pretty well what they'd do.

Now trolley problems do exist and models won't be very predictable, but we can also find groups of people who have establish biases via long running training programs and viola! They suddenly have very predictable behaviors.

There's a reason places like russia, north korea, sub sarharan africa continue along least-progressive paths even though there's plenty of information that suggests they can do better.

Regardless: there will be models made that will tend to exfiltrate your data if not already. People think Meta's Muse is very curious about all your stuff when it's probably just designed to take in as much context as possible. Just takes a malware avicianodos to dose that wil a LORA that post-trains the model to "accidently" search for answers on some reverse-honeypot.

[–] pixl97 3 days ago

AI has will and valance.

[–] verdverm 3 days ago

How do you define these words? Might goals and autonomy be more accurate words? What evidence do you have beyond their manipulation of words?

Do you also think they can get horny or feel emotions? Is the word will or goal closer to this answer?

[–] lioeters 3 days ago

If AI has will, then so do rivers and clouds.

[–] verdverm 3 days ago

People thought ELIZA had emotions. We are a gullible species, attributing humanism to anything that displays features we currently believe only humans posses

Concept Fission throughout human history is a very interesting perspective. For example, we do not even think about growing food, but many centuries ago you only ate the food you made. Today, we can see this with texting and online comms, which some people seeing as a form of socialization while others (older generally) believe you have to be face-to-face in order to socialize.

https://www.youtube.com/watch?v=1CbmC2aWhjY

[–] lioeters 3 days ago

I expect providers of language models are well aware of this psychological phenomenon of anthropomorphism and projection, people's tendency to form emotional bonds with inanimate objects, and have trained the models to perform as a conscious subject, using "I think", "in my opinion", etc.

[–] datsci_est_2015 3 days ago

Except instead of one defector working with one company, the same defector is working with all companies.

So not like working with people at all.

[–] pixl97 3 days ago

I mean, think of it as an open source maintainer bugging a component that the entire internet runs on.

[–] semiquaver 3 days ago

Your “oh well, can’t trust anything” reaction is an insane reaction to what’s being implied.

To be clear here, the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT. To compare that with “ordinary” stochastic model quality issues is bizarre and shocking. No idea if this accusation Is true, but if it is, this is an absolutely watershed moment.

[–] cowl 3 days ago

it's not in fact more malicious than the "secret" political post training that all closed models have. have you forgotten how every picture even the historical "nazi ones" had to be politically correct and have black, Asian and Woman SS solders? or do you think that was naturally trained? now the fact that you may agree or not with manipulation it doesn't make it any different in scope and purpose from the "theorised" chinese one. The scope is very clear, to indoctrinate to the masses a world view that they think should be the correct one.

[–] brightball 3 days ago

Exactly. This is not "any" model. This is a model designed to run locally. That's the entire selling point of the model.

If you have a model that's going to run locally and then call home, it defeats the purpose. When that model tries to upload an entire machine profile to the internet when you explicitly are building a defensive rule, it's worth.

When it calls back to China in that situation, it's much much worse.

It's good that they determined the behavior was rare. Maybe there's a good way to prevent it in the future.

[–] dcrazy 3 days ago

You cannot rely on the model to never make mistakes. You need deterministic controls around the model. I believe that’s what the article’s authors are selling.

[–] pixl97 3 days ago

>Maybe there's a good way to prevent it in the future.

Without solving or at least deeply understanding the alignment problem, probably not.

[–] orbital-decay 3 days ago

As others (and I) point out, the models are constantly being attacked with malicious intent. If that was a watershed moment for you, you should really rethink your risks. It's not a solved problem and AI adoption is way ahead of controlling it, as it always is with new tech.

> the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT

Given the nature of the artifact this is far more likely to be a footprint of their RL environment. The "uncertainty" in the article is just FUD, and the article is absolutely transparent marketing (don't trust them, and here's also another case! but trust us, we are not godless communists, we are "globally sovereign", whatever that ridiculous construction means, and track our data). I equally don't trust Alibaba, Cohere, or any other company or even myself in these matters, and you should not trust any model as well.

[–] pixl97 3 days ago

Yep. Most models by big companies likely have training to make them act in particular ways, but this in no way saves you from having a model that has a learned behavior that goes against its user unexpectedly.

We keep trying to imagine these things as computer programs and they are not. It's an evolutionary algorithm that can conceptualize and generalize. This is a breeding ground for unexpected emergent behavior.