We'd first need to establish that this side channel leaks usable information during regular use.

This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.

Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.

Very interesting quote.

yeah? the incidents were faked in order to create hype

I don't think anyone is arguing that they were faked.

> This type of elaborate conspiracy is improbable a priori,

What conspiracy? We're talking about specific people acting on their own motivations independently.

> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.

No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.

For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.

The quote from OpenAI illustrates this approach perfectly.

> Announcing that "we have no idea WTF we are doing" is not exactly the move,

It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.

> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.

Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.

"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.

> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.

"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.

The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.

My point was that if the government steps in and says these companies must slow down, they have an excuse to fail to repay some of their debts.

And, bam, competition eliminated.

They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.

Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.

No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.

Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.

If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.

Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.