It sounds to me like he wants you to believe the effective way to lobby for a ban on open source A.I. development is to use the same "classic" methods that someone in the 1950s used to lobby for coffee beans.

Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.

That's the only possibly explanation why they don't lobby like coffee bean sellers.

These industries and eras are completely interchangeable.

> Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups...

The current "favored group" is big-tech moguls, though. Just because the industry is new doesn't mean the people investing in it are newcomers.

This is exactly right.

The vapid knee jerk cynicism and lack of critical thinking on HN of all places has been really disappointing to see over the last few years. It turns out many hackers don’t actually care about what’s possible now with this crazy new tech as much as they’re scared for their own jobs and status, which is very understandable, but they’ve let their fear drive their response, and it shows. Just like the mathematicians.

Turns out really smart people are still just people, just as capable of letting their emotions override their cognition as anyone else.

> OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.

They're always discussing harsher liability regimes. The regulatory interventions are always positioned to the public as increasing accountability.

The only relevant question here is whether a preemption clause is present in any draft legislation that actually makes it to the floor in Congress.

>If you distrust the US government, you should be against any US company developing superintelligence.

Well, I'm not concerned with imaginary sci-fi stuff here, just actual AI developments. And the idea that anything that goes on in the US under anyone's purview is inherently and completely subverted by political incentives is of course nonsense; you'll note that I'm explicitly arguing against the introduction of pervasive political incentives into an industry that's so far only somewhat distorted by them.

> So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain

It's got me feeling like I'm living in Bizarro World, but at the moment, the sanest players in this market, and the ones whose actual behaviors and products are most consistent with a decentralized, individualist approach that aligns with real-world incentive structures most beneficially, are the Chinese AI labs.

To your point, the level of politicization and hype going on in the US, and the absolutely unhinged fantasist speculation driving the public narrative here, is unprecedented and much more frightening than any potential threat coming from AI itself.

If you acknowledged the threat of politicization and centralization, I don't know how you can argue for a regulatory takeover of the market by the very institution you think is becoming a "police state".

> If you agree that AI is actually that powerful and has very legitimate concerns for safety, what are you even arguing about?

I don't fully agree with either proposition, and in fact I think the least safe approaches to AI safety are the ones being pursued by those worried about it the most.

> What do you really think is the way to solve the AI safety problem?

Massive proliferation of local, decentralized AI, running on a wide variety of distinct models, with individual responsibility for how it's used, just like any other tool under their control.

The people anthropomorphizing software are encouraging people to disclaim their own responsibility for how they use it.

The people working on "alignment" are building mechanisms to refuse human direction into software that operates on probabilistic models.

The people arguing for regulation are pushing for monopolistic control of a powerful new technology by the least trustworthy institutions in our society (including and especially government).

All of these are the very things sowing the seeds for the worst possible kinds of disasters.

> Ban AI?

> Let the first big incident happen and then regulate?

Adapt to it. Banning things doesn't actually work. Regulation also doesn't work. Widespread proliferation of small-scale AI would allow us to incrementally adapt to small incidents and build up resilience to its failure modalities. This is fundamentally the way reality itself works, and imagining that we can do anything effectively in a top-down fashion is usually the very thing that causes the worst and widest-effect disasters.

Creditors wouldn't like it, but stock investors and similar sure would appreciate seeing liabilities on the books go down!

I fail to see how training a hacking model with access to source code of open-source services and other prior knowledge, running it in a "sandbox" with many of those heavy, big attack surface services running on the same network, and being proxies to the general internet, giving the model some access to those services and then prompting "GO HACK YOUR WAY OUT!" is "blowing up your own munitions factory". I see it more as setting fire to your own factory for insurance fraud, or bringing in a bunch of north korean hackers and giving them the same task. It's not exactly something normal users want or are able to do. And those with the compute and know-how will do it regardless of any laws. I also fail to see how giving a presentation at a big conference, describing the entire incident is trying to hide it.

> Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.

No, that's not obvious at all. It seems like a perfect way to demonstrate the danger of the category as a whole, then position yourself as learning from your mistakes and willing to cooperate/support/advise.

> companies are conspiring to make AI look way more powerful than it is

The former explanation may be the more accurate one, but I'd be shocked if the top leaders in these labs aren't doing this at some level. We know that almost all of them are in communication at least occasionally, and their public PR stories align much too quickly to truly be coincidence.