I did indeed mean it that way, but I agree with you. It sucks, but it's just the unfortunate reality of the world. Many of the most disruptive and successful startups made it there by ignoring legal or moral limitations.

Their core selling point is customer trust that their product works as advertised.

From my interactions with them, is that they really care about customer success, and everything including AI is a means to an end to them.

I think building a cool product with undefined reliability and capabilities, then selling fantasies about what it will possibly be able to do down the line isn't really what they do - or what their customers are paying for.

Incentives can produce locally rational and globally harmful behavior. (See also: "Moloch", although I'm less of a fan of the concept lately since it focuses exclusively on the harmful aspects of emergent systems, deleting the neutral and positive ones in the process.)

A relevant metaphor is the nuclear arms race. Nobody wants a world full of nukes pointed at each other, but if the other guy is gonna do it...

The argument in this case is, we can't slow down, because China certainly won't, and do you want those guys to build the machine God first? So it's "rational" to maximize p(accidental_skynet).

If the people that do stuff well fail, what does that leave us with?

I interned there for a summer working on evaluating text generation and despite the fact that I used no consumer data whatsoever, but rather a well-known public research dataset, I also couldn't egress from the machines I ran training on. Not even reports on the training (admittedly, this was 2021 when they had just introduced this new system). So it was even more of a damper on research progress than you might imagine.

> That they even bothered to do all that stuff is good. That others don’t is bad

good and bad isn't what Meta and AWS is running with. If AWS trains willy nilly on user data like Meta does, AWS is toast. AWS is already taking a huge risk by now using their own take of AWS FDEs competing against their established ecosystem of consulting companies that used to recommend all 583 of AWS products to host a static webpage. Turning of more customers by training on their data is too much of a risk. That's the "good" - a risk analysis.

Meta doesn't care if you stop using Facebook, Whatsapp etc. Frankly, if you are embedded in the Facebook, Whatsapp ecosystem, you don't have a way out as a normie. Even if you are technically capable, good luck publishing your custom Mattermost build for iOS.

Meta has a significant amount of investment in hardware, originally due to Metaverse. The way to monetize that investment is inference and training. That's a do or die decision for Meta. Meta isn't releasing open-weight models because they are "good".

Meta is releasing open-weight models for the same reason Alibaba, NVIDIA or IBM does. Or Google releases Gemma.

There's no "good" or "bad" - the moment Alibaba, NVIDIA, IBM or Google's objective functions change, "Do no evil" will quickly morph into "Do as much evil as necessary without the costs of getting caught exceeding E(outcome)"