Hacker News

points by poopingatwork 22 hours ago | 0 comments

It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.

[–] pianopatrick 20 hours ago

We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.

I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.

[–] poopingatwork 20 hours ago

Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.

I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.

A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).

And so poking on these issues, regardless of who asked to escape the box, is still worthy research.

[–] pianopatrick 19 hours ago

I don't mind the research. I just don't trust any software system to contain these AIs. I would prefer we use hardware and physics to contain them and not depend on software at all.

[–] estearum 21 hours ago

Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?

[–] creatonez 21 hours ago

Most virologists are not terrified of gain-of-function research the way the public is...

- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board

- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined

- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds

- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true

- It is quite useful for research, a lot is learned by doing it

[–] estearum 20 hours ago

I didn't ask "are most virologists terrified of gain-of-function research the way the public is"

I'm not even necessarily against GoF research, and I mostly agree with all of your points.

I'm asking about GP's logic.

[–] poopingatwork 19 hours ago

Software sandbox environments are far easier to manage than real world environments where life operates.

I think different standards can apply.

[–] estearum 19 hours ago

I don't see how the "you should want to know all avenues of exploit" goal is different depending on how many avenues of exploit there are

[–] poopingatwork 18 hours ago

They're just relevant to each other.

You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.

[–] estearum 15 hours ago

Right, so why would that not be a justification for essentially unbounded GoF research?

[–] poopingatwork 14 hours ago

How much should you spend on security?

Both lines of research are limited by some sense of waste: in time or money.

When should you stop? When some acceptable standard is reached and a certain amount of risk is accepted.

In the case of GoF, if it wasn't somehow profitable, then the research would likely cease to exist.

[–] stephenitis 20 hours ago

How do they ensure all comments speaking about the exploit stay out of next train run?

[–] poopingatwork 19 hours ago

Does it matter? It doesn't matter if an AI intended to escape, or if a human asked: testing if it escape, discovering the avenues, are how you begin to make something more secure.

Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.

[–] baxtr 22 hours ago

Insurers would disagree. You don’t need to know mechanism to understand / model risk outcomes.

[–] poopingatwork 19 hours ago

Insurers use stats, history, to determine future risk. Their entire business model is built up on this fact.

There isn't much history or stats when it comes to this kind of research.

[–] mrguyorama 16 hours ago

Seeing as the keep "discovering" things that are well known in the security research community, this is security research in the same way that a high school class is physics research.

Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.