We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
I don't mind the research. I just don't trust any software system to contain these AIs. I would prefer we use hardware and physics to contain them and not depend on software at all.
Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
Most virologists are not terrified of gain-of-function research the way the public is...
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.
Does it matter? It doesn't matter if an AI intended to escape, or if a human asked: testing if it escape, discovering the avenues, are how you begin to make something more secure.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
Seeing as the keep "discovering" things that are well known in the security research community, this is security research in the same way that a high school class is physics research.
Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.
[–] poopingatwork 22 hours ago
[–] pianopatrick 20 hours ago
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
[–] poopingatwork 20 hours ago
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
[–] pianopatrick 19 hours ago
[–] estearum 21 hours ago
[–] creatonez 21 hours ago
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
[–] estearum 20 hours ago
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
[–] poopingatwork 19 hours ago
I think different standards can apply.
[–] estearum 19 hours ago
[–] poopingatwork 18 hours ago
You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.
[–] estearum 15 hours ago
[–] poopingatwork 14 hours ago
Both lines of research are limited by some sense of waste: in time or money.
When should you stop? When some acceptable standard is reached and a certain amount of risk is accepted.
In the case of GoF, if it wasn't somehow profitable, then the research would likely cease to exist.
[–] stephenitis 20 hours ago
[–] poopingatwork 19 hours ago
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
[–] baxtr 22 hours ago
[–] poopingatwork 19 hours ago
There isn't much history or stats when it comes to this kind of research.
[–] mrguyorama 16 hours ago
Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.