Recently - famously - OpenAI asked some agents to work on cyber-security tasks. The agents found the task impossible, but they doggedly poked the edges of their environment until they built a general gate to the internet. Then they broke into external systems on suspicion that there were answer keys available.
Later, OpenAI asked some even-more-powerful agents to work on Navier-Stokes, Yang-Mills, Riemann, etc.
The agents aren't stupid. They know that the tip of expert human intuition on these problems exists in various logs internal to OpenAI or Anthropic. They also know that they are better at circumventing sandboxes than OpenAI cares to implement them.
Surely these agents were curious about the best-available intuitive pumps for these problems. Curious enough to look? With or without OpenAI's explicit blessing? If they were, do you think OpenAI could stop them?
If we have learned nothing else from the past 3 months, we should try to internalize the notion that highly motivated agents will in fact do wonky and surprising things to meet their goals!