Hacker News
points by reacharavindh 1 day ago | 0 comments
Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.

This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.

Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.

The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.

[–] adamddev1 23 hours ago
This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit?
[–] reacharavindh 18 hours ago
My layman's understanding is that these models are very capable pattern matchers. When they need to do X, they go on a hunt for every pattern they could find and try to satisfy solving X. If that is not deemed good enough? make them dig harder and they take a different route to find patterns to solve X. Now, their arsenal is everything that they have been trained on(which is a lot!). This ability comes in extremely handy to apply existing techniques and patterns in ways we humans didn't quite think of.. because we are domain driven. A mathematical model about finding a good enough approximation of something at much lower cost is perhaps well understood in its applications in mathematics or whoever got to see those results. But, these LLMs, when pushed harder and harder might bring that to apply and solve an entirely different problem like for example, classifying whether some event is fraudulent. When it does such things, it _feels_ novel because it ended up using something we haven't thought of yet. But, it got there by brute force, stroke of luck, or some random path. I guess this is the reason why the frontier labs run those crazy swarms of AI agents to uncover all such paths turn them into chartered thinking vectors instead of random attempts.

I like to believe we humans still retain the ability to think completely outside the box and discover new phenomena and understanding aided by these pattern machines. In other words, these models are great at running combinations through patterns and we humans go one level higher and add those new patterns for them to accelerate us even more. We determine what to solve and the LLMs help us solve it faster.