Hacker News

points by curt15 16 hours ago | 0 comments

> How much can we trust a system when nobody has experience of building it this way?

If humans are no longer reading the LLM outputs and internalizing the codebase themselves, there's no longer much point to source code. The coding assistant might as well take a natural language prompt and directly hand you an ELF binary that passes a test suite. Are companies ready for that?

[–] graemep 15 hours ago

There is a point to source code:

Lots of tools for analysing and debugging it

It can be cross platform - the same source code will run on x86 and ARM, and in many cases on Windows and Linux.

Changes have predictable consequences because compilers and interpreters are deterministic.

[–] luc_ 16 hours ago

I think some definitely are. Especially if there is a system that sits on top of that and pokes and prods the binary to see if it behaves like it's supposed to.

[–] user43928 15 hours ago

That's simply not true.

Until spec files can be worked with so efficiently that we can just regenerate the binaries from them for every change, what you said makes little sense.

You're going to need an intermediate portable form of storing the program. It so happens that all the tools and AI are optimized to work with source code.

[–] stvltvs 15 hours ago

Exactly, at what point do spec files become so precise that they are just another form of source code?

[–] user43928 14 hours ago

Probably at a point where my worries will be when my team of construction robots finishes my villa, rather than whether companies are prepared to ditch source code.

[–] rubicon33 16 hours ago

Exactly! I wouldn’t be surprised if some day in the distant future, that’s where this all goes. Coding languages may in fact be on their way to being dead. You can imagine a world where, regardless the platform, the lowest level instruction format is output.

[–] kylecazar 15 hours ago

I can see a future generation of models. For now, LLM's train on human artifacts and they handle Python/TS well because of the overwhelming amount of code they've seen, combined with the semantic labels high level languages use that they rely on for reasoning.

Also, consider token efficiency. Imagine how many tokens would be torched just translating an import statement of a popular library to its binary equivalent, let alone a big complex program

[–] stvltvs 15 hours ago

How would we train those LLMs? Feed them binary executables along with their business requirement docs? Is the corpus of training data large enough to be effective? What happens when a new CPU architecture is developed? Where would the training data come from for the new machine code?

[–] CrimsonCape 12 hours ago

Exactly! That's why the "pelican riding a bicycle SVG" test is unfair. It's a token predictor, not an intelligence. What corpus of SVG data has the LLM been trained on? Only in the last 10 years has "drawing using data" been in existence. Midjourney can generate you a pelican riding a bicycle artwork easily. But instead, if the test is intentionally based on a task with no corpus, then you get garbage.

[–] rjknight 16 hours ago

I would not be surprised if we ended up with binaries and Lean proofs.

[–] alienbaby 16 hours ago

isnt that kinda like downloading and using software a team you have no knowledge of has produced and released?

All you can go by is 'does it do what it says it does'

[–] ang_cire 15 hours ago

Yeah, all these devs pretending you can only trust code you understand, then going and pulling in 1000 random npm deps...

[–] batshit_beaver 15 hours ago

When people install npm packages, they typically only install ones the community trusts. These packages have clear APIs, open source, good docs, test coverage, security checks, and, importantly, usually no undocumented side effects.

Engineers introduce these packages into their own projects that have far fewer of the above guarantees than OSS, are built under pressure from deadlines, and have a constantly churning set of maintainers.

I think from the perspective of a business its own code does call for greater scrutiny than its dependencies.

[–] ang_cire 15 hours ago

> When people install npm packages, they typically only install ones the community trusts.

This is wishful thinking. Devs add in dependencies to do mundane stuff all the time, regardless of provenance.

As others pointed out, the default install for many modern web dev frameworks includes hundreds of dependencies.

[–] abirch 16 hours ago

Gary Tan has said markdown is code. Personally I see my job as establishing the architecture, creating tests, lint rules, etc. for my agents. Monitor the code, but if there's an issue create a new agent that monitors performance, etc.

[–] solarpunk_enthu 15 hours ago

I feel like ever since we stopped reading assembly we don't really internalize our codebases.

[–] gedy 15 hours ago

It's just one "Buy Now" button. How hard could that be? /s