Hacker News

points by brookst 3 days ago | 0 comments

“Software is degrading for obvious reasons” is just painfully lazy. Where are the metrics of degradation? Where are the claims of specific reasons, and the evidence of causality?

If the complaint is vibe coding makes bad software, I largely believe it, but I absolutely believe that vibe arguing makes tissue paper arguments. “Everyone knows” and “it’s obvious” are flimsy things to rest arguments on.

I’m writing dramatically better software for all the reasons this post says obviously make worse software.

Testing has gotten much easier, and bug fixing has gotten much easier. I looked at a pre-AI project I did a couple of years ago: test to code LOC was 0.15:1. The projects I work on now are around 1.3:1. The old project had no accessibility considered at all, the new ones are nicely done aria and beyond. Old project is stuck on old libraries because upgrading is a major pain. New projects take new libraries (after cooldown) no matter the effort.

So I’m left thinking that perhaps some people, too lazy to use the new powers of new tools, churn out the same or worse quality code/essays.

But I am seeing a lot of very high quality software that simply wouldn’t have been possible before, and I believe the same could be done with essays by leveraging these tools to research and find real data beyond “it’s obvious”. But that does take a lot more work.

[–] renegade-otter 2 days ago

There is plenty of evidence. For example:

https://www.youtube.com/watch?v=MR4i3Ho9zZY&t=37s

The reason is clear - without scarcity of compute or memory, performance is no longer a feature. That works until all the laziness compounds into a crawling bloat.

This is regarding desktop software. With services, microservices is basically a performance anti-pattern that is used without any due diligence in most cases.

[–] snarfy 2 days ago

You forgot cost. Performant software is out there, but you have to pay for it.

With most software being free, cost is optimized for the developers not users and we end up with Electron apps

[–] gf000 2 days ago

"hey Claude, rewrite it in rust"

I'm semi-sarcastic here, but this is actually possible now, while it was an often futile and very expensive roundtrip before AI resulting in two versions of your software that you have to simultaneously maintain.

[–] DaiPlusPlus 2 days ago

> "hey Claude, rewrite it in rust"

Things get lost-in-translation, and Claude doesn't seek your input for decisions that matter.

[–] renegade-otter 2 days ago

I always say - "give me the fastest baseline framework and I will destroy it with one bad SQL query".

[–] malfist 2 days ago

> I looked at a pre-AI project I did a couple of years ago: test to code LOC was 0.15:1. The projects I work on now are around 1.3:1.

Lines of code is a poor metric for measuring, both for application code and testing code

[–] bluGill 2 days ago

While in general that is true, this is one person and so we can assume a lot of the variables that make it a poor metric are constant in this case and so we can draw a few conclusions.

[–] OGWhales 2 days ago

From my own experience, LLM generated code having more test cases is neither surprising nor an indication of improved quality. I've found LLMs are quite good at making lots of test but the tests are often not that useful. I would argue that total count remains a poor metric for quality.

[–] bluGill 2 days ago

I don't know of a good metric for quality. Total count of tests is the best I've found, but I agree it is bad.

[–] johnnyanmac 2 days ago

Best metric is user satisfaction. Since that is ultimately what software is (or should be) about. Hence the leading header of "software is getting worse for obvious reason": we detached from user satisfaction as the metric to optimize for at least a decade ago.

[–] gf000 2 days ago

Ask for red-green tests. Without that they are often indeed useless.

[–] malfist 2 days ago

I'm curious, mind to tell me more about your process?

[–] gf000 2 days ago

Just simply ask for that, and they usually know what to do.

They should write a test and stash the fix they just wrote. In this state it should fail. Once they re-apply the patch, it should become green and this gave use enough information to deduce that the test does test something.

[–] topaz0 2 days ago

We can conclude that there are more tests per code perhaps. We can't conclude they're adding value.

[–] bluGill 2 days ago

my experience is 25% of valuable. Which is a lot better than nothing.

[–] brookst 1 day ago

I think you meant it's a poor metric for judging. It's fine for measuring. Or do you think git should stop showing +/- line counts for PR's.

Somehow LOC morphed from one metric among many to "you should never even think about LOC, 15 lines and 15k lines are exactly the same thing". It's weird.

[–] bemtevi 2 days ago

> But I am seeing a lot of very high quality software that simply wouldn’t have been possible before

Personally, I haven't seen too much of that. What I've seen is software that _looks_ pretty, has a shiny interface, a well-designed animated website made with god knows what, but under the hood it's a bit of an alpha mess....

[–] _bent 3 days ago

> I looked at a pre-AI project I did a couple of years ago: test to code LOC was 0.15:1. The projects I work on now are around 1.3:1.

Goodhart's law etc.

[–] nh23423fefe 2 days ago

lazy low context response to a comment about lazy arguments.

[–] tripleee 2 days ago

Should an AI have expanded it to 3 paragraphs to say the same thing?

[–] nh23423fefe 2 days ago

What point are you trying to make? The comment I responded to added nothing. Your snarky low value response to me is to suggest I believe expanding the nothing by use of AI would be good? Why would you make such a bad inference or worse joke?

[–] AlienRobot 2 days ago

But who is reading those tests? Tests to code LOC was 0.15:1 because humans only tested that 0.15. Now you are assuming that the test the machine wrote actually covers the features that should be tested.

Depending on machine-generated tests that nobody reads isn't better than having poorer coverage. The only thing that is guaranteed is that that mass of code returns "true" with the current implementation and "false" if the implementation changes in some ways. Unless people actually read the LLM-generated code, it's impossible to know if it tests anything that actually matters or just tests the same code path that would never be actually used in production in 5 different ways.

[–] otabdeveloper4 2 days ago

When your software crashes and doesn't do what it's told I will think back on the thousands of bullshit tests Claude wrote and feel less bummed out about it. (Maybe.)

[–] bcvvvy 3 days ago

Test cases doesn't mean shit works. It means test cases pass.

About 50% of the test cases I've ever seen are pointless.

[–] teeray 2 days ago

IME, nobody reads the tests in code review either. They just see "looks like it has tests" *check*. Many times, I've caught LLMs writing completely vacuous tests for configurations of components that were impossible. But coverage number go up, so those tests get written.

[–] galaxyLogic 2 days ago

I've thought about this, tests should be easier to add to your modules. We dont' need a ton of tests, but we need them to be in more places.

Tests fix how the software can be changed and when software-spec evolves lots of tests can become obsolete. Maybe with AI it is easier to change the tests (?) which is good.

But there is another purpose for tests, besides "testing". It is to document your modules in a way that shows you how to use them, and let's you easily verify is this still working, can I trust it.

[–] mysterydip 2 days ago

I was reviewing some coordinate conversion code one time and a developer’s method of creating unit tests was to run their code, generate a result, then make that output the correct result for the test. Except the result was wrong based on other utilities doing the same conversion, and also failed edge cases.

[–] thargor90 2 days ago

For the inexperienced dev audience: generally speaking that is a valid way to generate regression tests and document novel edge case behavior.

In your case of course any existing utilities behavior/specs should be tested first, regression tests just on top, if they are not already covered.

[–] LtWorf 2 days ago

I've seen a coworker ship a test that contained a=1; assert a == 1.

Yes it was vibe coded. The "a" variable was never used, and was tested to be 1 several times.

It was several hundreds lines of course so I didn't bother reviewing the rest.

[–] bluGill 2 days ago

Test cases are an indicator. They say that this situation I thought of works as I think it should. It doesn't cover everything, but it does cover a large number of things.

A test is mostly useful only in the case that by failing it alerts you to a bug.

[–] eviks 2 days ago

They don't say that because you didn't write the tests, AI did, so who knows what behaviors it thought of and wrote tests for. That's the reason these type of metrics are very bad, especially in this context when bad tests are easier to write

[–] rkozik1989 3 days ago

Dawg, you clearly haven't used any consumer-grade software in the past 5 years. Everything across the board is a worse experience and there in some cases are limitations that make no sense. Use any app such as Spotify or Uber Eats, its a worse experience than it was 5 years ago, hands down. Everyone outside of our bubble knows it.

[–] fsflover 3 days ago

Isn't it just the consequence of enshittification? https://pluralistic.net/2024/08/17/hack-the-planet/

It does not affect Debian, Fedora, or Qubes OS.

[–] WarmWash 2 days ago

I have long suspected that one of the driving issues is the continued need for people in the org to keep justifying their existence.

From the consumer POV you can make a great app that does the thing great, and then that's it. Pay once upfront. Just security updates and maybe light compatibility updates after.

Instead we get constant toying and adjusting, and even worse, get pigeon holed into a monthly subscription to bankroll the $200k salaries needed to pay the team that hides the menu button every 6 month to boost screen time.

[–] adjejmxbdjdn 2 days ago

Those aren’t consumer software though.

[–] fsflover 2 days ago

What do you mean? These are desktop operating systems for end users.

[–] pfannkuchen 1 day ago

They must be using “consumer” as “plebeian”.

[–] bigfishrunning 2 days ago

I consume Debian on a regular basis...

[–] zer00eyz 3 days ago

> worse experience

This too is subjective.

No one designs for users any more. It's all gas no breaks on the path to enshitification. A lot of this is the motivation of the people building software.

Raise your hand if you ever saw a PM get promoted for removing a bunch of bad and/or unused features?

[–] eviks 2 days ago

One relatively easy set of metrics - resource consumption driven by migration to resource hungrier stacks.

But in general sure, it's not like we have good databases that count and classify bugs and track their numbers and severity and determine causes so we can compare the dynamic

[–] lowbloodsugar 2 days ago

You are an outlier? I too take pride (and interest and joy) in my software. We are a tiny minority.

[–] lirolero 3 days ago

> But I am seeing a lot of very high quality software that simply wouldn’t have been possible before

cite 1

[–] Citizen_Lame 3 days ago

Probably his own vibe coded stuff.

[–] LtWorf 2 days ago

I imagine a github repository with a README and no source code.

[–] renegade-otter 2 days ago

Zero bugs, though.