Hacker News

points by bluGill 2 days ago | 0 comments

While in general that is true, this is one person and so we can assume a lot of the variables that make it a poor metric are constant in this case and so we can draw a few conclusions.

[–] OGWhales 2 days ago

From my own experience, LLM generated code having more test cases is neither surprising nor an indication of improved quality. I've found LLMs are quite good at making lots of test but the tests are often not that useful. I would argue that total count remains a poor metric for quality.

[–] bluGill 2 days ago

I don't know of a good metric for quality. Total count of tests is the best I've found, but I agree it is bad.

[–] johnnyanmac 2 days ago

Best metric is user satisfaction. Since that is ultimately what software is (or should be) about. Hence the leading header of "software is getting worse for obvious reason": we detached from user satisfaction as the metric to optimize for at least a decade ago.

[–] gf000 2 days ago

Ask for red-green tests. Without that they are often indeed useless.

[–] malfist 2 days ago

I'm curious, mind to tell me more about your process?

[–] gf000 2 days ago

Just simply ask for that, and they usually know what to do.

They should write a test and stash the fix they just wrote. In this state it should fail. Once they re-apply the patch, it should become green and this gave use enough information to deduce that the test does test something.

[–] topaz0 2 days ago

We can conclude that there are more tests per code perhaps. We can't conclude they're adding value.

[–] bluGill 2 days ago

my experience is 25% of valuable. Which is a lot better than nothing.