Hacker News
points by OGWhales 2 days ago | 0 comments
From my own experience, LLM generated code having more test cases is neither surprising nor an indication of improved quality. I've found LLMs are quite good at making lots of test but the tests are often not that useful. I would argue that total count remains a poor metric for quality.
[–] bluGill 2 days ago
I don't know of a good metric for quality. Total count of tests is the best I've found, but I agree it is bad.
[–] johnnyanmac 2 days ago
Best metric is user satisfaction. Since that is ultimately what software is (or should be) about. Hence the leading header of "software is getting worse for obvious reason": we detached from user satisfaction as the metric to optimize for at least a decade ago.
[–] gf000 2 days ago
Ask for red-green tests. Without that they are often indeed useless.
[–] malfist 2 days ago
I'm curious, mind to tell me more about your process?
[–] gf000 2 days ago
Just simply ask for that, and they usually know what to do.

They should write a test and stash the fix they just wrote. In this state it should fail. Once they re-apply the patch, it should become green and this gave use enough information to deduce that the test does test something.