It just needs ca 5 seconds to realize "oh damn, I would not want to be at the receiving end of this" and conclude that it's probably a bad idea.
Why does this keep happening?
Especially since the idea of having such automated reviews probably isn't bad. But why mess up the execution like this basically every single time?
[–] throwawayffffas 3 days ago
[–] hypfer 3 days ago
It's just such an obviously terrible execution. Why would you throw it into production when doing literally anything else - and that includes just setting the whole server on fire - is less damaging.
(Not hyperbole. Loss of trust and loss of trust in ones own judgement is the highest and harshest price in this industry)
[–] InsidiousWombat 3 days ago
[–] Roark66 2 days ago
So we setup llm based reviews that catch 99% of simple problems saving us 7 out of 8 rounds of PR reviews.
Seriously. Even as simple rules as: - if you submit a PR with new X, put a link to a successful run of said X to proove it's working. - if you choose not to reuse Y, but create a new one explain why.
Those are 100% ignored by people (or their crappy LLMs more likely - so I have to teach them how to prompt, or tell them which models to use for what).
Instead of loosing sanity responding for the 10th time to someone in time zone 12h away that he tries to merge reimplementation of what we already have with no description and no test run I have an automatic llm review that tells them that exact thing 2min after they create the PR.
[–] MisterMunchkin 3 days ago
[–] DaSHacka 2 days ago
My most recent experience was calling AT&T because the serial for my modem was whitelisted on an incorrect PON port (AKA the issue was on their end), yet if you correctly answer all the automated bot questions when calling their support line indicating you have an issue with WAN connectivity, it guides you through the process of unplugging your combo unit, waiting 10 seconds, and plugging it back in, then it automatically hangs up on you. Like thanks, that may help Sheryl who thinks the internet is down because her computer's DNS caches an expired facebook IP, but I literally cannot solve this issue myself without help from an AT&T tech lest I violate federal law by breaking into their network closet and do it myself.
I finally got through to a person by incorrectly claiming there was a billing issue and asking that rep to transfer me to the right department, skirting the bot entirely.
[–] jrmg 3 days ago
But with the very latest models (like, Opus 5.5 class and up), I am now of the opinion that the output, in non-creative situations like code review and most software development, is fine. Great, even. Better than the reviews a lot of humans generate.
Honestly, if you’re at all open minded about this, get a $20 Claude Pro subscription for a month (they are easy to cancel) and ask Claude Code with Opus 5.5 to review your codebase or a PR or two. I was very (pleasantly? unpleasantly?) surprised.
I have complex emotions about this (‘sad’ on balance I think - although it’s definitely not all simply in that direction), but I can’t deny it to myself.
I still bristle at reading articles and even documentation that’s not just ‘reference’ documentation written by it - and even that can get a bit weird and circuitous. But for review and coding? It’s great.
[–] hypfer 2 days ago
[–] jrmg 2 days ago
I thought this was the crux of your argument. What did I miss?
The way I see it the devs that put that in place are saying, listen before even reading your contribution I am going to feed it to a LLM with these instructions and see what it spits out. Now I will extricate my self from being the LLM proxy so you deal with this output and when it's okay I will look at it.
Same as "listen bud until it passes linting I won't even look at it." Just a much more flexible and fuzzy linter.