- Slow PR review
- Nitpicks on style, not substance
- Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
- A lot of a ceremony around straightforward changes
We use LLM reviews at my company and it's caught things human reviews have not (as in in the early days we human-reviewed code and then did tests running the LLM afterwards and it caught things we didn't). It can review code at any time of the day and re-review it just as quickly as you can address the feedback.
I predict we (my company) are less than a year out from having a fully automated pipeline for reviews with only humans getting called in when it touches certain parts of the code. We already have a crude system for detecting certain file paths that we want a human to review if touched and we tweak that list over time as needed.
[–] ImPostingOnHN 3 days ago
[–] RHSeeger 2 days ago
And the person that wrote the code is expected to be the review.
Unless it's an automated pipeline that fixes the things found - I'm not a fan of that unless it's made VERY clear the LLM needs to be VERY sure it's right. And even then, I'd prefer it go through a human to decide which things are valid, then send them off to be fixed.
[–] Roark66 2 days ago
Github copilot review is s great example of the worst possible way this can be misused. It is the worst waste of tokens there is.
Instead we use a process we created ourselves that runs opus or sonet (depending on complexity) to do a review (taking into account pr description and comments - copilot ignored those) and then to re-review the issues found to cut out BS.
This has been working very well mainly thanks to the prompts and docs we made it read, as well as us having lots of example code that shows how things should and should not be done.
One crappy thing Github does is that it doesn't allow our workflow based review's tokens to be accounted to individual PR authors like "copilot review" does, but accounts tokens to the organisation. It is obvious they are trying to herd people to using gh copilot review instead. But the quality difference is like night and day. So we are staying.
[–] joshstrange 2 days ago
We also settled on a custom solution for reviews. We started with reviewers that claude/codex on GH actions and those were "ok" but kind of dumb and hard to tweak (GH Actions is just miserable to debug/iterate on). We still use them occasionally (@tag them to kick off a review) but we've shifted to a piece of software that we wrote that watches GH PRs and once certain labels have been added and the ticket in our ticket tracker (not GH issues) is marked as "Ready for Review" it kicks off a review and it reviews _all_ the PRs connected to the ticket. We don't have a monorepo, yet, so a new feature might span 3-4 repos (server, client1, client2, ...) and we want that work reviewed together.
We are close to having our review code spin up a fully isolated devstack running all our, many, services so it can poke around in the UI and write custom playwright scripts to test the new functionality. We have the devstack VMs working, just gluing the last of it together, it's pretty cool to see it all come together.
[–] othmanosx 2 days ago
[–] joshstrange 2 days ago
Personally I have an agent review the feedback and break down each item so it’s easier to review (helps remove some of the jargon/short-hand that agents use sometimes). Then I decide what to have an agent address and what to push back on. Like I said, I’m working on finding a way to automate part of that but haven’t settled on one I like yet.
> Nitpicks on style, not substance
> Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
> A lot of a ceremony around straightforward changes
Those aren't great, but you what I like even less?
- Wall of nondeterministic bot text spam that may or may not be correct.
- Bot output has not been verified for correctness by the person who set up this bot output pipeline.
- Bot output has not even necessarily been read by the person who set it up.
- You have 24 hours to respond to this 5 page wall of text which may or may not be correct, or even make sense, or you will be penalized.
An appropriate response to this might be a reply flatly claiming that, actually, there aren't any problems after all, but that nobody has checked that that is true, and it is incumbent upon a human reviewer to verify your claims, like it was incumbent upon you to verify their bot's.
Can an LLM catch some errors? Absolutely. Can it be trusted to be correct absent review of the output? Absolutely not. So the simple solution seems to be, a human reviewer is free to run an LLM against code to review, on their own computer, and then they review the output, and they post the output only after verifying its correctness, taking personal responsibility for any issues with their posted output.