[–] harry8 4 days ago
[–] hasley 4 days ago
For AI, the borders seem to have blurred a bit between academic scientists (e.g. at universities) and external scientists at companies or even private people). The problem with the latter is that it is much harder to enforce good scientific practice. Think only about the time it takes to write the state of the art of an ordinary conference paper.
I am also a bit drawn between giving everyone the chance to contribute to research and ensuring good scientific practice. Just writing down your latest ideas and successes is not the same as writing a scientific paper that shows the research in the context of past research and needs, for instance, to be much more mathematically rigorous.
[–] harry8 3 days ago
This is nothing like academic journal peer review, where you can't see the failed submissions for the paper and why. You have no clue if the reviewers were interested in the topic, diligent, drunk or playing politics "support papers from these institutions because they're the good guys, undermine those because they are or aren't from Europe." You have no idea if the paper has since failed to replicate on multiple occasions. You have no idea if you are the first person to actually read the paper and when you have you thoughts, no matter how insightful, well articulated and supported by evidence, those thoughts do not matter to the publication process of that paper, which has been peer reviewed.
Public review by the knowledgeable who care is a very different thing to so called "peer review."
[–] elendilm 4 days ago
It's fairly basic. For a work published on a public platform, the same guys paid by the journal can also review it if they so choose to in addition to those people in the public who were not paid by anyone.
[–] mkremins 3 days ago
[–] elendilm 3 days ago
The logic stands. A reviewer, even those who get zero compensation, can always review publicly published papers just like others.
[–] blagie 4 days ago
In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.
Retractions are non-existent, even with clearly proven academic malpractice.
[–] hasley 4 days ago
Usually I got quite critical comments (positive and/or negative). Yes, sometimes a reviewer does not write even a single sentence before giving an average score to complete the review process. But this has not been the regular case.
And as a reviewer for a conference paper of 4 to 6 pages, I usually assigned 1 to 3 hours to read it (journal papers: 1 to 2 days) and to give comments.
[–] blagie 3 days ago
Most of the rest of my papers were in completely different disciplines.
It was an awesome paper -- in my mind, groundbreaking -- but I'm not convinced anyone beyond the reviewers ever read it. I see about a dozen citations which popped up years later, clicked through a few, and they were inane.
Problem is it was work in a very, very classical field, alongside Butterworth, Chebyshev, and similar, and while quite fundamental, and with many applications, there is no contemporary research there.
My most popular papers were mostly rather.... random.
[–] harry8 3 days ago
[–] Grimblewald 3 days ago
Retractions do happen as well, especially if it's clearly proven malpractice. I've been behind 2 retractions so far, but it has never (in my experience) happened at the "contact corresponding author and seek clarification" stage of escalation. Typically it requires escalation to the journal itself.
It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).
Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.
Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.
We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.