points by hasley 4 days ago | 0 comments
I am not against some kind of "public review" and I really like arxiv.org. But I do not see it as a replacement for a peer review. It may start after a peer review when the paper is published, which IMHO should almost always be in an Open Access way.
For AI, the borders seem to have blurred a bit between academic scientists (e.g. at universities) and external scientists at companies or even private people). The problem with the latter is that it is much harder to enforce good scientific practice. Think only about the time it takes to write the state of the art of an ordinary conference paper.
I am also a bit drawn between giving everyone the chance to contribute to research and ensuring good scientific practice. Just writing down your latest ideas and successes is not the same as writing a scientific paper that shows the research in the context of past research and needs, for instance, to be much more mathematically rigorous.
This is nothing like academic journal peer review, where you can't see the failed submissions for the paper and why. You have no clue if the reviewers were interested in the topic, diligent, drunk or playing politics "support papers from these institutions because they're the good guys, undermine those because they are or aren't from Europe." You have no idea if the paper has since failed to replicate on multiple occasions. You have no idea if you are the first person to actually read the paper and when you have you thoughts, no matter how insightful, well articulated and supported by evidence, those thoughts do not matter to the publication process of that paper, which has been peer reviewed.
Public review by the knowledgeable who care is a very different thing to so called "peer review."