I think you’re placing a lot more weight on the authority of a single scientific paper than any actual scientist ever would. If you have one paper, you have one paper. If you have a series of papers all put out by the same lab… maybe there’s something there, maybe not. If other groups start publishing similar papers, okay this is sometime serious.
In some of the messier sciences, like medicine, people will publish meta-studies, where they combine results from similar, but independently published, papers and see what they can come up with using the combined data. People will also publish literature reviews, where they essentially try to summarize the state of the science in their particular little niche. To trust a single study in medicine is to hitch your horse to a wicket.
The peer review process doesn’t stop wrong papers from getting published, just obviously wrong or bad ones. I’m not entirely sure what you could even do to stop wrong papers from making into journals, since often times the problem isn’t in the published experimental design or analysis. Plus, there’s some papers that used to be right, but have become wrong as things change.
they’re apparently lower effort than a good reddit comment
They’re not, people are being flippant. People frequently complain about having to do peer reviews specifically because it’s unpaid labor. Regardless, if the paper is so wrong it would warrant a community note on Twitter, the paper would be strongly rejected. The standard for acceptance is way higher than that. Remember that it gets reviewed by fellow experts in the field. They will easily spot small errors.
Is it the best possible system? Heck if I know. It works. Moving to a different system would require everyone to recalibrate their understanding of what good science looks like. We know how to identify it under the current publication model, it would take a fair bit of time to adjust to the new one.
Edit: Oh yes, re: letters. It can take a year or more between publications. Letters might be slow, but it’s not terribly important. It takes time to do science, you don’t need to clap back in an instant.
I think you’re placing a lot more weight on the authority of a single scientific paper than any actual scientist ever would. If you have one paper, you have one paper. If you have a series of papers all put out by the same lab… maybe there’s something there, maybe not. If other groups start publishing similar papers, okay this is sometime serious.
In some of the messier sciences, like medicine, people will publish meta-studies, where they combine results from similar, but independently published, papers and see what they can come up with using the combined data. People will also publish literature reviews, where they essentially try to summarize the state of the science in their particular little niche. To trust a single study in medicine is to hitch your horse to a wicket.
The peer review process doesn’t stop wrong papers from getting published, just obviously wrong or bad ones. I’m not entirely sure what you could even do to stop wrong papers from making into journals, since often times the problem isn’t in the published experimental design or analysis. Plus, there’s some papers that used to be right, but have become wrong as things change.
They’re not, people are being flippant. People frequently complain about having to do peer reviews specifically because it’s unpaid labor. Regardless, if the paper is so wrong it would warrant a community note on Twitter, the paper would be strongly rejected. The standard for acceptance is way higher than that. Remember that it gets reviewed by fellow experts in the field. They will easily spot small errors.
Is it the best possible system? Heck if I know. It works. Moving to a different system would require everyone to recalibrate their understanding of what good science looks like. We know how to identify it under the current publication model, it would take a fair bit of time to adjust to the new one.
Edit: Oh yes, re: letters. It can take a year or more between publications. Letters might be slow, but it’s not terribly important. It takes time to do science, you don’t need to clap back in an instant.