This is not just about the pressure put on academics to publish, but it is a whole systemic rot, that is not even remotely living up to the “peer reviewed evidence” myth.
The whole idea of an intermediary authority for scientific publishing is a scam, and it corrupts people who want/need to be in the pyramid. The whole thing is ill-conceived, needs to be abolished, and a new thing should be put in its place. At some point someone said, “I can ditch all this and just publish research on my blog, then people will criticize and build upon that”. No publisher, no paywall, no problem. If we follow this example, all of these issues can disappear overnight. But the vast majority of professionals value their career more than anything else, including our tantamount tenets of what science communication should look like.
You might object that “intermediary authorities” and “peer review” are essential to prevent disinformation and conspiracy theories. Well, we are past this point aren’t we? Did this system prevent conspiracy theories and disinformation, hoaxes, and fraudsters this far? No, so how exactly will it prevent all of these terrible things in the future? If anything, building arguments in the open without paywalls might deter at least some of the conspiracy theorists that brandish paywalls as further evidence of cover-ups and secrecy, and ditching the horrible jargon and high-brow style might actually help the common sense of scientific arguments just shine, and combat the rising anti-intellectualism of right-wing conspiracy theorists.
Like, if you explain Elsevier’s etc business model to any lay person (Pay me money so that I let you publish to my super-selective journal and feed your vanity) they have the most funny reactions, because to anyone who is not conditioned to this absurdity, it just sounds like a pyramid scheme.
Only thing I’d mention slightly counter to that is peer review - as a process - is still something I believe is useful.
That is, the process of people with relevant domain expertise critiquing methodology, findings etc. When its done right, it absolutely produces better results which everyone benefits from.
Where it fails is when cliques and ingroups are resistant to change on principle, which is ofc actually an anti-scientific stance. To put it another way, the best scientist wants to be proven wrong (or less correct) if that is indeed the truth.
It also fails, as you identify, when the corrupt rot of powerful publishers (who are merely leeches) gate-keep the potential for communicating alternate models.
It also fails where laypeople parrot popsci talking points without understanding that peer review is far from infallible. Even the best of the best journals still contain errors - any genuine scientist is the first to admit this. Meanwhile popsci enthusiast laypeople think that just because something was printed in any journal, that it must be unequivocally 100.000% truth, and are salivating at the opportunity to label any healthy dose of skepticism as “antiscience” or “conspiracy theorist” etc.
It also seems to fail when popsci headlines invariably don’t include the caveats all good scientists include with their findings etc.
Final point which I think would help enormously is its very very difficult to get funding or high worth publications in reproduction. The obsession with novelty is not only unhealthy, it’s unproductive.
Reproduction is vastly undervalued. Sadly its not easy to get funding or support for ‘merely’ reproducing recent results. There’s two reasons why this should change, firstly it will ofc help with the reproducibility crisis, and it will also afford upcomers excellent opportunities to sharpen their skills, and properly prepare for future ground-breaking work. To put another way, when reading a novel paper you think you understand it. Only when you take it to the lab do you truly understand.
As far as I know the peer reviewers are in most cases now selected by the editor, they self-select to respond, are not paid for their work, and the process for alarmingly many journals is not even blind. I always thought that this makes the process vulnerable to network effects in the field, since people are obliged to a certain etiquette when commenting on established figures in their own field. So yes, I get where you are coming from, but similar to the scientific method, peer review is also great to describe in theory, in practice it would require much more precise protocols, like Web protocols I might say. I really don’t want to be a pessimist about science in the current political climate, but if we want these great ideals (Scientific method, Peer Reviewed evidence) we will have to abandon the existing situation as soon as possible.
agreed the existing system is deeply flawed and currently on a trajectory to critical failure.
regarding peer review itself, this is another point. people regard peer review as this binary thing which takes place prior to publication and is like a box which is ticked after publication.
which is ofc ridiculous, peer review is an ongoing process, meaning many of the important parts take place after publication. fortunately this does happen in a variety of fields and situations, however not being the norm leads to a number of the issues in discussion. further it creates an erroneous mindset that simply because something has been published that its now fully vetted, which is ofc absurd.
also agreed, the process should be blind. i believe it often already means the reviewer’s identities are hidden, but i also agree the authours should be hidden during the process too.
don’t see the role as unpaid being a problem though, introducing money would complicate things alot and create even more conflicts of interest and undermine what little integrity the process still has.
i really love your idea of standardising the process in a network-like protocol. this would actually make an excellent RFC and i’d totally support that.
on a similar vein, this is why i’ve been advocating for a complete restructuring of support given to reproduction. as you mentioned, the current process is vulnerable to a variety of human network effects. and among other issues with that problem, i also see the broken reproduction system playing a role here.
as it currently stands, reviewers can request more explanation or data, introduction of changes/additional caveats etc or reject the paper entirely. what this means is a reviewer can only really gauge whether something sounds right, or plausible. and as you correctly identify, certain personalities or flavours of prevailing culture will play a role in the reviewer’s assessment of what merely seems like it’s plausible or correct etc. this has shown to make major breakthroughs more difficult to communicate and face unfair resistance, which has frankly held back society at large.
whereas if there was an organised system of reproduction it’s no longer left to just a matter of opinion in how something sounds. this is ofc how its supposed to work already, and sometimes does, but all too often does not. imo it would be a great detail to include in your idea for a protocol-based review process.
i don’t envision this as always being something which must take place prior to publication, it can and should be an ongoing process. where papers could have their classification formerly upgraded over time. currently the only ‘upgrade’ a paper really receives is publicity or number of citations. the flaws of which are yet another discussion again.
Yes, that would be great. People put so much stock in peer review because there is the myth that every statement undergoes under a rigorous process of verification in multiple laboratories. The reality is, as you said, there is a culture of active discouragement of reproduction and the pushing of novel results.
Not to mention that to foster reproductions, researchers should be trained into a culture of replication and collective metanalyses. As it is now, reproductions are less than an afterthought for the vast majority of researchers, and virtually none knows how to handle multiple replicatory studies instead of p-hacking.
I don’t think that “intermediary authorities” and “peer review” are the problem here nor will they completely eliminate disinformation and conspiracy theories on the internet. Getting rid of them does not help at all with those goals though. The big problem with publishers ATM is the closed access and processes that go on.
IMO places like Open Science Journal and PLOS are vastly better and attempting to solve the issues with the current closed and restrictive publishing models.
This is not just about the pressure put on academics to publish, but it is a whole systemic rot, that is not even remotely living up to the “peer reviewed evidence” myth.
The whole idea of an intermediary authority for scientific publishing is a scam, and it corrupts people who want/need to be in the pyramid. The whole thing is ill-conceived, needs to be abolished, and a new thing should be put in its place. At some point someone said, “I can ditch all this and just publish research on my blog, then people will criticize and build upon that”. No publisher, no paywall, no problem. If we follow this example, all of these issues can disappear overnight. But the vast majority of professionals value their career more than anything else, including our tantamount tenets of what science communication should look like.
You might object that “intermediary authorities” and “peer review” are essential to prevent disinformation and conspiracy theories. Well, we are past this point aren’t we? Did this system prevent conspiracy theories and disinformation, hoaxes, and fraudsters this far? No, so how exactly will it prevent all of these terrible things in the future? If anything, building arguments in the open without paywalls might deter at least some of the conspiracy theorists that brandish paywalls as further evidence of cover-ups and secrecy, and ditching the horrible jargon and high-brow style might actually help the common sense of scientific arguments just shine, and combat the rising anti-intellectualism of right-wing conspiracy theorists.
Like, if you explain Elsevier’s etc business model to any lay person (Pay me money so that I let you publish to my super-selective journal and feed your vanity) they have the most funny reactions, because to anyone who is not conditioned to this absurdity, it just sounds like a pyramid scheme.
@whydudothatdrcrane Great analysis!
I generally agree. The system is utterly rotten.
Only thing I’d mention slightly counter to that is peer review - as a process - is still something I believe is useful.
That is, the process of people with relevant domain expertise critiquing methodology, findings etc. When its done right, it absolutely produces better results which everyone benefits from.
Where it fails is when cliques and ingroups are resistant to change on principle, which is ofc actually an anti-scientific stance. To put it another way, the best scientist wants to be proven wrong (or less correct) if that is indeed the truth.
It also fails, as you identify, when the corrupt rot of powerful publishers (who are merely leeches) gate-keep the potential for communicating alternate models.
It also fails where laypeople parrot popsci talking points without understanding that peer review is far from infallible. Even the best of the best journals still contain errors - any genuine scientist is the first to admit this. Meanwhile popsci enthusiast laypeople think that just because something was printed in any journal, that it must be unequivocally 100.000% truth, and are salivating at the opportunity to label any healthy dose of skepticism as “antiscience” or “conspiracy theorist” etc.
It also seems to fail when popsci headlines invariably don’t include the caveats all good scientists include with their findings etc.
Final point which I think would help enormously is its very very difficult to get funding or high worth publications in reproduction. The obsession with novelty is not only unhealthy, it’s unproductive.
Reproduction is vastly undervalued. Sadly its not easy to get funding or support for ‘merely’ reproducing recent results. There’s two reasons why this should change, firstly it will ofc help with the reproducibility crisis, and it will also afford upcomers excellent opportunities to sharpen their skills, and properly prepare for future ground-breaking work. To put another way, when reading a novel paper you think you understand it. Only when you take it to the lab do you truly understand.
As far as I know the peer reviewers are in most cases now selected by the editor, they self-select to respond, are not paid for their work, and the process for alarmingly many journals is not even blind. I always thought that this makes the process vulnerable to network effects in the field, since people are obliged to a certain etiquette when commenting on established figures in their own field. So yes, I get where you are coming from, but similar to the scientific method, peer review is also great to describe in theory, in practice it would require much more precise protocols, like Web protocols I might say. I really don’t want to be a pessimist about science in the current political climate, but if we want these great ideals (Scientific method, Peer Reviewed evidence) we will have to abandon the existing situation as soon as possible.
agreed the existing system is deeply flawed and currently on a trajectory to critical failure.
regarding peer review itself, this is another point. people regard peer review as this binary thing which takes place prior to publication and is like a box which is ticked after publication.
which is ofc ridiculous, peer review is an ongoing process, meaning many of the important parts take place after publication. fortunately this does happen in a variety of fields and situations, however not being the norm leads to a number of the issues in discussion. further it creates an erroneous mindset that simply because something has been published that its now fully vetted, which is ofc absurd.
also agreed, the process should be blind. i believe it often already means the reviewer’s identities are hidden, but i also agree the authours should be hidden during the process too.
don’t see the role as unpaid being a problem though, introducing money would complicate things alot and create even more conflicts of interest and undermine what little integrity the process still has.
i really love your idea of standardising the process in a network-like protocol. this would actually make an excellent RFC and i’d totally support that.
on a similar vein, this is why i’ve been advocating for a complete restructuring of support given to reproduction. as you mentioned, the current process is vulnerable to a variety of human network effects. and among other issues with that problem, i also see the broken reproduction system playing a role here.
as it currently stands, reviewers can request more explanation or data, introduction of changes/additional caveats etc or reject the paper entirely. what this means is a reviewer can only really gauge whether something sounds right, or plausible. and as you correctly identify, certain personalities or flavours of prevailing culture will play a role in the reviewer’s assessment of what merely seems like it’s plausible or correct etc. this has shown to make major breakthroughs more difficult to communicate and face unfair resistance, which has frankly held back society at large.
whereas if there was an organised system of reproduction it’s no longer left to just a matter of opinion in how something sounds. this is ofc how its supposed to work already, and sometimes does, but all too often does not. imo it would be a great detail to include in your idea for a protocol-based review process.
i don’t envision this as always being something which must take place prior to publication, it can and should be an ongoing process. where papers could have their classification formerly upgraded over time. currently the only ‘upgrade’ a paper really receives is publicity or number of citations. the flaws of which are yet another discussion again.
Yes, that would be great. People put so much stock in peer review because there is the myth that every statement undergoes under a rigorous process of verification in multiple laboratories. The reality is, as you said, there is a culture of active discouragement of reproduction and the pushing of novel results.
Not to mention that to foster reproductions, researchers should be trained into a culture of replication and collective metanalyses. As it is now, reproductions are less than an afterthought for the vast majority of researchers, and virtually none knows how to handle multiple replicatory studies instead of p-hacking.
I don’t think that “intermediary authorities” and “peer review” are the problem here nor will they completely eliminate disinformation and conspiracy theories on the internet. Getting rid of them does not help at all with those goals though. The big problem with publishers ATM is the closed access and processes that go on.
IMO places like Open Science Journal and PLOS are vastly better and attempting to solve the issues with the current closed and restrictive publishing models.
Perhaps we could benefit from sth like MetaCritic for science.