Bogus science paper reveals peer review's flaws
John Bohannon found that many journals are not living up to their peer review promises
A bogus scientific paper about a new anti-cancer compound was accepted by more than half of 300 scientific journals it was submitted to in the past year, despite having obvious and serious scientific flaws.
CBC Radio’s The Current explores what that means about the peer review and science publishing system, and what the implications are for public confidence in science.
Science journalist John Bohannon reported the results of his experiment in the journal Science.
In an interview with The Current, Bohannon, who holds a PhD in molecular biology and is a visiting scholar at Harvard University’s program in ethics and health, said the experiment was inspired by the experience of a colleague in Nigeria, who felt deceived by a certain journal — one with a business model that involves charging fees to the scientific authors ranging from $50 to more than $3,000.
Traditional scientific journals charge only small fees to scientific authors, but high fees to academic libraries and anyone else wanting to read the papers after publication.
Fee-charging open-access journals charge much higher fees to authors, but are free to read after publication. That means they are available to the people, including the general public, who normally cannot access the scientific literature, even to read the results of research funded with their tax dollars.
Following his colleague’s experience, Bohannon wanted to find out whether fee-charging open access journals were actually keeping their promise to do peer review — a process in which scientists with some knowledge of a paper’s topic volunteer to check it for scientific flaws.
Bohannon described the details of his experiment to The Current host Anna Maria Tremonti.
In the end, what he concluded was that “a huge proportion” of the journals were not ensuring their papers were peer reviewed.
Even in cases where peer review happened, it didn’t always function correctly. For example, the Ottawa-based InternationalJournal of Herbs and Medicinal Plants clearly sent the paper out to be reviewed by real scientists, who pointed out some flaws, Bohannon recalled. Even so, when Bohannon submitted a revised version of the paper without correcting any of the flaws, it was accepted.
As of Friday, only one out of four Canadian publications that accepted the fake paper had responded to The Current’s request for a comment.
That publication, the Journal of Nephrology and Renal Transplantation, did not publish the paper in the end because Bohannon withdrew it. The journal told The Current that it relies on feedback from peer reviewers to accept or reject a paper.
“Unfortunately, peer review has its flaws and such incidents are not avoidable if not identified by the reviewers,” it said.
Peer review 'crucial'
Bohannon said peer review is “crucial” so that readers of a scientific paper know it has “at least passed muster with a couple of experts who are in a position, hopefully, to judge.”
He added that his experiment could be the tip of the iceberg, and that peer review at traditional journals – not just fee-based open access journals — could be just as bad.
“It could be the whole peer review system is just failing under the strain of the tens of thousands of journals that now exist.”
He added that if peer review isn’t working, then people with what amounts to fraudulent scientific credentials and publication records “are slowly filling university departments and government offices, making important science-based policy decisions.” In addition, “terrible science” is polluting the global pool of knowledge.
Among the journals to which Bohannon submitted the fake paper, only one flagged its ethical and research flaws.
Michael Eisen, a professor of molecular and cell biology at the University of California, Berkeley and co-founder of the Public Library of Science, one of the largest publishers of open access journals, told The Current he thinks Bohannon’s experiment “revealed something that people need to know — the process of peer review is often not what it appears to be.”
He said while the public has the idea that the peer review process is “infallible,” practising scientists would disagree.
“At best, even when people are carrying out peer review in an honest and sincere way, it represents the view of a couple of scientists at a single fixed point in time. And very, very often, those scientists miss clear flaws in the work, they miss poor design in the experiment.”
Eisen said he thinks the two major problems with peer review are that it is done in secret and only done at the time a paper is submitted to a journal. He thinks peer reviews should be public and should continue to be published alongside the paper throughout its useful lifetime.
“Right now,” he said, “the scientific community is polluting the public discourse and the public use of science by not doing a good job.”