Jelte Wicherts, research methodology and peer review specialist at the University of Tilburg
“This case certainly gives insights into how things go in practice. The peer review process was a bit messy here, but certainly not atypical. What stands out in particular is that the authors here were not experts, and that the reviewers had played a very active role in improving the paper. In the end, one of them was actually giving his own data a plug. That may be defensible in substantive terms, but it shows that entirely independent assessments are difficult to achieve.
“The reviewers were not trying to get away with half measures. However, they clearly did not look at the statistics closely, and that happens a lot. Reviewers very often do not have expertise in all aspects of a study. In some journals, reviewers are therefore asked explicitly if they have understood enough of the analysis – that’s something they won’t admit quickly otherwise. I think it’s fairly peculiar in this case that a statistician did not look at the manuscript. That’s something that the editor should arrange.
“But what I find particularly striking is the turnaround that the editors at Nature made [after the manuscript was first rejected]. You don’t normally get a second chance with them. It seems as if something made them realise that this paper could be newsworthy and influential. And that is in fact their business model.
“In 2013, I visited a congress about peer reviewing at which Philip Campbell, the editor-in-chief of Nature, spoke. He said that Nature sometimes lowers the standards for scientific rigour a bit and publishes articles nevertheless if the results are exciting. In my opinion, Nature and Science are taking conscious risks, and I suspect that that is the core of the problem. This case is justifiably putting question marks by the use of the term ‘leading journal’ for Nature.”
Richard Smith, former editor-in-chief of the British Medical Journal and now a well-known critic of the classical peer review system
“I read with interest that Nature’s primary questions to be answered in an ‘ideal review’ are concerned with novelty value and not with methods. But I’m not surprised. We know that the scientific papers published in ‘top journals’ are the most likely of all to be false, because they are interested in what’s new and sexy.
“We know little about the editorial process except that the editors reversed the decision to reject the paper. That strikes me as unusual and suggests that the editors were attracted by the ‘sexiness’ of the paper.
“I found it disturbing that Jay Olshansky [a reviewer] says that there should be less focus on the statistics. I strongly disagree. Nor do I agree with the statement by Jean-Marie Robine [another reviewer] that ‘you cannot recommend rejecting a manuscript purely because you disagree with the interpretation of the results’. For me, the primary job of a reviewer is to judge whether the conclusions are supported by the data and to make clear why they disagree with the interpretation of the results.
“That inevitably means a close examination of the data and statistics. I find it disturbing that the reviewers did not scrutinise all the data, but I do understand why: peer reviewing is a tough, time-consuming job for which there is no reward.
“So yes, I see all of these as failures in the process, but I don’t think that they are unusual. Peer reviewing is a highly imperfect process, and we have little evidence of its benefits. We need more studies, and it might be that detailed examination of individual cases such as this one could complement other evidence.”
A Nature spokesperson
“We are concerned by and would not agree with your characterisation of the peer-review process for the paper and regret that, due to confidentiality constraints, we are unable to provide further clarification of the specifics of the case. As we have noted before, for confidentiality reasons, we are unable to discuss the specifics of the editorial history or review process of any published Nature paper with anyone other than the authors. We can, however, provide further clarification on the following points.
“We cannot identify individual editors for published Nature papers as several editors, including but not limited to a Senior Team Leader and the Chief Biological Sciences Editor, will be involved in any decision despite only one person communicating the decision to the authors and/or referees.
“The Nature journals do employ statisticians as consultants on certain papers, at the editors’ discretion and as suggested by referees.
“Papers submitted to Nature are considered on the basis of how well they meet our criteria for publication: original scientific research, of outstanding scientific importance, which reaches a conclusion of interest to a multidisciplinary readership.
“Every published paper undergoes rigorous peer review, typically by two or three reviewers (but sometimes more if special advice is needed). Reviewer expertise is, of course, subjective, but our professional editors select a spectrum of reviewers who can assess different aspects of the paper, including the techniques used and the scientific findings.”