As an editor of a scientific journal, one of my key duties is to organise the peer review of submitted scientific papers. There, I ask other experts to take a look at a paper and let me know their opinion on technical correctness of their findings, and perhaps also what the importance and impact of a paper could be. The reviewers are aware of the identity of the authors, whereas the identity of the reviewers is not revealed to the authors.
The requirement to use peer review is not set in stone, but it has proven a very useful tool to assess a scientific paper. However, given the huge amount of work involved where scientists review each other’s work. Indeed, a lot has been said about the peer review process, whether it should be opened, or completely abolished etc. Here I just like to focus on the issue of transparency, which has been subject of a commentary by Bernd Pulverer from the European Molecular Biology Organization in last week’s issue of Nature. Access is free.
Historically, peer review as such is known for a long time, but is only systematically been used since around the mid-20th century. Certainly the very idea of peer review has been a new concept to Albert Einstein, when following peer review his paper on gravitational waves was rejected by Physical Review in 1936:
Dear Sir,
We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the — in any case erroneous — comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.
Respectfully,
Albert Einstein
And even later on, peer review has not necessarily always been done. The famous 1953 Nature paper by Watson and Crick on the structure of DNA has not been peer reviewed. Competition was very tough in this case, and as John Maddox, former editor of Nature, allegedly said:
The Watson and Crick paper was not peer-reviewed by Nature… the paper could not have been refereed: its correctness is self-evident. No referee working in the field … could have kept his mouth shut once he saw the structure
Clearly, peer review was not always considered necessary by scientists as well as publishers. To me, it remains essential. But I think the system could improve, and one area where this could be done with ease is its transparency. At the moment, the process is not fully transparent, neither to authors nor to scientists other than the reviewers. As it stands, there is a lot of implicit trust in the work of journal editors…
More transparency!
Bernd Pulverer’s commentary describes an effort to increase transparency at The EMBO journal. Since 2009, the journal has been running a trial where in case of publication the anonymized referee reports sent to authors, the editorial decision letters as well as the author’s rebuttals to reviewers are published as a supplementary file along with the paper. (disclaimer: Nature Publishing Group publishes this journal on behalf of EMBO).
Overall, it seems this system is a success, at least it is not a failure. Most authors and referees accept taking part in this trial. Only 5.3% of authors declined this. Interestingly, the most popular supplementary files accessed on the web site are those of what are judged to be the most exciting papers, rather than controversial ones.
The supplementary files highlight the many changes to a paper, which creates a continuous platform for debate. This debate now begins with the point of submission and at many journals such as Nature or PLOS ONE also continues after publication in the form of comments on published papers that everyone can leave on the paper’s web page. This all the more relevant if one considers activities such as Open Notebook Science, where the entire process from conceiving scientific experiments to writing a paper is in the open.
How popular such steps could prove remains to be seen. Post-publication commenting on papers remains infrequent. And the system of transparency does not necessarily reflect all the communication between editors and authors as well as referees. There can be a lot of email correspondence between parties that is not documented in official decision letters. And contrary to EMBO’s policy, I consider the facility for referees to submit confidential remarks to editors as useful.
Still, transparency to me is a valuable ingredient for any scientific debate. And as the process introduced at The EMBO Journal does not really seems to impact the peer review process, I consider it useful. What is your opinion? I’d love to hear it! Or alternatively, you should of course comment on Bernd’s article itself!
Disclaimer: The views expressed here are my own, personal thoughts on this topic and may not necessarily represent those of my employer.
Reference:
Pulverer, B. (2010). Transparency showcases strength of peer review Nature, 468 (7320), 29-31 DOI: 10.1038/468029a
Welcome to my personal blog, where I write about the latest exciting developments in the natural sciences and bring my own perspective to scientific trends in these areas.
November 10, 2010 at 16:09
I just completed an “open” peer review, which is different from a transparent one in that it all happens in confidence, but the authors and reviewers names are revealed to each other. I’m not exactly sure what the benefit is supposed to be, except that I was perhaps a bit more conscious about being kind in my tone. I expect it could discourage honest commentary — you don’t want to pis off the authors, especially if any of them are influential and could do something about a grudge. In this case one of them is rather famous, and the paper wasn’t very good. I feel I was forthright in my review, but it’s still a little bit awkward. Some people might choose to suck up.
November 10, 2010 at 21:09
Although as a political scientist I am working in a different discipline, I’m generally in favour of a more transparent scientific process like the one described, especially to be able to judge the final result of papers. I’d even say that you may sometimes find more productive insights by reading reviews or the author-reviewer discussions than by reading the actual paper.
However, I suppose that since the number of possible reviewers for each subfield is not endless and since the writing/arguing styles of reviewers should be quite recognisable, one will soon be able to recognise the same reviewer or even identify concrete persons simply by comparing a number of the published reviews.
That is not particularly problematic; it would simply mean that the reviewers would de facto be known to an informed audience after a sufficient number of reviews will have been published.
November 11, 2010 at 10:11
@Cervantes, @Ron: you both touch a bit on the topic of open peer review. The British Medical Journal for example has an open peer review where referee names are revealed. It seems to work well for them. But as also Bernd Pulverer mentioned, there are issues, e.g. if a young untenured researcher would get in the situation of writing a negative report on a big shot scientist. In many cases where I am involved as an editor, referees whose names are not disclosed feel much more free and open in their reports. Of course, in some rare cases they could abuse their hidden identity, but this is where in my opinion an editor would have to step in. (and I do that!)
But Ron is right, authors often guess who might be the referee, already in the existing system. Sometimes they get it right, in most cases not. But we never tell them of course 😉 But I see this of less concern. Unless a referee has been overruled, it would be a positive referee report where everyone is happy at the end, and the reports on rejected papers are not published. It seems an issue that hasn’t come up at the EMBO Journal yet.
November 14, 2010 at 18:04
An Estonian translation of this blog post is now available here:
http://www.fyysika.ee/uudised/?p=10554
Kindly translated by Jaan-Juhan Oidermaa from the Estonian Physical Society.
January 25, 2011 at 02:05
Joerg,
You’re a fantastic human being for getting this up so quickly!
I think that the one of the aspects of peer review (and the job of the editors) is to work through the details when there are opposing reviews. I think that this would really educate many scientists not only on the process but, more importantly, on the telling technical details of a scientific paper. I think that many of us wondered what the reviews for the arsenic-bacteria paper looked like after receiving such harsh criticism from other experts. Were there any dissenting opinions on that paper? How were they resolved? What did the reviewers see as the most important issue. Granted, scientists are always going to argue over certain details. But, I feel that there is a great deal that can be learned from making peer review accessible. I don’t know what Nature Publishing Group’s stance on this is. Do they bring in an extra reviewer? I am really curious about how the majority of these cases get resolved.
From the stand-point of a person who has reviewed many papers in the past, an open process would keep me on edge the whole time. I’d have to be at the top of my game while reviewing a manuscript. (This may be the un-spoken yet biggest complaint about the open process.) My former advisors always made sure that my tone was collegial. So, personally, worrying about how someone else views my comments was never the biggest worry for me.
As an author, I’d be really concerned about what I was told to put in a paper. We all know that there are things that we overlook when finishing a manuscript. But, what kind of reactions would there be when other scientists are made aware of our mis-steps.
In all, I think that transparent reviews should benefit the scientific endeavor. It really is an interesting topic.
January 25, 2011 at 10:30
Matt
(please note that this post was written a few weeks ago. The latest Nature Materials editorial from the February issue took a while to come through.)
Just in case: here is the link to the recent Nature Materials editorial on the same topic. Access if free for registered users of nature.com
http://dx.doi.org/10.1038/nmat2952
With respect to opposing reviewers: The referee reports are advice to us editors. Whether we need to assign an additional opinion depends whether we think we need it or not. If one reviewer raises pretty tough and seemingly valid technical concerns or wider issues (e.g. discovers a related paper) no further advice will be needed. It is really down to the individual circumstances.
As for the issues on mis-steps and the enhancement during the peer review process: well, finished papers can often look very polished and complete. What is often hidden is that perhaps in some of those cases the paper was beaten into shape by the reviewers. Such efforts (and they can be significant!) are not recognized at present.
Last but not least, please note that even as of now all the reviewers of a paper get to see the comments from the other reviewers…
Thanks again for your interesting comments!!
(and please email Nature Materials, too – we are interested in printing useful comments such as this one)
Joerg
October 25, 2011 at 22:39
Joerg,
as you know, I am a stance supporter of open review; at least, let’s have the comments of the reviewers (anomously if you like) and responses to them by the author,s published as extra supplementary information (without disclosing the reviewer if she/he doesn’t like it — fair enough). Having though the comments by the reviewers and responses published will –without doubht — be educational for all of us (and assist us to eductate students and younger peers), and as mentioned above, will also aid in giving credits to the reviewers. Indeed, how often did comment of referees often seriously assist in ‘polishing’ up a manuscript? In my experience many times (though, of course, we often only realise later, as we simply are too keen to get the work published).
I am up for it and hope for the necessary changes to get peer review more transparent!
Natalie
October 26, 2011 at 23:16
Hi Natalie
thanks a lot, excellent points! Sometimes the discussions between authors and reviewers are much longer than the paper itself, and reveal fascinating insights. At present, this is only for the benefit of authors, reviewers and the editor.
In future, we will come to see peer reviewing as the beginning of a debate, that thanks to transparent schemes can carry on immediately after publication of a paper. Isn’t that part of the purpose of publishing your research in the first place, to enable discussion between the researchers? So I agree, the consequential step is to open up this debate from the beginning, from the peer review.
Sadly, however, many journals seem not willing to do that, although I think the momentum behind such transparency is still increasing, so let’s see.
Joerg
June 27, 2012 at 14:51
Or yet another option is to make it double blind. While in many cases the general authorship of the paper can be guessed, it will be not an absolute certainty,