All That Matters

I’m glad I’m travelling this week, as yesterday the impact factors have been announced. I actually forgot about the pending announcement yesterday, when some of my hosts here in Singapore reminded me about it at dinner(!). And sure enough, last night twitter was all abuzz with impact factors.

The reason I am happy I’m away from all the buzz is that the impact factors tend to get quite overemphasized. I mean, really, what does it mean, ‘impact’ factor. Per definition, the 2010 impact factor counts the citations in 2010 to papers published in 2008 and 2009, divided by the number of those papers. For more details, there is an explainer by Thomson Reuters, who publish the Impact Factors.

So what kind of impact does it measure? That of papers published there? Well, citations to papers vary a lot. Take my journal, Nature Materials. The most cited paper from 2008 for example is a review on Biosensing with plasmonic nanosensors that has 473 citations so far – coincidentally, that’s a review that I commissioned, and sure I’m glad I did. Going further down the ranks of the most cited papers from 2008, the paper at 10th place got 175 citations so far, that at 20th place 104. In other words – the impact factor has not much to do with the quality of a single paper, the distribution of citations varies a lot.

Therefore, the impact factor certainly doesn’t measure the ‘impact’ of papers, and by implication nor does it measure that of researchers. It provides an average number for a journal. But that even that doesn’t look like an absolute measure to me either. The impact factor of Nature Materials is 29.897 (yes, it’s calculated with such silly accuracy). In comparison, that of our sister journal Nature Physics is 18.423.  So does that mean that Nature Materials is 62% better than Nature Physics? You better start reading the journals, as that’s certainly not the case. Of course, a journal like Nature Materials might be perceived to publish papers on average better than some journals with an impact factor of say below 10. But as a researcher you would have already known that from reading the papers published in a journal, wouldn’t you?

So what use is the impact factor number? Well, being cynical one could say it is a quick measure for those that don’t read the journals but still want to know how good they are on average. The danger is of course that this is then used as a kind of metric to assess the quality of research or to decide on the career of researchers. As it’s clear from the examples above, it certainly should not be used for that purpose.There are better ways to judge the merits of published research, such as article-based metrics and not journal-based ones. Not even my salary as an editor depends in any way on the impact factor of the journal I edit, so certainly it shouldn’t impact those of researchers.

And that’s also because the impact factor is a woefully short-term metric. On Monday at the conference here in Singapore I listened to an interesting talk by Jonathan Adams from Thomson Reuters, and he showed a citation statistics whereby for most disciplines in the physical sciences the number of citations to papers steadily increases over the years until it reaches a maximum at around 12 years (give or take a few years). So even on average for all publications considered, measuring citations for only the last two recent years can mask the true impact of a paper.

Where such short-term metrics can be useful, however, is as evidence for considerable editorial efforts by a journal. For example in case of the remarkable 30% increase in impact factor for Nature Materials’ competitor, Advanced Materials, whose impact factor now stands at 10.857. My congratulations to them on their hard work! But all in all, we shouldn’t overplay the relevance of impact factors.

Photo by UncaughtException via flickr.

This week Science magazine has an interesting special issue on scientific data, covering a variety of topics from data backup and data visualization to open data. It seems these contributions are free to access for registered user of their web site, and it certainly is worthwhile to have a look.

The editorial in particular lays out Science’s policy on open data. Sharing scientific results is of course a motivation for publishing a paper in the first place. And to allow for independent verification of scientific results, the data contained in a publication has to be available and shared with other scientists. This sharing has to be done in a permanent way that guarantees access to archives also in future.

Is the data analysis traceable?

However, there is another point that hasn’t come across that strong from this special issue, but one that I also consider to be very important. And that is that data processing itself needs to be tracked, by which I mean the steps from the raw scientific data as measured all the way to the plots in a scientific paper need to be traceable. […]

As an editor of a scientific journal, one of my key duties is to organise the peer review of submitted scientific papers. There, I ask other experts to take a look at a paper and let me know their opinion on technical correctness of their findings, and perhaps also what the importance and impact of a paper could be. The reviewers are aware of the identity of the authors, whereas the identity of the reviewers is not revealed to the authors.

The requirement to use peer review is not set in stone, but it has proven a very useful tool to assess a scientific paper. However, given the huge amount of work involved where scientists review each other’s work. Indeed, a lot has been said about the peer review process, whether it should be opened, or completely abolished etc. Here I just like to focus on the issue of transparency, which has been subject of a commentary by Bernd Pulverer from the European Molecular Biology Organization in last week’s issue of Nature. Access is free.

Historically, peer review as such is known for a long time, but is only systematically been used since around the mid-20th century. Certainly the very idea of peer review has been a new concept to Albert Einstein, when following peer review his paper on gravitational waves was rejected by Physical Review in 1936:

Dear Sir,

We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the — in any case erroneous — comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.

Respectfully,
Albert Einstein

And even later on, peer review has not necessarily always been done. The famous 1953 Nature paper by Watson and Crick on the structure of DNA has not been peer reviewed. Competition was very tough in this case, and as John Maddox, former editor of Nature, allegedly said:

The Watson and Crick paper was not peer-reviewed by Nature… the paper could not have been refereed: its correctness is self-evident. No referee working in the field … could have kept his mouth shut once he saw the structure

Clearly, peer review was not always considered necessary by scientists as well as publishers. To me, it remains essential. But I think the system could improve, and one area where this could be done with ease is its transparency. At the moment, the process is not fully transparent, neither to authors nor to scientists other than the reviewers. As it stands, there is a lot of implicit trust in the work of journal editors…

More transparency!

Bernd Pulverer’s commentary describes an effort to increase transparency at The EMBO journal. Since 2009, the journal has been running a trial where in case of publication the anonymized referee reports sent to authors, the editorial decision letters as well as the author’s rebuttals to reviewers are published as a supplementary file along with the paper. (disclaimer: Nature Publishing Group publishes this journal on behalf of EMBO).

[…]