The dark side of peer review

Peer review may provide a level of quality control but what can be done to improve it? Keith Smith explained to Graihagh Jackson how he ensures they're up to standard...

Keith - Well, the reviewers reports will always be read by the editor before being sent to the authors.

Graihagh - Keith Smith again, from the journal, Science.

Keith - There are various reasons why we need to do that; we need to check that there aren't personal attacks or inappropriate things in there, for example. But the editor will often have consulted several referees (maybe two or three), and will need to weigh up the opinions from the different referees who might disagree with each other. You might have one referee says "this paper is too long, it needs to be cut down," and another referee who says "this paper doesn't have enough detail in it, you need to add more method" and, obviously, you need to balance those things.

We use a system called "cross review." which is showing the reports to all the other referees and asking them to, essentially, check each other, but those are really new systems that are coming into effect. Most journals still reply upon the editor to do that collating.

Graihagh - And you mentioned checking reports for things like "inappropriate behaviour" - is that common and could you perhaps give me an example of where that's happened?

Keith - It's not common at all! I've been an editor for scientific journals for five or six years now and I've only seen it come up twice. That's over all the thousands of papers I've handled in that time.

There are occasions where somebody will put something in there that suggests they have not assessed the paper on the basis of the science within it but, instead, they've assessed the paper on the basis of who wrote it.

There was an example in the media a few months ago of a referee's report, which was sent to a group of female authors, which said in it "I suggest that you add a man to your author list in order to balance this paper out," and that was clearly not acceptable again. And how, in that case, it got past the editor and, to be fair, the editor resigned, but that's the sort of thing that we're checking for on a very, very basic level. But I am pleased to say it is very, very rare and, if we do see anything like, we immediately discard the report and get a different referee to look at it.

Graihagh - And we talked a little bit about the dark side of peer review, but is that the only sort of problem you get in peer review or are there other things that need fixing or improving upon?

Keith - So peer review is something that everybody agrees is not perfect; there is a lot of debate as to what exactly can be done to improve it. But some of the things we need to watch out for are referees who have a conflict of interest, for example. If I've asked a referee to assess a paper and it turns out they are working on a very similar piece of research themselves and they have it in process at another journal at the same time; they're trying to get their paper out first, then they might be tempted to deliberately slow the process down, and that is a real danger. We do ask referees to declare any possible conflicts of interest but not everybody does it. Sometimes you do find out but, often, only too late.

Graihagh - Although not perfect, the peer review is in place for a reason - it does provide a level of quality control but what could be done to improve it? Keith suggested more transparency, in particular making raw data available, so referees can also check that too, but my favourite solution involves an award, after all, everyone likes a certificate...

Keith - There has to be an incentive to referees to ensure that they are doing a good job and that they're doing it in a timely fashion. At the moment, if a referee is late with their report the worst thing that happens to them is they get an annoyed email from the editor because the only person who knows who they are is the editor. They don't suffer any damage to their reputation, for example, if they become known as being a slow referee.

Now I don't think we should be punishing referees in any way; they do this work for free and it's a community service. But perhaps we could be rewarding referees who do a good job. We could be, for example, at the end of the year on the journal website we could be saying "we especially these 20 referees who did a very good job for us this year." And that would encourage people to make sure they're doing a good referee job and that they're reviewing papers in a way they would want their own papers to be refereed at other journa