Peer review under the microscope

Peer review is something that we all take for granted. We wouldn’t dream of publishing something credible these days without it. But the system wasn’t always robust and someone who’s helped to really shake things up over the last 30 years, founding in the process an internationally recognised conference series called “The Peer Review Congress”, is former medical doctor and high altitude physiologist Drummond Rennie...

Drummond: What happened was that I was working in a huge cardiac clinic in Guy’s Hospital in the early ‘60s and I got interested in problems from blue babies, kids with congenital defects in the heart, and they couldn’t have their blood fully oxygenated. Now, you get blue at high altitude and I did all the research I could do in London. I moved to the US in 1967 and the research I did that went very well indeed. This was high altitude in the Andes, in the Himalayas, and Alaska, and the Yukon, and so on. And an extraordinary thing happened to me.

I’d finished some studies in the mid-1970s and I had what I consider in retrospect the luck to fall a long-ish way. Actually, a very long way down a mountain in the Canadian Yukon. From being a really second rate climber, I became, I suppose a fourth rate one and could no longer take full responsibility for experimental people who had agreed to be experimented on that I’d taken to high altitudes. The lucky part of that was that I was absolutely forced to change my career. I’d probably run out of ideas anyhow. It stopped me from getting killed later! 

I became the first Deputy Editor of the New England Journal. Looking at what I had to handle and deal with, absolute abysmal standards, the shear weight of rubbish, it was astounding to me. This stuff would flow in and we’d flow it back again. The good research was superb but I’d say a third of the research sent to us were trite rubbish. One of the common features that occurred to me of these trite rubbish papers was that they’d all passed rigorous peer review.

One way or another, I became Deputy Editor of JAMA, the Journal of the American Medical Association. At that time, it was in a bad state and I was intrigued by the idea of helping to rescue it. I found that people there were interested in peer review also but were prepared to go a little further. In 1986, I wrote an editorial announcing that we would back a conference to discuss peer review. But I put in and insisted we would only present research. I was fed up with editors just saying, “We’ve had this disaster with peer review, but our peer review system, we declare to be excellent on basis of no evidence whatsoever.”

So, what a nice thing to gather evidence and an even nicer thing to get clever people out there – if there were any – to find this evidence. And so in 1989, we held the first conference on this and it was an amazing event – angry arguments and a great deal of excitement. So, we’ve been having them every 4 years since. The first thing that happened was the amount of rubbish being printed was confirmed. So, these conferences during the ‘90s and the 2000s, early 2000s, gave a tremendous impetus to the idea that the literature was full of biases and to the literature that said, “These are ways of correcting all of these biases and then of helping journals follow guidelines that would enable evidence based reporting.” So, that is what happened very broadly.

The last thing I’d say is something a little different. There was one woman for every ten men at the first congress. That ratio has changed completely. My parents got married in Cambridge around 1933. Before that, my mother put herself through medical school in the 1920s. She had been very happy with that ratio, don’t you think...?