Unreliable antibodies

Apart from winning a Nobel prize for their discoverers, and along the way enabling millions of women to tell whether they’re pregnant, antibodies have been revolutionary for research. We use them therapeutically to detect and treat diseases; and we use them for research all the time to activate and block cell signals, and to reveal microanatomical features in histology. We’d like to think that we can rely on them as rock solid science: doing what they say on the tin. But, as Rick Khan, from Emory School of Medicine, and Peter McPherson, from McGill University, argue, as the industry has expanded, the detail, breadth and quality of the characterisations being carried out on the antibodies routinely used to drive research has nosedived. Scientists, they highlight, tend to take as read that if another practitioner has used and published work based on an antibody, then it’s probably okay for them to use it too; yet the actual performance in the context they’re seeking to use it may never have actually been ascertained. Once the implications for the reproducibility of research and integrity of our discoveries are appreciated, it’s impossible not to feel a shiver go down the spine. Rick first…

Rick - Antibodies are incredibly valuable and useful reagents. Initially they were developed in researchers lab and then shared with their colleagues. Over time this became such a big important field that companies began to distribute them and, as a result, work on demonstrating their utility and their proper use fell by the wayside. And this resulted in a wide scale problem of poorly characterised antibodies. And as a result, using them improperly or overly interpreting their data leading to science problems and or just spending money on an antibody that didn't work and they just waste money.

Chris - Indeed, because you highlight the scale of the problem in the manuscript you've written. Because you point out that a couple of decades ago we had a few thousand antibodies and now we've got north of 6 million.

Rick - Right. And it's increasing rapidly as a result of new technologies.

Chris - And there's just no kind of industry standard that says we have evaluated each of these and these are its operating parameters. This is what you can rely on, this is what you can't, this is the uncertainty, that's just not done for some of these antibodies.

Rick - One of the real problems here is they're such valuable reagents. They're used in a variety of different assays. So each use needs its own verification. And so for a company to sell an antibody and to do all the verification themselves, they'd be losing money.

Chris - Presumably then that means that there's going to be a trickle down effect. Because if I publish a paper having used one of these antibodies, any inherent shortcomings in that antibody will be reflected in my science. I've done good science. But the data may be flawed.

Rick - Correct.

Chris - Well what's the scale of that problem then? That sounds huge.

Rick - It is huge And part of the problem is the people using them don't always appreciate that. When you go to buy anything, you assume the seller has done proper characterisation and know that what they're selling you is legitimate. That is not true with the antibodies because the company's distributing them or simply the middleman distributing them.

Chris - What's it going to take to sort this out, Peter?

Peter - What it's really gonna take is more of a fundamental cultural change. We need to make the scientists understand that the reagents that they're using are not always up to the professional contemporary standards and they're using reagents that simply aren't validated.

Chris - Do you think that phenomena like this are what underpin parts of the reproducibility crisis that have been documented in certain areas of science.

Peter - The reproducibility crisis is real and we believe antibodies are a major contributor to that.

Chris - How cognisant, Rick, do you think scientists but also industry are of this problem and the scale of it?

Rick - That's a great question. <laugh>. It's hard to judge. I have run two different antibody core facilities where we generate and work with researchers and from that work it was clear very talented, very smart scientists simply don't understand the scope of the problem and they assume if we generate an antibody for them, it will do everything they want it to do. And it simply does not. You know, it costs money whether the companies do it, which they don't, or the researcher does it. It costs time and money to do that work. And too many of us are too eager to get on with the interesting experiment. And so we're not training our folks enough to realise, 'hey, you've got to first convince me in the world that that antibody is doing what you think it's doing in that particular usage.'

Chris - I suppose we've also got, Peter, a problem, both a forward direction problem but also a reverse direction problem, which is if we discover or if we implement some evaluation of antibodies and we discover, which inevitably we will, that some are unreliable for the use we're employing them for, we're then going to undermine the evidence base going backwards in time and cast doubt on publications backwards in time. But we're also then potentially going to cause problems going forward. Where do we go from here? Because we're relying on dodgy data from yesteryear, but also how does that inform how we change things going forward to do the next set of experiments?

Peter - Correct. So we looked at this and we've looked at what criteria do the scientists use. And one of the criteria that scientists use is, has this antibody been used in a previously published paper? And it turns out that's not a very good criteria for judging whether you should use an antibody or not. So you're absolutely right. It's a problem that propagates, a scientist looks at a paper, says, 'well this other scientist uses antibody, they used it in their paper, therefore I think that it must be a good reagent to use. I'll use it in my paper.' And it turns out it's really not.

Chris - The problem is if we say, well look, we've got this reproducibility crisis, so we're going to repeat experiments if we repeat experiments and use the published method that someone went through to reproduce their data and we use the same reagent with the same inherent flaw, we're going to get the same flawed outcome.

Peter - Exactly correct. And this is part of this cultural shift that's required to not think that just because an individual has used this antibody in a published manuscript or published study, that you can work on the assumption that it's the right reagent, it's a business model and a cultural model that needs to be changed and how to do that? These are not trivial questions, this is stuff that's been going on for a very long time. And part of what our initiative is trying to do is trying to figure out how to push the cultural change as much as the scientific change.

Chris - What do you think then, Rick, some kind of level from the funders down that says you have to use antibodies that have been approved in the following way or validated in the following way. So in the same way as we had a big shift towards open access publishing, for example, led by funders. Do you think that funders, and possibly also journals, should be more rigorous in this sort of thing and that's how we effect culture change?

Rick - That can be a big part of it. I have been in contact with the leadership at several of the institutes in the National Institute of Health, as well as the National Science Foundation, to bring up this issue and to encourage them to encourage change. And the first answer you get is they are not a regulatory body. So on one hand they don't want to take a heavy hand in this because that's not how they see their role. On the other hand, they have been increasingly asking for, when you submit a grant, to have a section where you describe and deal with the reproducibility issues. I think it's now inherent that one potential solution or partial solution is for reviewers of grants and papers to raise this issue and make sure that the folks doing the work understand it and will appropriately address it. That's really part of the cultural change that Peter is alluding to. I think the funding agencies can do a better job. I should mention also there are some private foundations, disease related and other, that are also doing a great job and they can be more persistent in demanding such work. And just pointing out one, the Michael J. Fox Foundation I think is doing a great job and I think they can be a prototype for others.