Kat Arney

But How do we Know it's True

Publication of Science Research

This is how we do it
Although it may seem an anathema to hard-core, fact-loving geeks, there is a rich seam of philosophy lurking behind science. The key concept that concerns us here is the principle of the "scientific method" - basically the way in which science works. In its purest form, the scientific method is a kind of question-and-answer game. You come up with an idea, or hypothesis, which asks a question (e.g. all people who carry gene "X" have disease "Y"), then do the research to either prove or disprove your idea (e.g. screen lots of people for gene "X" and see if they have the disease or not). Your hypothesis may just be a wacky idea, or it may be based on observation (in my example, noticing a few people with both gene "X" and disease "Y"). Once you have proved or disproved your hypothesis, you have moved forward the frontiers of science! Either you can say with certainty that your hypothesis holds true, at least under your experimental conditions, or that it does not. The next step is to formulate a new hypothesis. If your original idea turned out to hold true, does it work under all conditions and in all circumstances? Are there any exceptions? Can other people get the same results? Alternatively, if your idea turned out to be wrong then your next hypothesis might be another idea you think might be right. In this rather tortuous way, scientists have managed to demonstrate millions of principles about life, the universe and everything.

But another tenet of the scientific method is that experiments should be controlled. This doesn't mean that an uncontrolled experiment is one in which you flail wildly round the lab, rather that you are controlling things to be sure your method is working and that your results are reliable. Controls can be either "positive" or "negative", and both are important in experiments in all scientific disciplines from ecology to particle physics. Positive controls ensure that your experimental methods are actually working. This is like doing a "dead cert" experiment, where you know what should happen. If your positive control doesn't work, you can't trust the rest of your data as you can't be sure that the experiments were working for all the other samples you are investigating. Negative controls are basically the opposite (you expect something not to work), but are equally important. In the example I gave above, the negative control would be looking at people without disease "Y" to make sure none of them had gene "X". So, in a nutshell, science progresses by doing controlled experiments which attempt to prove or disprove a hypothesis. Let's assume we've done that, got some interesting results and now we want to tell the world!

Tell me about it
As a scientist, I can't just rattle off a few exciting experiments then phone up the newspapers and tell them to hold the front page. How can I be sure my experiments are reliable and that I have interpreted the results correctly? And how can I tell the rest of the scientific community about it? At this point, scientific journals play a key role. A multitude of journals are published around the world, brimming with new findings and ideas. You may have heard of some of the top ones like "Science" or "Nature", and they range from the all-encompassing (such as the "Proceedings of the National Academy of Sciences") to the highly specialised ("Blood", "Gut" and "Brain", to name but a few). Remember the furore that surrounded Dr. Arpad Pusztai's research showing that feeding GM potatoes to mice was harmful? The scientific community found it hard to trust research that was first published in a tabloid newspaper rather than in a respected journal.

But how do the journals ensure they are printing reliable data?
In order to get your research published in a journal, you first write it up as a research paper. This includes your original idea that you were trying to prove, the methods with which you tried to prove it, the results you found and what you think it all means. As much raw data as possible is included, such as photographs and measurements. The paper is then sent to the editor of a journal, who decides if it is the sort of thing they want to publish. Of course, not all journals are created equal, and some have a much higher profile in the scientific community than others. Generally, all journals publish reliable research, although the equation which dictates the standing of a particular publication is a complex one. This relates to how ground-breaking (or trendy) their papers are, how rigorously the results have been proved and also how interesting the research is to the wider scientific community.

The main thing that links virtually all the journals is the process of peer review. Once an editor is interested in a paper, it will be sent out to around three other senior scientists who work on similar things. These people know about the subject and the other experiments that have been done in that area. They will read the paper and assess it, ultimately reporting back to the journal editor whether they think it is genuine research or not. Often a paper will be returned to whence it came, with suggestions for new experiments to do or other interpretations of the results which must be addressed before the paper can be accepted for publication. Sometimes a paper may be completely panned by the critics and demand a serious rethink about the entire thing. For scientists the most frustrating thing can be when reviewers suggest that although the science is OK, the paper would be suited to a less high-profile journal. After a long and fretful process, the paper is finally accepted and published in the public domain. Science journalists will then see what new research is being published and write stories based on these papers, bringing the hottest science straight to your desktop.

Although this system of peer review works well, and seems to have maintained the integrity of the body of scientific knowledge over the years, there are a few holes in it. The principle problem is that the identity of the reviewers of your paper is hidden. In principle, this gives reviewers the freedom to make fair positive and negative criticism. Unfortunately, your identity is not hidden from them. In the worst case scenario your paper could be sent to a major competitor who might swipe your ideas, reject your paper then cash in on it themselves. Or they might be very close to publishing similar work themselves and deliberately try to stall the publication of your results. And, as we are all just human, people may choose to use the peer review process to grind personal axes or push through the papers of their friends.

One other problem with the journal system is that it is very hard to get negative results published. By this I mean results that disprove an idea. In the example I used above, the investigation might find that there is no strong evidence to link people with gene "X" and their likelihood of having disease "Y". Providing the study was controlled and thorough, this is still a valid piece of data. However, unless the established dogma is that the two things are linked (and you've just proved they're not), it's hardly going to set the world on fire. As a result, it may be difficult to get such work accepted by a journal and other researchers working on either gene "X" or disease "Y" will never know about it. People might then needlessly repeat the same experiments, wasting time and money, when it might be better to investigate other genes or diseases. Believe it or not, there is actually now a Journal of Negative Results, aiming to combat this problem.

So now you know how we know what we know is true! (At least, until someone proves it wrong…)

- July 2004

About the Author

Kat Arney is a writer and member of the naked scientists radio programme. She is based at Cancer Research UK