Randomized controlled trials (RCTs) are generally regarded as the gold standard in assessing treatments. There are some good reasons for this, but there is also some tarnish on the gold standard.First, let me say that I am not talking about RCTs that are done badly or, worse, fraudulently. Those do exist, but the problems with badly or fraudulently done RCTs are similar to those for badly or fraudulently done observational studies.
Why are RCTs regarded as the gold standard and as “better” than observational studies? In one word: Randomization. In an RCT, subjects (usually people) are randomly assigned to treatment or control conditions. This randomization makes a huge range of statistical results applicable. In an observational study, on the other hand, this randomization does not take place. Therefore, any differences in results might be due to differences in who gets treated and who gets the placebo. So, that’s a clear win for RCTs. It’s widely acknowledged.
So where’s the tarnish?
First, while RCTs randomly assign people, they rarely randomly select people from the proper population. Indeed, RCTs usually come with inclusion and exclusion criteria that make it a distinctly non-random sample that gets into the trial. This affects generalization. For instance, an RCT for a drug might exclude people with “other medical conditions”. But, drugs are typically given to people who do have conditions. In addition, all RCTs are subject to self-selection: It is not ethical to force people into an RCT.
Second, people in RCTs behave differently than people in general. They are more closely observed. They are likely to get more attention than would patients in a regular medical practice. They may be more likely to adhere correctly to the regimen. They may, therefore over-represent the treatment effect that will occur in the real world and they may under-report side effects.
It comes down to internal vs. external validity. Validity can be defined as the extent to which a measure does what it purports to do. RCTs have much better internal validity than observational studies. They are less subject to bias because people are randomly assigned. But they are more subject to external validity: They measure what they measure quite well, but it is not exactly what they ought to measure.
Even when both RCTs and observational studies are available, we need to look at both and make decisions based on reason, not simply a knee-jerk reaction that the RCT must be right.