How Your Doctor Looks at Research: the Error Factor
Last reviewed by Faculty of Harvard Medical School on January 13, 2011
By Robert H. Shmerling, M.D.
If you look at most medical news stories years later, there is often little to show for them. The latest "groundbreaking research" has gone nowhere and the news has moved on to other "breakthroughs." Why does this happen?
In some ways, the answer is predictable. It happens because medical research is designed and performed by fallible humans using imperfect methods, because the news media want to attract and hold onto an audience, and because we all want hopeful, positive results. Each of these factors is at play and each plays havoc with our ability to tell the difference between an interesting but preliminary finding, with uncertain importance, and the "next big thing."
Of these, perhaps the one factor that dooms many scientific studies and is most difficult to eliminate is error. The fact is, no matter how careful and talented researchers may be, error is common in medical research.
In broad terms, medical researchers identify two main types of error:
Researchers take pains to minimize the chances of either type of error. For example, a study will try to anticipate every potential confounder. Once information about these factors is collected, there are statistical methods to account for them, eliminating their contribution to the results.
Frequently, however, there is an important variable that was not anticipated, because our knowledge of the relevant factors is simply incomplete. And studies may not recruit enough subjects to "even out" the occasional unusual or irrelevant result.
Because studies are expensive, require volunteers and the work of many physicians, nurses, pharmacists and other health professionals, they often last for the minimum amount of time thought necessary to answer the question of interest.
While studies lasting only a few months or up to a few years are common, we really need information about the longer term because:
Short-term studies are not reliable measures of the long-term. Unless the disease or symptom is a short-term issue, the inadequate follow-up of many studies is a significant limitation and source of error.
Yet, there are major challenges to performing long-term research, including lack of personnel (who change jobs or graduate), difficulty keeping track of the volunteers (who move away or grow weary of the monitoring), or changes in technology. For example, it may be impossible to compare the results of an MRI today with results from 15 years ago because the quality of images obtained has improved over that span of time.
Research begins by asking a question and then designing a way to answer it. But because of how research is performed, the question may change to be something related that is easier to study, but not exactly the ideal question. For example, there is a lot of focus on cholesterol, but it's cardiovascular disease and death that we really care about. Knowing that there's a connection between cholesterol and these other important outcomes has led researchers to focus on cholesterol results. So, instead of asking, "Does this medication prevent heart attack and prolong life?" we ask, "Does this medicine reduce cholesterol?"
Relying on research that uses "proxies" (a variable that "fills in" for the one you're really interested in) is not as reliable as those that directly look at the key endpoints of interest.
Animal studies are another way that the question is revised for the sake of study design. If you want to ask, "Does this medicine prevent age-related changes in the brain?" you can actually examine the brains of rats and mice in a way that you cannot for humans. So, the question becomes, "Does this medicine prevent age-related changes in the brains of mice?"
Finally, the question may be shaped based on who is paying for the research. While you may want to know, "Does this medicine work better than another medicine?" a pharmaceutical company may change it to "Is our medicine better than nothing?" Why? Because that's the minimum requirement of the U.S. Food and Drug Administration for approving a drug and allowing it to be marketed in this country.
That question may be much less interesting do you really want to take a medicine that's "better than nothing" or the best one among all options for treating your problem?
Medical research is vitally important, but it has some significant limitations, and some of them contribute to how the medical news you hear today may be contradicted by other research next week or next year. Here's my advice: Remain skeptical, recognize that it's rarely a good idea to bank on a single study, and keep in mind that there could always be error, no matter how good the research (or the researcher) sounds.
Someday, there may be major changes in our system of funding and executing medical research. We need funding sources that do not alter the question that is asked, we need better ways to minimize error, and we need research designs with a long-term vision. Increasingly, regulators (such as the FDA) are asking pharmaceutical companies to study new drugs even after their approval. This is a promising development. Still, medical research will never eliminate all sources of error. So, it's best to take your medical news with a grain of salt or perhaps a couple. It's likely that your doctor already does.
Robert H. Shmerling, M.D., is associate physician at Beth Israel Deaconess Medical Center and associate professor at Harvard Medical School. He has been a practicing rheumatologist for over 20 years at Beth Israel Deaconess Medical Center. He is an active teacher in the Internal Medicine Residency Program, serving as the Robinson Firm Chief. He is also a teacher in the Rheumatology Fellowship Program.