How to Determine Reliability of Nutritional Studies
Getting reliable information about diet and nutrition can be a challenge. Most of your diet and nutrition information is likely to come from nutritional studies discussed on TV and radio talk shows or in your daily newspaper.
Regardless of the source, nutrition news should always pass what you may call The Reasonableness Test. In other words, if a story or report or study sounds ridiculous, it probably is.
And then what happens? Weeks, months, or years down the road, a second, equally prestigious group of scientists publishes a study proving that the first group got it wrong: In fact, this study shows coffee has no effect on the risk of heart disease — and may even improve athletic performance; salt does not cause hypertension except in certain sensitive individuals; only some fatty foods are risky.
That leaves you, a layperson, on your own to come up with the answer. Never fear — you may not be a nutritionist, but that doesn’t mean you can’t apply a few common-sense rules to any study you read about. To assess a study, keep the following in mind:
Does this study include human beings? Animal studies can alert researchers to potential problems, but working with animals alone cannot give you conclusive proof.
Different species react differently to various chemicals and diseases. For example, mouse and rat embryos suffer no ill effects when their mothers are given thalidomide, the sedative that’s known to cause deformed fetal limbs when given to pregnant monkeys and human.
(Here’s an astounding turn: Modern research shows that thalidomide is beneficial for treating or preventing human skin problems related to Hansen’s disease [leprosy], cancer, and/or autoimmune conditions, such as rheumatoid arthritis, in which the body mistakenly attacks its own tissues.)
Are enough people in this study? The study must include sufficient numbers and a variety of individuals, too. If you don’t have several hundred to many thousand people in the study, the effect may have occurred by chance. A study must include different types of people: young and old men and women of different racial and ethnic groups.
Is there anything in the design or method of this study that may affect the accuracy of its conclusions? Some testing methods are more likely to lead to biased conclusions. For example, a retrospective study (which asks people to tell what they did in the past) is always considered less accurate than a study that follows people while they’re actually doing what the researchers are studying.
Are the study’s conclusions reasonable? When a study comes up with a conclusion that seems illogical to you, chances are it is. For example, in 1990, the long-running Nurses’ Study at the Harvard School of Public Health reported that a high-fat diet raised the risk of colon cancer. But the data showed a link only to diets high in beef. No link was found to diets high in dairy fat. In short, this study was begging for a second study to confirm (or deny) its results.