In my previous post, I went over some of the ways that information can be bastardized from person to person or in the media’s (mis)representation of academic studies. Today we’re going to look at the actual study design and explore how the type of study conducted has a huge impact on what kind of conclusions can be made.
If you recall from my last post, the scientific method is the blueprint to use for setting up a study. Once we have asked our question and formed our hypothesis, we have to figure out what kind of experiment to run; in other words we have to pick what sort of study we will use. There are 5 types of studies, of which two are by far the most common in the health and nutrition world.
Randomized Controlled Trials (RCTs)
This is the gold standard for experimental design. Essentially all the participants of the study are divided up randomly, and the variable being studied is randomly assigned to different groups. For example, 100 people are selected to take a new supplement. Half of them are given the supplement, half of them are given the placebo, and neither group knows which one they received. The best way to do this is a “double blind” RCT, in which the researchers have their aides administer the supplement/placebo, so that both the subjects and scientists don’t know who’s in what group until the experiment ends. This is a great way to eliminate any possible bias. Obviously not all studies can be done double blinded, such as testing out different macronutrient ratios among subjects or different amounts of weight training volume. RCTs are the only type of study that can confidently conclude cause and effect within the study.
While RCTs can be amazing ways to prove cause and effect, there are definitely some issues that can arise that will negatively impact the conclusiveness of the results. Sample size, setting, and confounding variables can all affect the validity of the study. A very well designed study that only looks at 10 people probably has too small a sample size to draw conclusions from - had the same study examined 100 people it would be a lot more applicable to the real world, and have better statistical power. Most studies are conducted in a setting that won’t be all that similar to what would happen in the real world, so this can also be a problem based on what’s being studied.
The biggest issue in RCTs usually comes in the form of confounding variables. When you are designing an experiment, you form a hypothesis on an independent variable, which will affect your outcome, the dependent variable. For example, your hypothesis could be “A high sugar diet will negatively impact blood markers and make you gain more weight than a low sugar diet.” The independent variable here would be daily sugar intake that the subjects consumed, and the dependent variables would be whatever markers the researches decided to measure. The main confounding variables here would be the non sugar parts of the diet. To eliminate them, the researchers would want to make sure the subjects all ate the same amount of calories and macronutrients, as those would obviously have an effect on the outcome. In fact this was an actual study done in 1997, and the results were that so long as total calories and macros were equated, sugar content didn’t impact health, and they were comparing 43% of calories from sugar compared to 4%! Carbs were 71% of the diet so both these groups had equal amounts of carbs, just one of them was far more sugar than the other. This is one of the main reasons keeping calories in check is still far and away the most important thing with whatever diet you choose. For more info on the science of sugar, check out this article which includes links to research.
Cohort Studies
Cohort studies are observational studies that examine large groups of people over long periods of time and then compare different variables with different outcomes. For example, cohort studies could compare people who regularly exercised with those who didn’t and find trends in different health markers like blood pressure or bone mineral density. They can either be prospective, in which the groups are picked and then observed over years or even decades to gather information, or retrospective, in which the groups are gathered and take questionnaires.
All the studies linking various things to cancer in humans are conducted as cohort studies and not RCTs for obvious reasons, however there are sometimes rat studies done as RCTs to determine cancerous effects of various substances on mammals. Cohort studies are great for identifying trends amongst large groups of people, but one of their main downfalls is that there are essentially limitless confounding variables when you’re just surveying people on their habits; surveying people on their habits is itself the other main issue. In the context of nutrition, asking people to accurately describe their daily eating habits in terms of grams or ounces of certain foods per they eat per day is not very accurate, let alone when they’re supposed to recall months or even years worth of eating. That being said, cohort studies are awesome to guide more research in the form of RCTs to pick apart single variables and test them to see what sort of effects they may have on people.
Case Control Studies
Case control studies are also observational studies, but these studies attempt to look at a very similar demographic of people, with the exception of some of the people either having a certain habit or disease that the rest of the group does not. These are often used to learn about diseases to determine risk factors that might impact them, but still have some of the same “self report” issues that cohort studies have.
Cross-Sectional Studies
These are very simple, survey based studies that usually just ask a random sample of a certain population a few questions and share the data. Classifying states as republican or democratic would be a great example. These have little to no use in health and fitness outside of determining different fads or trends.
Qualitative Studies
Qualitative studies give us information on a more personal and human level. An example would be surveying individuals fighting a certain disease to understand what life is like with it. These are not based on numbers or statistics whatsoever.
In addition to various types of data, there are a couple other ways we review scientific data that are worth mentioning. You may have heard of the terms systematic review or meta-analysis before. A systematic review is when all of the available scientific evidence we have on a certain subject is reviewed and summarized. A meta analysis is the term used to described running all this data through various statistical equations to arrive at a qualitative answer.
For example, “What is the optimal amount of protein per pound of bodyweight to consume if your goal is to gain muscle mass?” Different studies will undoubtedly come up with somewhat different results, but so long as the majority of the studies were conducted using sound experimental design, and the scientists did a good job of collecting the data, we can be relatively sure that we will find our answer. These are very useful for health and nutrition, and are how I can give nutritional recommendations without feeling like I have to sift through hundreds of research papers to know that what I am saying is backed in science, as someone else has already done it for me.
I want to tie a few things back to part one of this article series. In the RCT portion of this post, I linked an article about sugar science written by Dr. Layne Norton, a relatively prominent name in the fitness industry with his PhD in nutrition. He helped popularize the terms “Flexible Dieting” and “If it fits your macros” or IIFYM, which essentially means you are free to include some “less healthy” foods in your diet so long as your control your macros and calories. He sometimes takes heat from nutrition zealots that feel their diet (keto, vega, paleo, etc.) is the only way to be healthy, but he always uses science to back up his viewpoints. The sugar study I linked had one group of women consume 11 grams of sugar per day and the other consumed 118 grams, and they showed almost zero differences between groups. This does not mean he (or I) are saying “It’s healthy to eat 10 times the amount of sugar you're currently consuming." That is not what the study said: it said if calories and macros are controlled, sugar intake probably doesn't really matter. However, after he posted that article, he came under fire for being “pro sugar.” Issues like this are one of the reasons that people get confused about science - often it's the media or other people taking data and twisting it.
In some extremely unfortunate cases, it's the people who conduct the study themselves who do this. If you haven’t heard about it by now, there is a vegan propaganda “documentary” on Netflix right now called The Game Changers. In this movie, Dr. Dean Ornish claims that a vegetarian diet is the only diet that is backed by research that will reverse coronary heart disease. He should be familiar with it, because he conducted it. What he neglects to mention whatsoever is that the study didn’t compare a plant-based diet to an omnivorous one.
“In a prospective, randomized, controlled trial to determine whether comprehensive lifestyle changes affect coronary atherosclerosis after 1 year, 28 patients were assigned to an experimental group (low-fat vegetarian diet, stopping smoking, stress management training, and moderate exercise) and 20 to a usual-care control group.”
If you have been paying attention throughout this article and the prior one, you won’t need me to explain the problem with his statement. It's no wonder that people are so confused about health and nutrition when we have unscrupulous individuals in the industry who stand to make money off of our ignorance spouting off misinformation and lies. However, perfectly timed as I was writing this article, I received an email from my nutritionist certification company with this article that should clear up some stuff for you about nutrition. I cannot recommend you read and share it enough!
Next month, we’ll finish this science series and use some of what was discussed the last two posts to break down certain parts of The Game Changers. For anyone reading this who has changed their lifestyle due to watching this movie, I would urge you to read this article in the meantime as it dives way deeper into the movie's claims than I will.
The Chopping Block offers cooking classes for every type of lifestyle from vegetarian to vegan to meat lover. No matter what you choose, the key is to have a balanced diet. Learn how to eat healthy without sacrificing flavor in the upcoming Wellness Boot Camp.
