The authors write that "[t]here were no statistically significant changes pre vs post or between groups for any of the body composition variables." However, if the data in Table 2 are correct, then there appears to have been a statistically significant pre–post increase in BW and FFM for the HP group. Using a one-sample t test with 19 degrees of freedom, 2-tailed p-values for the change in BW and FFM are, respectively, .0008 and .002. Although these are raw p-values, they should be small enough to retain significance after adjustment for multiple comparisons. The authors state that they analyzed the data by using 2-way ANOVA, a method that may have been inappropriate given the unbalanced study design and the need to model subjects as a random factor.
Potentially more serious is the large drop-out rate (33%) in the HP group. After a large differential drop-out rate, the original randomization cannot be trusted to balance confounding variables between treatment groups. The study essentially becomes observational and will likely require control of confounding during the analysis; however, the authors give no indication of having considered confounding. Table 1 shows that at least one potentially serious confounder, gender, was substantially imbalanced, with the HP group comprising 45% females but the control group only 20%. When large differences like this are not controlled in the analysis, results can be distorted via Simpson's paradox, which can reduce a true between-group difference in an outcome variable to non-significance, zero, or even reverse its direction. It seems plausible that the observed lack of effect on fat mass of the addition of 800 kcal/d to the diet for eight weeks could be an artifact of Simpson's paradox.
Errors in the analysis
28 May 2014
The authors write that "[t]here were no statistically significant changes pre vs post or between groups for any of the body composition variables." However, if the data in Table 2 are correct, then there appears to have been a statistically significant pre–post increase in BW and FFM for the HP group. Using a one-sample t test with 19 degrees of freedom, 2-tailed p-values for the change in BW and FFM are, respectively, .0008 and .002. Although these are raw p-values, they should be small enough to retain significance after adjustment for multiple comparisons. The authors state that they analyzed the data by using 2-way ANOVA, a method that may have been inappropriate given the unbalanced study design and the need to model subjects as a random factor.
Potentially more serious is the large drop-out rate (33%) in the HP group. After a large differential drop-out rate, the original randomization cannot be trusted to balance confounding variables between treatment groups. The study essentially becomes observational and will likely require control of confounding during the analysis; however, the authors give no indication of having considered confounding. Table 1 shows that at least one potentially serious confounder, gender, was substantially imbalanced, with the HP group comprising 45% females but the control group only 20%. When large differences like this are not controlled in the analysis, results can be distorted via Simpson's paradox, which can reduce a true between-group difference in an outcome variable to non-significance, zero, or even reverse its direction. It seems plausible that the observed lack of effect on fat mass of the addition of 800 kcal/d to the diet for eight weeks could be an artifact of Simpson's paradox.
Competing interests
None.