Michael McLeod · @Michael300891
5th Mar 2014 from TwitLonger
Many scientists across the world have already criticized the methods employed in the recent protein-mortality study grabbing media headlines (http://www.cell.com/cell-metabolism/abstract/S1550-4131%2814%2900062-X ) but the issues with the statistical approach employed has gone largely overlooked.
From in-depth inspection it becomes apparent that the study was geared up to investigate a correlation between protein intake and diabetes mortality: This is the first statistical model they ran, and also the first thing reported in the results section.
Although an effect was found, the authors probably realized a sample size of 1 (low-protein, diabetic mortality, see supplementary tables: http://dx.doi.org/10.1016/j.cmet.2014.02.006) as a reference to calculate confidence intervals was not going to hold much ground.
Following this, the authors move on to examine the correlation between protein and cancer mortality across the entire >50 population. Having run two models, they have still not found any condemning evidence against moderate or high protein intake. “Cox proportional hazard models were rerun, testing for an interaction between protein consumption and age.”
Finally, having found no main effect, the authors publicly state “Based on these results, we stratified the population into two age groups…. and re-examined relationships between protein and cause-specific mortality”.
At this point, the authors have run two statistical models and arbitrarily selected subset samples for their third analysis. If you are going to arbitrarily select sub-samples, you may as well through your statistical analysis in the bin as your going to (no matter how good your intentions) inevitably preference your bias.
Knowing that one in twenty statistical models result in a false-positive finding, the authors have publicly declared running three, but how many did they actually run before finding this ‘protein does something bad’ correlation.
Whilst this data-trawling approach is certainly not unique to this publication, a progressive statistical analysis examining negative correlations against protein, is concerning, especially when considering that the head author is the founder of L-Nutra, and thus would financially benefit from a downfall in protein. As is often quoted, inappropriate use of statistics can always find a significant result.
As for the corrections of animal vs plant protein, the issues here have already been highlighted, if you ‘account’ for the animal protein intake, which was the substantial majority of protein intake, any effects are likely to dissipate.