Randomisation is a scientific method of eliminating the effects of factors that may conceivably affect the outcome of controlled trials, but which we do not wish to examine. Carelessness in the randomisation process reduces the validity of the results.
Randomised controlled trials are medical science’s answer to the controlled experiments of physics in which the researcher keeps all factors constant apart from the one whose effect is being examined (1, 2). What constitutes a healthy diet varies in line with the most recent opinions, and to some extent nutritional research has acquired a dubious reputation (3). However, many of us believe in the Mediterranean diet, a set of nutritional guidelines from over 50 years ago that was intended to explain why the population around the Mediterranean (and Portugal) were less vulnerable to cardiovascular disease than those of us who lived further north in Europe (4). Moreover, in 2013, we found further backing for our belief: the New England Journal of Medicine published an article about a randomised controlled trial in which 7447 women and men aged 55–80 years at high risk of cardiovascular disease were randomly divided into three groups, one of which received a low-fat diet, while the other two followed a Mediterranean diet with extra supplements of nuts or olive oil (5).
Randomised controlled trials are medical science’s answer to the controlled experiments of physics in which the researcher keeps all factors constant apart from the one whose effect is being examined
The results were convincing. The two groups with a Mediterranean diet and nuts or olive oil showed a 30 % lower incidence of cardiovascular disease during the follow-up. The article has been cited over 3500 times, and a search on ‘PREDIMED’ (Prevención con Dieta Mediterránea) results in more than 311 000 hits. The message became popular.
‘Irregular procedures’
However, the article was retracted five years later in June 2018 (6). What had happened? The authors explained in a letter to the New England Journal of Medicine dated 13 June 2018 that they had retracted the article due to ‘irregularities in our randomization procedures’, at the same time as they published a revised version of the original trial (7). The ‘irregularities’ were due to the fact that participants belonging to the same household had been given the same type of food. This turned out to include 10 % of the participants. In addition, an entire village had been put on the same diet, and elsewhere the randomisation had not been carried out in accordance with the study protocol. In other words, the participants had not been randomly distributed to the three groups. Unfortunately, the 2013 report did not take this into account. The results were based on the random distribution of individuals to three groups. It would have been possible to distribute by household or even by village, but such a design is less effective than randomisation at the individual level. In order to achieve publishable results, the Spanish researchers were forced to exclude the 1588 individuals who were non-randomly distributed. The new analysis stressed that these results were not based on a randomised trial (7).
Anaesthesiologist with an eye for p-values
The article was retracted as a result of external pressure. In 2017, John B. Carlisle published an article in Anaesthesia in which he described the results of a systematic review of 5087 randomised controlled trials with a focus on data fabrication and non-randomised distribution (8). He pointed to anomalies in a number of articles, including the 2013 report, and consequently started the story of irregular conditions in the Spanish study.
Carlisle examined baseline values in the randomised trials for continuous variables such as height, weight, age, blood pressure etc. Then he compared the p-values between the groups at the start of the study, and the anticipated distribution of the p-values in relation to what had been observed. In principle, a successful randomisation should result in fairly similar groups in terms of background variables, with a p-value of around 0.5. Carlisle found deviations in 15.6 % of the 5087 trials he reviewed.
Poor statistics and fabricated data
Some of the deviations Carlisle found could be explained by the confusion of standard deviation and standard error, leading to the calculation of incorrect p-values. Others appeared to be unintentional errors because of muddled statistics, or – even more alarming – indicated fabricated data. Carlisle emphasised that the probability of finding incorrect data or scientific misconduct is greater in trials with extreme distributions of mean values.
For we readers of scientific articles it is more important that researchers follow their study protocols
Estruch et al.’s work was given a new chance. This was a generous act on the part of the New England Journal of Medicine and was perhaps influenced by the editorial staff’s views on the Mediterranean diet. However, for we readers of scientific articles it is more important that researchers follow their study protocols and analyse the results of trials in line with the intention-to-treat principle. When the principles of randomisation are violated, this opens the door to unknown confounding variables and uncertain results.
