Commentary: Should you eat meat, or are you confounded by methodological debate?BMJ 1994; 308 doi: https://doi.org/10.1136/bmj.308.6945.1671 (Published 25 June 1994) Cite this as: BMJ 1994;308:1671
This paper raises several questions.1 The crux of interpretation is in the selection of the people in the study cohorts.
The non-meat eaters were volunteers, mainly recruited through advertisements and word of mouth in an organisation that promotes vegetarianism. Such volunteers will differ greatly from the general population, not only in age and sex, but also in education, social class, and other aspects of lifestyle. To counter these obvious objections the authors asked the non-meat eaters to recruit friends or relatives as controls. These people ate meat more regularly but belonged to the same social strata and might also have shared other habits.
As expected, the overall and cause specific mortalities were lower in both groups than in the general population. The reduction in mortality was greater among the non-meat eaters. There is, however, a possibility that the two groups were not comparable and that the “healthy person effect” might have been operating. People who are enthusiastic about their vegetarian life may have volunteered to prove how healthy vegetarianism is. Their meat eating friends and relatives would not be subject to such self selection. The authors performed an additional analysis omitting the first five years of follow up to allow the healthy person effect to wear off2: the difference between the meat and non-meat eaters diminished, but unfortunately the confidence intervals became too wide to allow meaningful interpretation.
Problems with statistical adjustment
About two thirds of the vegetarians were women, while about half of the slightly older meat eaters were men (apparently many young women asked their partners to volunteer). The authors tried to control for these differences by comparing the standardised mortality ratios of the meat eaters with those of the non-meat eaters. The two groups were therefore not directly compared with each other but with a third reference group (the general population). The problem is that the two standardised mortality ratios are not directly comparable, one relating mainly to young women and one mainly to slightly older men.
The idea that two standardised mortality ratios are not always comparable is well known.3 In practice this theoretical objection can often be ignored except when groups differ greatly.4 The question then becomes how large does a difference have to be before it calls for a direct comparison?
Other data available on both groups were their smoking habits, weight for height, and social class. As there were some differences - for example, the meat eaters were heavier - the authors tried to adjust for everything at once by Poisson regression, using the expected values as rate multipliers. This regression model calculates the ratio by which one standardised mortality ratio should be multiplied to arrive at the other. Since this Poisson regression is only a comparison between standardised mortality ratios, it remains a comparison between potentially incomparable quantities.4 A Cox regression analysis would have solved the problem because it compares the groups directly.
For those who prefer a Poisson regression a model using the person years of the two groups as the “rate multiplier” would have yielded a direct maximum likelihood comparison.
Two schools of thought exist about which type of bias is most important in observational studies. Some epidemiologists emphasise that selective recruitment is no problem in follow up studies and can even be beneficial. The archetype is the study on smoking and lung cancer in British doctors.5 The investigators contrasted a group of male doctors who were cigarette smokers with male doctors who were non- smokers. These groups were not representative of either the general population or the profession, but the two groups were thought to be comparable, and this guaranteed that the excess risk of cancer in smokers could be attributed to smoking. The investigators knew who to invite and invited everybody in a similar way. Even if all those invited did not participate (only two in three did) there was little scope for bias because lung cancer is an unexpected side effect of tobacco so people cannot (self) select themselves into a study. The effect of smoking on lung cancer was, however, terribly strong and might have overshadowed any remaining incomparability between the groups.
It also remains a matter of opinion whether this reasoning can be applied to the comparison between non-meat eaters and meat eaters. The vegetarians were volunteers, many of whom had given up meat because of the presumed benefits to health, while the meat eaters were selected afterwards. Epidemiology textbooks give plenty of examples where self selection of subjects has been held responsible for results that proved totally wrong.3
Other epidemiologists emphasise that the greatest problems in observational research relate to selection of the study population.6,7 They doubt that a few statistical adjustments for age, sex, slimness, smoking, and social class will wipe out all differences between the groups. Life, they say, is not caught in five variables. They would point out that the contrast in eating habits between the two groups in this study was crudely defined and not checked. This, however, would lead only to a blurring of the difference between the groups and not to a positive finding. Nevertheless, it makes it difficult to interpret what aspect of vegetarian life might be responsible for the findings.