What people say they ate

Much of what you've read about food and long-term health rests on one kind of measurement: somebody asked people what they ate. When that measurement was checked against the body, in five large studies pooled together, the calories people reported on a food questionnaire correlated with the calories their bodies actually burned at 0.21.⁵ A correlation of 1 would be perfect agreement and 0 would be none. This lesson is about what that number means, why two camps of serious researchers read it so differently, and what happened when the field stopped asking and ran what its defenders call "the most expensive human study ever conducted".³

Three ways of asking

Nutrition research asks people about their diet in three main ways, and you'll meet their names in the methods of the studies below.

A food frequency questionnaire, or FFQ, asks how often you eat a list of foods. It's the instrument of the big cohorts: the ones behind lesson 1's saturated-fat table answered one every four years.¹⁰ A 24-hour recall asks what you ate yesterday. A food record has you write it down as you go. The 2015 reply to the critics that this lesson reads below groups recalls and records together as "short-term" instruments and calls the questionnaire "long-term".⁴

Diet has the gap between asking and measuring that Focus and Deep Work lesson 1 taught, and something most subjects lack: a measure of the truth that doesn't depend on anyone's memory.

Checking memory against the body

Two measurements sit underneath this whole lesson. Doubly labelled water gives a person's total energy expenditure without asking them anything, and in someone whose weight is steady, what they burn equals what they eat.⁶ Urinary nitrogen does the same job for protein. The OPEN study calls the pair "unbiased biomarkers of energy and protein intakes", and Freedman's pooled study calls the two of them recovery biomarkers.⁶⁵ Doubly labelled water rests on assumptions of its own, and lesson 3 takes up the argument over them.

The OPEN study was run in Montgomery County, Maryland, from September 1999 to March 2000. It gave 484 men and women aged 40 to 69 both a food questionnaire and 24-hour recalls, and checked both against doubly labelled water and urinary nitrogen.⁶ This course read the abstract.

Then came the sentence that matters most for the rest of this lesson: "There was little underreporting of the percentage of energy from protein for men or women."⁶ On the questionnaire people got the amounts wrong by about a third, and the abstract says, without naming an instrument, that the proportion came out nearly right. Hold on to that. It's the basis of the defence you'll meet below.

Five studies pooled

In 2014 Freedman and colleagues pooled five large US validation studies from 1999 to 2009.⁵ Walter Willett, one of the defenders you'll meet below, is a co-author. This course read the abstract.

It reports correlations between what people said and the biomarker truth, for three things: total energy, total protein, and protein density (protein as a share of energy).

The chart puts the abstract's nine figures on one scale.⁵ Read it in order.

Energy is the worst-measured item on every instrument. Averaging three recalls helps everything: the abstract says so in terms, "the use of multiple 24-hour recalls substantially increases the correlations when compared with a single 24-hour recall".⁵ And on the questionnaire, protein as a share of energy (0.41) does much better than protein in grams (0.29). The abstract draws that conclusion too: "FFQs have stronger correlations with truth for protein density than for absolute protein intake".⁵ On the recalls it is the other way round, though only just.

Then the size of the gap: "The average rate of under-reporting of energy intake was 28% with a FFQ and 15% with a single 24-hour recall, but the percentages were lower for protein."⁵ And the detail the critics care about most: "Personal characteristics related to under-reporting were body mass index, educational level, and age."⁵ Body mass index is weight relative to height.

Two kinds of error, and why the difference matters

A measurement can be wrong in two ways that do opposite things to a study. The reasoning in this section is the course's own, built on the findings above.

Noise that hides a real effect

Suppose a food doubles the risk of some disease. Take 1,000 people: 500 truly eat a lot of it and have a 4 percent risk, and 500 eat little and have a 2 percent risk. The true risk ratio is 2.

Now give them a questionnaire that puts 30 percent of each group in the wrong box, at random. The people it reports as heavy eaters are 70 percent true heavy eaters and 30 percent true light ones, so their risk is 0.7 × 4 + 0.3 × 2, which is 3.4 percent.

This is the comfortable kind of error for a defender of cohorts. If the only problem were noise of that kind in the food itself, the real effect of that food would tend to be larger than the study reports, not smaller. Noise in the things a study adjusts for is another matter: adjusting for a badly measured variable removes only part of its influence, which can leave a false association behind.

A bias that follows the people

The uncomfortable kind is error that is not random, and the pooled studies found it. Body mass index "strongly predicts under-reporting of energy and protein intakes".⁵ So heavier people's reports fall further below the truth than lighter people's do.

Archer's analysis of the US national survey, NHANES, points the same way. From 1971 to 2010, reported intake fell short of estimated expenditure by 281 calories a day for men and 365 for women on average, and by 716 and 856 for men and women with obesity.¹ This course read the abstract. One caution: in that study expenditure was estimated from equations, not measured with doubly labelled water, so "short" depends on the equations.

Now think about what that does. Suppose a disease is more common in heavier people. If heavier people also report eating less than they do, then in a cohort the people who go on to get that disease will tend to have reported eating less. That can produce an association nobody's food caused, or cancel one that's real. Again this is the course's reasoning from the body mass index finding, not a calculation any of these studies ran.

Epidemiologists call the first kind of error nondifferential and the second differential. John Ioannidis, whom lesson 1 met as a critic of the field, puts his worry in one line: "Indeed, self-reported data have error, but there is no guarantee it is nondifferential."² The pooled finding on body mass index is the evidence that his worry is not hypothetical.

What survives the error

If calories are measured this badly, why does anyone still use the questionnaires? The defenders' answer has two parts, and both come out of the data you've already seen.

First, amounts and rankings are different questions. For many questions a cohort does not need to know that you ate 2,400 calories, only whether you ate more of something than the person next to you. The defenders argue, in this course's summary of their position, that the error is smaller for ranking people and for nutrients adjusted for energy than it is for absolute amounts, and that it can be corrected for.³⁴

Be careful with that argument when it comes to energy itself. A correlation of 0.21 is a poor ranking too. The defence is strongest for the things in the chart that did better: protein density on the questionnaire, and averages of several recalls.

Second, proportions can survive when amounts do not. The reason is arithmetic, and the arithmetic is the course's own.

A day that shrank

Take somebody who truly eats 2,500 calories and 100 grams of protein. Protein has 4 calories a gram, so that's 400 calories from protein, 16 percent of the day.

Suppose they under-report every food by the same 28 percent. They report 1,800 calories (2,500 × 0.72) and 72 grams of protein (100 × 0.72). Their reported protein is 288 calories, and 288 out of 1,800 is still 16 percent. The amounts are badly wrong and the share is exactly right.

Real people do not shrink every food by the same fraction; the pooled studies found protein under-reported less than energy.⁵ But OPEN's finding of "little underreporting of the percentage of energy from protein" is what this arithmetic predicts when the shrinking is roughly even.⁶

The arithmetic fits the defenders' third recommendation. Subar and colleagues, dietary-assessment researchers at the US National Cancer Institute and allied institutions, wrote in 2015: "do not use self-reported energy intake as a measure of true energy intake", and in the same breath "do use self-reported energy intake for energy adjustment of other self-reported dietary constituents to improve risk estimation in studies of diet-health associations".⁴ This course read the abstract. Energy adjustment, in the form this lesson has worked, means expressing a nutrient relative to reported calories, as a share or a density. That is this course's gloss, not the reply's.

One catch. The 28 percent is an average across five studies' participants. OPEN's figures show how unevenly it falls: on the questionnaire, about a third of the men were under-reporting both energy and protein, and most weren't.⁶ So you can't take one person's questionnaire total and divide it by 0.72. Freedman's team found that "Calibration equations for true intake that included personal characteristics provided improved prediction", which is a way of correcting a study's numbers on average, not a way of correcting yours.⁵

The critics and the defenders

Now you've seen the data, you can read the argument about it. Both sides have read the same biomarker studies. They draw different conclusions.

The critics

Edward Archer and colleagues make the most absolute case. In 2015 they wrote that "the subjective (ie, not publicly accessible) mental phenomena (ie, memories) from which M-BM data are derived cannot be independently observed, quantified, or falsified; as such, these data are pseudoscientific and inadmissible in scientific research."⁷ (M-BM is their term: memory-based dietary assessment methods.) Their conclusion: "M-BM data cannot be used to inform national dietary guidelines".⁷ In 2018 they described what the field had collected as "millions of unverified verbal and textual reports of memories of perceptions of dietary intake."⁸ This course read both at abstract level.

Part of their argument rests on things this lesson has shown: under-reporting that is large, varies from person to person and follows body weight. Their 2013 analysis of the national survey concluded that "EI data on the majority of respondents (67.3% of women and 58.7% of men) were not physiologically plausible."¹ (EI is energy intake.) But the core of their case isn't about the size of the error. The 2015 paper sets out four numbered arguments, among them that the protocols "mimic procedures known to induce false recall".⁷ The 2018 paper says the reports were "impermissibly transformed (i.e., pseudo-quantified) into proxy-estimates of nutrient and caloric consumption".⁸ Read against the defence, that sentence answers energy adjustment too: if the numbers were never measurements, dividing one by another does not make them one. That's this course's reading; the abstract does not mention energy adjustment.

The critics' strongest card is one the defenders dealt them: the 0.21 for self-reported energy comes from a pooled study with Willett and Subar among its authors.⁵

Ioannidis is the more measured critic. He calls for reform rather than for throwing the data away: "the emerging picture of nutritional epidemiology is difficult to reconcile with good scientific principles. The field needs radical reform."² He proposes open cohort data, standardised analyses of every nutritional factor measured, and more large trials. Lesson 1 met him on confounding. Here his point is the one you've just worked through: error that may be differential.

The defenders

The defence comes from inside the field. Satija, Yu, Willett and Hu, whom lesson 1 met, grant the error and deny that it's disabling. They write that "most dietary assessment methods have a component of error", and in the next sentence that the methods "have shown good validity with use of multiple criteria".³ They also argue that biomarkers cannot replace asking, since "many foods and nutrients lack sensitive or specific biomarkers".³ Hold "good validity" next to the 0.21 in the chart. Both can stand only if validity means something narrower than getting calories right, which is the defenders' argument about rankings and shares that you met above. Their case is that the critics have misread the field: "These criticisms, to a large degree, stem from a misunderstanding of the methodologic issues of the field and the inappropriate use of the drug trial paradigm in nutrition research."³

Subar and colleagues answer Archer directly. Their first recommendation is to "continue to collect self-report dietary intake data because they contain valuable, rich, and critical information about foods and beverages consumed by populations that can be used to inform nutrition policy and assess diet-disease associations".⁴ Their second is the one you've already read: do not use self-reported energy as true intake.

The defenders' strongest card is the same fact seen from the other side: the error has been measured, by them. Subar is first author of OPEN, and Subar and Willett are among the pooled study's authors. The fixes the reply proposes (energy adjustment, analyses that allow for measurement error, both short-term and long-term instruments, better biomarkers) follow from what those studies found.⁴⁵⁶

Where they actually part

Line the two up and, on this course's reading, they agree on more than the rhetoric suggests.

The real dispute is the last row. For the defenders and for Ioannidis, whether an error this size, measured this well, can be corrected enough to trust what's left is an empirical question, and this course's research file names what would settle it: calibration against recovery biomarkers inside the big cohorts, with the calibrated and uncalibrated results published side by side; analyses that report every exposure a cohort measured rather than a selection; and long trials of diets where they're feasible.⁹ For Archer it is not an empirical question at all: he argues the reports aren't measurements of intake, so no calibration can rescue them. Long trials have been tried on the largest scale in the Women's Health Initiative.

The most expensive study

In 1993 the Women's Health Initiative began a trial that did what the critics ask for: instead of asking women what they ate and waiting, it assigned them a diet. Satija and colleagues call the WHI "the most expensive human study ever conducted".³

The breast cancer paper states the premise in its opening line: "The hypothesis that a low-fat dietary pattern can reduce breast cancer risk has existed for decades but has never been tested in a controlled intervention trial."¹¹

The design, from the 2006 abstracts, which this course read:¹¹¹²

• Who. 48,835 postmenopausal women aged 50 to 79, at 40 US clinical centres, without prior breast cancer. A later paper adds that they had to be eating at least 32 percent of their energy as fat to enrol.¹³
• What. 40 percent (19,541) were assigned to "Intensive behavior modification in group and individual sessions designed to reduce total fat intake to 20% of calories and increase intakes of vegetables/fruits to 5 servings/d and grains to at least 6 servings/d."¹² The other 60 percent (29,294) "were not asked to make dietary changes."¹¹ The heart disease paper adds that "The comparison group received diet-related education materials."¹²
• How long. An average of 8.1 years of follow-up in the first reports.¹¹

A hazard ratio, which Mental Fitness lesson 6 read, compares rates over time, and 1 means no difference. The heart disease figures are from the companion paper.

Read the breast cancer interval with Habits and Self-Discipline lesson 5 in mind, which taught what an interval spanning no effect does and doesn't mean. The interval runs from a 17 percent lower risk to a 1 percent higher one. It includes no effect, and most of it lies on the side of benefit. The authors said as much in their own conclusion: "However, the nonsignificant trends observed suggesting reduced risk associated with a low-fat dietary pattern indicate that longer, planned, nonintervention follow-up may yield a more definitive comparison."¹¹

What the women actually ate

The target was 20 percent of energy from fat. Satija and colleagues report that "most of the participants randomly assigned to the low-fat group were unable to achieve their fat reduction target of 20%".³ The breast cancer abstract gives the size of the gap between the groups: the difference in the change in percentage of energy from fat "varied from 10.7% at year 1 to 8.1% at year 6."¹¹

One thing the abstracts don't say is how fat intake was measured, and this course did not read the trial's methods. If it was by self-report, everything in the first half of this lesson applies to the adherence figures too. What does not depend on anyone's memory is the blood, and it points both ways. LDL cholesterol, the blood measure lesson 6 reads, fell by 3.55 mg/dL (milligrams per decilitre) in the diet group compared with the comparison group, a small change but a measured one, while HDL cholesterol, triglycerides, glucose and insulin "did not significantly differ".¹² Satija and colleagues read the unchanged HDL and triglycerides as a sign the diet wasn't eaten as designed, since both "are known to change on low-fat diets".³

The heart disease abstract also says what came off the plate. By year 6, total fat was 8.2 percent of energy lower in the diet group, "with small decreases in saturated (2.9%), monounsaturated (3.3%), and polyunsaturated (1.5%) fat", and vegetables and fruit rose by 1.1 servings a day and grains by 0.5.¹²

So the trial tested eat less fat of every kind, and more plant foods. It didn't test replace saturated fat with unsaturated fat, which is the question lesson 6 takes up. The trial supports a narrower claim than it's sometimes used for: asking women to cut total fat this way didn't reduce heart disease over eight years. The research file's reading, which this course shares, is that it can't settle whether fat quality matters.⁹

Nineteen years later

The trial's investigators kept counting. Chlebowski and colleagues reported in 2020, after a median of 19.6 years of cumulative follow-up.¹³ This course read the abstract.

Deaths after breast cancer, which this course reads as death from any cause in a woman who had developed it, were lower in the diet group: 359 against 652, hazard ratio 0.85 (0.74 to 0.96). And deaths from breast cancer itself reached statistical significance: 132 (0.037 percent a year) against 251 (0.047 percent a year), hazard ratio 0.79 (0.64 to 0.97).¹³

A 21 percent relative reduction. Now the absolute size, which Logic and Argument lesson 9 taught you to ask for: 0.047 minus 0.037 is 0.010 percent a year, about one fewer death from breast cancer per 10,000 women each year. That's the course's arithmetic from the abstract's annualised rates.

The authors' conclusion: "Adoption of a low-fat dietary pattern associated with increased vegetable, fruit, and grain intake, demonstrably achievable by many, may reduce the risk of death as a result of breast cancer in postmenopausal women."¹³ Notice their verb, "may".

Three readings of one trial

Serious researchers read the WHI in three ways.⁹

A large null trial. A trial of nearly 49,000 women ran for eight years and found no significant effect on its main outcomes. The heart disease paper's own conclusion is that the diet "did not significantly reduce the risk of CHD, stroke, or CVD" (coronary heart disease, stroke, or cardiovascular disease as a whole).¹² On this reading the dilution is part of the answer: this is what happens when the advice is given, which is what a guideline has to know. Ioannidis makes a related point about large diet trials: even when their results are negative, they can inform guidelines with what he calls "intention-to-eat" data.²

A trial that never tested its hypothesis. Satija and colleagues: "The WHI, hence, failed to test its original hypothesis, and its null findings were largely uninformative with respect to the causal effect of a low-fat dietary intervention."³ On this reading the women did not eat the diet the hypothesis was about, so the null says little about that diet.

A slow, real effect. The 19.6-year follow-up found fewer deaths from breast cancer. On this reading, a diet change in late middle age took more than a decade to show in deaths from breast cancer itself, which is lesson 1's point about latency.¹³

Each reading has a weakness its opponents would press. In this course's words: the first has to set aside that the contrast was diluted and that the breast cancer interval leaned towards benefit. The second has an awkward implication for the defenders: if a trial this large, with intensive support, can't get people to the target, that says something about the advice as well as the trial. And the third rests on an outcome that emerged well after the intervention ended, among many outcomes this long-running trial has reported, so it is best held as suggestive rather than settling. The 2006 authors had asked for longer follow-up, though of breast cancer incidence, the trial's outcome, rather than of deaths.¹¹

What people get wrong

"Calorie counts from food diaries are accurate if you're careful." The biomarker studies in this lesson tested questionnaires and recalls, and every one under-reported energy on average: 28 percent on the questionnaire and 15 percent on a single recall in the pooled data.⁵ This course has no biomarker figure for diaries kept as you eat, so it cannot give you theirs. But the characteristics that predicted under-reporting were body mass index, education and age,⁵ which are not things care changes, so being careful is not evidence that the problem has gone away.

"The critics say all diet research is fake." Archer's case is close to that. Ioannidis's isn't: he asks for reform, open data and trials. And the defenders concede the central point on energy in their own recommendations. Most of the argument is about what can be rescued, not about whether the error exists.

"The defenders think self-reported calories are fine." The mirror image, and just as wrong. Subar and colleagues' second recommendation is not to use self-reported energy as true intake, and the 0.21 comes from the defenders' own pooled study.⁴⁵

"A null trial proves no effect." A null result says the trial couldn't tell the effect from zero, as the diet was actually eaten, over the time it ran. It does not say the diet as designed does nothing. Nineteen years on, the WHI reported fewer deaths from breast cancer, a result that is suggestive rather than settling.¹¹¹³

Practice

Connections

Back. Lesson 1 set out the designs; this lesson looked at the instrument under the cohort, and lesson 1's "instead of what?" decided what the WHI tested.

Forward. Doubly labelled water carried this lesson because it doesn't ask. Lesson 3 uses it and the metabolic ward to answer questions about energy balance that self-report can't reach. Lesson 6 returns to the WHI's untested question: what happens when saturated fat is replaced with something specific.

Go deeper

• Freedman and colleagues, 2014, free at PubMed Central. This course read the abstract. The pooled biomarker study, and the source of most of the numbers in this lesson's chart.
• Subar and colleagues, 2015, free at PubMed Central. This course read the abstract, which lists all seven recommendations. The defence at its most concrete, conceding what it concedes.
• Archer, Pavela and Lavie, 2015, free at PubMed Central. This course read the abstract. The critique at full strength, in four numbered arguments.
• Chlebowski and colleagues, 2020, free at PubMed Central. This course read the abstract. The WHI's long follow-up, with the figures for both breast cancer outcomes.

Sources

1. E. Archer, G. A. Hand and S. N. Blair, "Validity of U.S. nutritional surveillance: National Health and Nutrition Examination Survey caloric energy intake data, 1971-2010", PLoS One 8(10), 2013, e76632, doi 10.1371/journal.pone.0076632. Read: the abstract. The caution that expenditure was estimated from equations is the research file's.
2. J. P. A. Ioannidis, "The Challenge of Reforming Nutritional Epidemiologic Research", JAMA 320(10), 2018, pp. 969 to 970, doi 10.1001/jama.2018.11025. Read: the full text (2 pages), from a reprint of the publisher's PDF. His proposals are given in the research file's summary.
3. A. Satija, E. Yu, W. C. Willett and F. B. Hu, "Understanding nutritional epidemiology and its role in policy", Advances in Nutrition 6(1), 2015, pp. 5 to 18, doi 10.3945/an.114.007492. Read: the abstract and, in full, the sections "Introduction", "Can We Reliably Measure Dietary Intakes...", "What Is the Role of Nutritional Epidemiology in Inferring Causality?" and "Is the Drug Trial Paradigm Relevant..."; the rest skimmed. The defenders' argument about ranking and energy-adjusted nutrients is the research file's summary of this paper and of [4] and [5], not a quotation.
4. A. F. Subar and colleagues, "Addressing Current Criticism Regarding the Value of Self-Report Dietary Data", Journal of Nutrition 145(12), 2015, pp. 2639 to 2645, doi 10.3945/jn.115.219634. Read: the abstract. Supports the grouping of instruments and recommendations 1, 2, 3, 5 and 6.
5. L. S. Freedman and colleagues, "Pooled results from 5 validation studies of dietary self-report instruments using recovery biomarkers for energy and protein intake", American Journal of Epidemiology 180(2), 2014, pp. 172 to 188, doi 10.1093/aje/kwu116. Read: the abstract. Willett's and Subar's co-authorship is from the author list.
6. A. F. Subar and colleagues, "Using intake biomarkers to evaluate the extent of dietary misreporting in a large sample of adults: the OPEN study", American Journal of Epidemiology 158(1), 2003, pp. 1 to 13, doi 10.1093/aje/kwg092. Read: the abstract. That doubly labelled water gives intake in weight-stable people is the research file's explanation.
7. E. Archer, G. Pavela and C. J. Lavie, "The Inadmissibility of What We Eat in America and NHANES Dietary Data in Nutrition and Obesity Research and the Scientific Formulation of National Dietary Guidelines", Mayo Clinic Proceedings 90(7), 2015, pp. 911 to 926, doi 10.1016/j.mayocp.2015.04.009. Read: the abstract. Supports the four numbered arguments and the term M-BM.
8. E. Archer, C. J. Lavie and J. O. Hill, "The Failure to Measure Dietary Intake Engendered a Fictional Discourse on Diet-Disease Relations", Frontiers in Nutrition 5, 2018, 105, doi 10.3389/fnut.2018.00105. Read: the abstract. The source italicises "memories of perceptions of dietary intake".
9. This course's research file, Part A sections 2.4 and 5.2 to 5.4: what would settle the dispute, the reading that the WHI tested total fat, and the three readings of the trial with the caution on its long-term outcome. Its own synthesis, not a published source.
10. Y. Li and colleagues, "Saturated Fats Compared With Unsaturated Fats and Sources of Carbohydrates in Relation to Risk of Coronary Heart Disease: A Prospective Cohort Study", Journal of the American College of Cardiology 66(14), 2015, pp. 1538 to 1548, doi 10.1016/j.jacc.2015.07.055. Read: the abstract. Supports the questionnaire every four years.
11. R. L. Prentice and colleagues, "Low-fat dietary pattern and risk of invasive breast cancer: the Women's Health Initiative Randomized Controlled Dietary Modification Trial", JAMA 295(6), 2006, pp. 629 to 642, doi 10.1001/jama.295.6.629. Read: the abstract.
12. B. V. Howard and colleagues, "Low-fat dietary pattern and risk of cardiovascular disease: the Women's Health Initiative Randomized Controlled Dietary Modification Trial", JAMA 295(6), 2006, pp. 655 to 666, doi 10.1001/jama.295.6.655. Read: the abstract.
13. R. T. Chlebowski and colleagues, "Dietary Modification and Breast Cancer Mortality: Long-Term Follow-Up of the Women's Health Initiative Randomized Trial", Journal of Clinical Oncology 38(13), 2020, pp. 1419 to 1428, doi 10.1200/jco.19.00435. Read: the abstract. Supports the 32 percent eligibility. The absolute difference per 10,000 women is this course's arithmetic.