The Effectiveness of a Primer to Help People Understand Risk: Two Randomized Trials in Distinct Populations FREE

—The Editors

People face a bewildering array of medical decisions (for example, should I be tested for BRCA1? Should I be screened for prostate cancer? Should I have lumpectomy and radiation to treat my breast cancer?). If people are to make informed decisions, they need to understand risk: What is my chance of staying healthy if I undergo this intervention? What is my chance of staying healthy if I forgo it? What is the chance of harm?

Understanding risk data entails a set of skills. First, people must be able to work with probabilities and changes in probabilities, the typical measures used to communicate risk information. Next, they need to have a framework on which to organize data. In essence, this means having a sense of what additional information is needed to give meaning to a statement about probability (conversely, how to know when such additional information is lacking). For example, putting a particular risk in context (“your chance of breast cancer is 1 in 8”) entails knowing what outcome is being considered (diagnosis vs. death), being clear about the time period (5 years vs. lifetime), and having some perspective on the magnitude of the probability (How does the risk for a particular condition compare with other important health risks?). Third, people need to be able to make some basic assessment about the quality of the evidence to know whether they can believe the numbers that they are given.

Anecdotal experience and a growing body of literature document the trouble many people have in making sense of risk information (1 - 4), which is now ubiquitous in health messages. This is not surprising: Much more effort goes into disseminating health information (for example, drug advertisements, media reports, and decision-making aids) than into preparing the target audiences to understand it. To address this gap, we developed a primer to help people develop the basic skills needed to make sense of the medical risk data that surround them. In this paper, we report on 2 randomized trials that tested the primer. We sought to learn how people would respond to the primer: Would they read it? Would they value this knowledge? Most important, would the primer improve patients' abilities to interpret medical data?

To test the effect of the primer on how well people understand risk, we conducted 2 randomized trials in distinct populations: people with high and low socioeconomic status (SES). Figure 1 shows an overview of the study design. The Committee for the Protection of Human Subjects at Dartmouth College approved this project, and the survey cover letter was considered to be informed consent (signed consent was waived).

We calculated our sample size under the most conservative conditions—when the pooled proportion of passing scores was 50%. We asserted that a 20-percentage point absolute difference in the proportion passing the test in the primer group versus the control group would be clinically important. In calculating the sample size requirements, we therefore assumed that the proportion passing would be 50% in the control group and 70% in the primer group. When a power of 0.8 and a 2-sided P value of 0.05 were used, 100 patients were required for each study group. Assuming that 10% of participants would be lost to follow-up, we planned to enroll 110 patients per group in each trial.

To study the effect of the primer in a highly educated and affluent group, we recruited alumni from Dartmouth's “Community Medical School.” This 9-lecture series on various health and medical topics is taught by Dartmouth faculty and guests; is held annually in Hanover, New Hampshire, and Manchester, New Hampshire; and has a $25 registration fee (5). For this study, the program organizers mailed recruitment letters to 1138 “alumni.” A total of 334 people who responded to the letter were eligible (that is, they met the age criterion of 35 to 79 years, spoke English, and had attended the Community Medical School in the past) and were subsequently randomly assigned. Ninety-six percent (n = 322) returned a completed survey (completion rates were not significantly different between the primer and control groups [95% vs. 98%, respectively]) (P = 0.192).

To study the effect of the primer in a sample with lower income and less formal education, we recruited veterans and their families at the White River Junction Veterans Affairs (VA) Medical Center, White River Junction, Vermont, by posting study advertisements in waiting areas of the outpatient clinic. A total of 221 people who responded to the advertisement were eligible (that is, they met the age criteria of 35 to 79 years, spoke English, and were a veteran or the family member of a veteran who was enrolled in a VA clinic) and were subsequently randomly assigned. Ninety percent (n = 200) returned completed surveys (completion rates were significantly lower in the primer group than in the control group [85% vs. 96%, respectively]) (P = 0.005).

The letter and the advertisement asked people to participate in a research study to learn how to better give people health information. We did not mention our interest in enhancing quantitative skills. People who responded to the recruitment letter (high SES group) or to the advertisement (low SES group) were first interviewed to confirm eligibility. We only accepted 1 participant per household. Within each trial, we assigned participants on an individual basis to receive either the primer or control booklet. A list of random numbers (created by using a random-number generator) was given to a research assistant and was used to determine assignments (randomization was not stratified or blocked within each trial). The research assistant had access to participants' characteristics. The investigators did not have access to the assignment list, and the deidentified code was only revealed after recruitment and completion of data collection.

The appropriate booklet and a survey were mailed or given in person to participants. The survey included the major outcomes that will be described in the next section. To make the survey seem relevant to the control group, it also included additional questions about facts presented in the health booklet (we did not analyze responses to these questions). All participants were asked to read the booklet and return the completed survey within 2 weeks using the self-addressed stamped envelope included with the survey. Reminder letters were sent to nonresponders. Participants who returned surveys were given their choice of a $25 gift certificate to a local bakery, restaurant, bookstore, or large retail store. We recruited participants from October 2004 through August 2005.

The participants in the primer group received the booklet entitled “Know Your Chances: Understanding Health Statistics.” The goal of the primer is to teach people how to understand risk messages and health statistics. The first part of the primer teaches people how to understand disease risk by using the example of colon cancer. The second part focuses on how to understand the benefits and harms of interventions by using a Zocor direct-to-consumer advertisement for secondary heart disease prevention. Figure 2 shows the summary of main concepts that were taught, which is excerpted from the booklet. The contents of the primer and many of the examples have been developed and revised over years of teaching and through focus groups with people across a diverse socioeconomic spectrum.

Because many people are intimidated by numbers and statistics, we worked hard to make the primer inviting and nonthreatening by liberal use of cartoons and figures; by working through examples, separating the most technical material into optional “learn more boxes”; and by providing readers with quizzes (with answers) to assess their mastery of the material as they read the primer. Figure 3 shows sample pages from the primer. Most of the primer is written at the eighth-grade or lower reading level (6) and is a color document approximately 80 pages in length.

The control group received a 70-page booklet entitled “The Pocket Guide for Good Health for Adults,” which is published by the U.S. Department of Health and Human Services Agency for Health Care Research and Quality (7). We chose this booklet because the length is similar to that of the primer, it is written at a similar reading level (eighth grade or lower) (6), and it contains general information about risk and reducing risk by following recommended prevention or screening activities. However, it does not include training on how to interpret quantitative information.

All outcome measures were assessed in the survey.

The primary outcome was a measure of participants' abilities to interpret medical statistics. To measure this outcome, we developed the 18-item data interpretation test during the same time as the primer and validated it in a separate study (8). The test is available at www.vaoutcomes.org/research_tools.php. It extends our original 3-item numerical measure (4) to include a much broader set of skills beyond simple mathematical manipulation: such skills are the ability to compare risks, to put risk estimates into context, and to know what additional information is necessary to give meaning to a medical statistic. The data interpretation test does not test recall of facts; instead, it asks people to interpret real-world information (for example, drug advertisements and news stories). The skills being tested are the same as those in the primer because we feel that they are key skills for people to understand. However, the medical data interpretation test requires that the respondent apply these skills to data (within various messages) that do not appear in the primer.

Scores from this test range from 0 to 100—higher scores represent higher abilities. We consider a score of 75 or higher as “passing” and a score of 90 or higher as “outstanding” (the latter threshold corresponds to the mean score of physician experts who teach “evidence-based” medicine in our validation study (8)).

Secondary outcomes included measures of interest and confidence in interpreting medical statistics. These outcomes were assessed twice: before randomization (at the eligibility interview) and again as part of the survey. These previously validated measures (STAT-interest and STAT-confidence scales) are also scored on 0 to 100 scales in which higher scores represent higher interest or confidence (9). In addition, we asked participants to rate each booklet on various dimensions (for example, Did you find it helpful? Did you learn something new?, and Would you recommend it to others?).

The main results were based on complete case analysis with additional sensitivity analyses to address the impact of missing data. We used the chi-square test to compare differences in proportions and t-tests to compare means. All comparisons were 2-sided and were considered statistically significant at P values less than 0.05. We used Stata, version 9.1 (Stata Corp., College Station, Texas) for all analyses.

Drs. Woloshin and Schwartz are supported by Robert Wood Johnson Generalist Faculty Scholars Awards and receive infrastructure support from a Research Enhancement Award from the Department of Veterans Affairs. The study was supported by a grant from the National Cancer Institute (R01CA104721). The funding sources did not play a role in the design, data collection, analysis, or interpretation of the study. The authors had full access to the data files for this study.

Table 1 summarizes the characteristics of the participants who completed the survey. As designed, the 2 trials had different participants. In the high SES trial, 70% had a household income of $50 000 or greater and 80% had a college or postgraduate degree. In the low SES trial, approximately 70% had a household income less than $50 000 and approximately 50% had a high school degree or less formal education. As expected, participants in the low SES trial were mostly men (recruitment was done at a VA hospital) and the participants in the high SES trial were mostly women (as has typically been the case at the Dartmouth Community Medical School). The participants in the low SES trial had lower self-reported health status, were more likely to smoke, and had more medical conditions than those in the high SES trial. Within each trial, however, there were no statistically significant differences between the primer and control groups regarding any characteristic assessed.

In both trials, the primer resulted in higher medical data interpretation scores than did the control booklet (Figure 4). In the high SES trial, the mean score was 81 in the primer group versus 75 in the control group—a 6-point difference (P = 0.0006). In terms of score thresholds, 74% of participants in the primer group received a “passing grade” (score ≥75) versus 56% of participants in the control group (P = 0.001). Twenty-six percent of participants in the primer group received an “outstanding grade” (score of ≥90), which is the mean score of physician experts who teach evidence-based medicine in a validation study of this scale versus 7% of participants in the control group (P < 0.001). In the low SES trial, the mean scores were 69 in the primer group versus 62 in the control group—a 7-point difference (P = 0.008). Passing grades were 44% versus 26% (P = 0.010) and outstanding grades were 10% versus 2% in the primer and control groups, respectively (P = 0.014). Finally, results were similar in additional analyses that adjusted for age, sex, level of education, and medical conditions: The difference in mean interpretation scores was 7 points (P < 0.001) in the high SES trial and 6 points (P = 0.019) in the low SES trial for the primer versus the control groups, respectively.

The specific skills for which participants in the primer group most outperformed those in the control group were as follows: recognizing that counts of events without denominators do not convey risks (77% vs. 59% in the high SES trial and 54% vs. 41% in the low SES trial answered correctly) and knowing that a risk statement applies to you requires knowing about the age and sex in the source data for the risk (81% vs. 55% in the high SES trial and 62% vs. 37% in the low SES trial answered correctly).

To assess the effect of nonresponse on our findings, we considered various possibilities of pass rates among the nonresponders. Assuming that nonresponders have lower data interpretation skills, the most plausible “worst-case” scenario is that the nonresponders would have failed the test. In this case, the statistical significance of the low SES trial becomes borderline (37% vs. 25%; P = 0.057) but remains significant in the high SES trial (70% vs. 54% ). In the least plausible and most extreme “worst-case” scenario (assuming that all of the nonresponders in the primer group would have failed the test but that all nonresponders in the control group would have passed the test), the difference in pass rates in the low SES trial becomes nonsignificant (37% vs. 29%; P = 0.20), but remains significant in the high SES trial (70% vs. 57%).

Interest in medical statistics was significantly higher (6 points) in the high SES trial (a 4-point increase vs. 2-point decrease from baseline; P = 0.004) and by 8 points in the low SES trial (a 6-point increase vs. a 2-point decrease from baseline; P = 0.004) after exposure to the primer compared with the control booklet. The primer, however, did not improve participants' confidence in interpreting medical statistics beyond the control booklet; if anything, confidence improved more with the control booklet: by 2 points in the high SES trial (a 2-point increase vs. a 4-point increase; P = 0.36) and by 4 points in the low SES trial (a 2-point increase vs. a 6-point increase; P = 0.166).

Finally, participants spent substantial time reading the primer and rated it highly (Table 2). In the high SES trial, 72% of the primer group reported spending an hour or more reading the primer, whereas only 28% of the control group spent that much time reading their booklet (P < 0.001). Similarly, in the low SES trial, 74% of the primer group spent an hour or more reading the primer, whereas 47% of the control group spent this much time reading their booklet (P = 0.001).

In the high SES trial, all ratings were significantly higher for the primer group than for the control group. For example, 91% found the primer “helpful” or “very helpful” versus 69% for the control booklet (P < 0.001). In the low SES trial, the primer ratings were high but were not statistically different from the ratings of the control booklet. For example, 94% and 90% of participants rated the primer and control booklet “helpful” or “very helpful,” respectively (P = 0.26).

No analyses, including those for the secondary outcomes and booklet ratings, include participants who did not complete the survey. Because completion rates were lower in the primer group versus the control group in the low SES trial (85% vs. 96%), the secondary outcomes and booklet ratings may favor the primer group to the extent that nonresponders had a less favorable response to the primer.

We developed and tested a general education primer designed to teach people basic medical data interpretation skills. The primer improved these skills in 2 distinct populations: participants with high SES and a demonstrated interest in health information (that is, they had attended a “community medical school” lecture series) and those with generally low formal education and limited quantitative skills. Participants liked the primer: A substantial majority of participants in both trials said they found it useful, it taught them something new, and that they would recommend it to others.

Our findings have several limitations. First is the question of generalizability. Our study samples were chosen deliberately to ensure the participation of people across a broad spectrum of age, income, and formal education. Because almost all participants were white and English speakers, we cannot be certain how the primer would perform in a more ethnically diverse setting. Also, because all participants were paid—those recruited from an actual health care setting and those recruited from the community—we cannot be certain how unpaid persons would respond to the primer. Second, there may be concern regarding the appropriateness of the control booklet. We chose an educational booklet (7) (published by the U.S. Department of Health and Human Services Agency for Healthcare Research and Quality) that was similar to the primer in length and tone. Because the control booklet did not teach readers about interpreting risk, it worked as a “placebo” with respect to the main outcome measure (that is, the tests results on medical data interpretation in the control group should reflect existing abilities). Third, in both trials, completion rates were lower in the primer group than in the control group, particularly in the low SES trial. This observation is not hard to explain: The primer covered more challenging material (it required readers to do math) than did the control book. However, completion rates were high overall (95% vs. 98% in the high SES trial and 85% vs. 97% in the low SES trial).

Curiously, although the primer improved ability and stimulated general interest in medical statistics, it did not increase participants' confidence in interpreting these statistics any more than did the control booklet. This finding probably reflects the educational impact of the primer. For some people (especially those with the least formal education), acquisition of new data interpretation skills may highlight what they did not know before and may expose overconfidence.

How big was the change in data interpretation skills? Because readers may be unfamiliar with our main outcome measure, they may have trouble gauging the clinical significance of our results. One way to put our findings into perspective is to consider the distribution of medical data interpretation skills observed in our previous validation study. In that study, the average difference in scores between experts (medical school faculty who teach evidence-based medicine) and other respondents with postgraduate degrees (similar educational attainment but no relevant expertise) was 20 points (8). The 6- and 7-point differences observed in the present study correspond to gaining approximately one third of the difference. Another way to put our findings into perspective is to consider the magnitude of the primer's effect in relation to other educational interventions. One way to make comparisons across interventions is to translate our findings into generic “effect size” units by using a metric called Cohen's d (primer minus control medical data interpretation scores divided by the pooled standard deviation of the scores) (10). Our results correspond to Cohen's d effect sizes of 0.38 (high SES) and 0.42 (low SES); by convention, these effect sizes are considered to be between “small” and “medium,” the range covering many successful educational interventions (11). For context, on average, taking a practice examination is associated with an approximate 0.30-effect size unit increase on standardized test scores (12).

Finally, it is important to acknowledge that although we have demonstrated that exposure to the primer improved medical data interpretation skills, we did not prove that the primer leads to better decision-making. Because informed decision-making is not possible without understanding the relevant facts, we think improving data interpretation skills is clearly a legitimate outcome in itself and a necessary prerequisite to making good decisions.

Implementing the primer, like decision aids, guidelines, or research innovations in general (13) may be challenging. We believe there are several ways that the primer could be used in clinical practice. It might be distributed either before or after routine clinic visits when patients are making decisions regarding screening and prevention. It might be even more important for patients facing such high-stakes decisions as surgery or chemotherapy. It is possible that patients facing a new and serious disease (for example, a new diagnosis of cancer) may have greater motivation to really understand the data regarding benefits and harms of various interventions and would find the primer very useful. Alternatively, they may feel too emotionally overwhelmed to use the primer. If this were the case, the primer might be useful for families or friends who are helping the patient with decision-making. Questions of when and where the primer might be most effectively used and how the reader's emotional state, decision-making style, and preferences might influence its impact are clearly a topic for further study.

In conclusion, the primer “Know Your Chances: Understanding Health Statistics”—a simple, inexpensive, low-tech intervention—improved medical data interpretation skills and was rated highly by well-educated participants as well as by those with less formal education.