0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Academia and the Profession |

Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy

Steven N. Goodman, MD, PhD
[+] Article and Author Information

From Johns Hopkins University School of Medicine, Baltimore, Maryland.


Ann Intern Med. 1999;130(12):995-1004. doi:10.7326/0003-4819-130-12-199906150-00008
Text Size: A A A

An important problem exists in the interpretation of modern medical research data: Biological understanding and previous research play little formal role in the interpretation of quantitative results. This phenomenon is manifest in the discussion sections of research articles and ultimately can affect the reliability of conclusions. The standard statistical approach has created this situation by promoting the illusion that conclusions can be produced with certain “error rates,” without consideration of information from outside the experiment. This statistical approach, the key components of which are P values and hypothesis tests, is widely perceived as a mathematically coherent approach to inference. There is little appreciation in the medical community that the methodology is an amalgam of incompatible elements, whose utility for scientific inference has been the subject of intense debate among statisticians for almost 70 years. This article introduces some of the key elements of that debate and traces the appeal and adverse impact of this methodology to the P value fallacy, the mistaken idea that a single number can capture both the long-run outcomes of an experiment and the evidential meaning of a single result. This argument is made as a prelude to the suggestion that another measure of evidence should be used—the Bayes factor, which properly separates issues of long-run behavior from evidential strength and allows the integration of background knowledge with statistical findings.

Figures

Grahic Jump Location
Figure 1.
The parallels between the processes of induction and deduction in medical inference (top) and statistical inference (bottom).

Δ = treatment difference.

Grahic Jump Location
Grahic Jump Location
Figure 3.
The bell-shaped curve represents the probability of every possible outcome under the null hypothesis.PP

Both α (the type I error rate) and the value are “tail areas” under this curve. The tail area for α is set before the experiment, and a result can fall anywhere within it. The value tail area is known only after a result is observed, and, by definition, the result will always lie on the border of that area.

Grahic Jump Location
Grahic Jump Location
Figure 4.
Possible outcomes of two hypothetical trials in six patients (Appendix)

. The only possible overlapping results are the observed data and the result in which treatment A was preferred in all patients.

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Journal Club
Topic Collections
PubMed Articles

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)