0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Academia and the Profession |

Incorporating Quality of Evidence into Decision Analytic Modeling

R. Scott Braithwaite, MD, MSc; Mark S. Roberts, MD, MPP; and Amy C. Justice, MD, PhD
[+] Article and Author Information

From Yale University School of Medicine and Connecticut Veterans Affairs Healthcare System, New Haven, Connecticut, and University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.


Grant Support: By grant K23 AA14483-01 from the National Institute of Alcohol Abuse and Alcoholism.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: R. Scott Braithwaite, MD, MSc, Yale University School of Medicine, Veterans Affairs Connecticut Healthcare System, Mailstop 11 ACSLG, 950 Campbell Avenue, West Haven, CT 06516; e-mail, ronald.braithwaite@va.gov.

Current Author Addresses: Drs. Braithwaite and Justice: Yale University School of Medicine, Veterans Affairs Connecticut Healthcare System, Mailtop 11 ACSLG, 950 Campbell Avenue, West Haven, CT 06516

Dr. Roberts: University of Pittsburgh School of Medicine, 200 Meyran Avenue, 2nd Floor, Pittsburgh, PA 15213.


Ann Intern Med. 2007;146(2):133-141. doi:10.7326/0003-4819-146-2-200701160-00008
Text Size: A A A

Our objective was to illustrate the effects of using stricter standards for the quality of evidence used in decision analytic modeling. We created a simple 10-parameter probabilistic Markov model to estimate the cost-effectiveness of directly observed therapy (DOT) for individuals with newly diagnosed HIV infection. We evaluated quality of evidence on the basis of U.S. Preventive Services Task Force methods, which specified 3 separate domains: study design, internal validity, and external validity. We varied the evidence criteria for each of these domains individually and collectively. We used published research as a source of data only if the quality of the research met specified criteria; otherwise, we specified the parameter by randomly choosing a number from a range within which every number has the same probability of being selected (a uniform distribution). When we did not eliminate poor-quality evidence, DOT improved health 99% of the time and cost less than $100 000 per additional quality-adjusted life-year (QALY) 85% of the time. The confidence ellipse was extremely narrow, suggesting high precision. When we used the most rigorous standards of evidence, we could use fewer than one fifth of the data sources, and DOT improved health only 49% of the time and cost less than $100 000 per additional QALY only 4% of the time. The confidence ellipse became much larger, showing that the results were less precise. We conclude that the results of decision modeling may vary dramatically depending on the stringency of the criteria for selecting evidence to use in the model.

Figures

Grahic Jump Location
Figure 1.
Schematic diagram of decision analytic model.

This model was constructed with extreme parsimony solely to illustrate the approach. Squares signify choices (for example, use the adherence intervention [ADH INT] versus do not use the adherence intervention [NO ADH INT]), circles signify potential consequences (for example, live versus die), and triangles signify consequences that may endure over a significant time frame and therefore may affect quality or quantity of life (for example, treat HIV). The circle inscribed with “M” is a Markov node (MarkovAdh), denoting that the model has the capacity to represent changes in clinical status over time. HIVtreat = receives and adheres to HIV treatment; HIVnotreat = does not receive or does not adhere to HIV treatment.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Cost-effectiveness plane and 95% confidence ellipses for directly observed therapy in the absence of any evidence criteria (A) and with quality of evidence criteria applied to research design (B), internal validity (C), and external validity (D).

Each point signifies the incremental cost-effectiveness of a particular model run. Points that cluster within a narrow 95% confidence ellipse (the analogue of a 95% CI) suggest great precision, whereas points that scatter throughout a wide 95% confidence ellipse suggest low precision. The location of each point on the cost-effectiveness plane indicates its cost-effectiveness. As a general guide, when points lie to the left of a decision making threshold ($100 000 per quality-adjusted life-year [QALY] is a commonly used threshold), incremental costs are high relative to incremental benefits, and cost-effectiveness is unfavorable. In contrast, when points lie to the right of that threshold, incremental costs are low relative to incremental benefits, and cost-effectiveness is favorable. Our results became notably less precise when stricter evidence criteria were applied, most dramatically with an external validity (EV) criterion. IV = internal validity.

Grahic Jump Location
Grahic Jump Location
Figure 3.
Acceptability curves for directly observed therapy (DOT) in the absence of any evidence criteria (A) and with strength of evidence criteria applied to research design (B), internal validity (C), and external validity (D).

The horizontal axis shows a range of values that society may be willing to pay for health benefits, and the curve's elevation (on the vertical axis) denotes the probability that DOT has an incremental cost-effectiveness that is more favorable than the corresponding willingness to pay. DOT became notably less cost-effective when stricter evidence criteria were applied, most dramatically with an external validity (EV) criterion. IV = internal validity.

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Topic Collections
PubMed Articles

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)