Download citation file:
From Cook County (Stroger) Hospital and Rush Medical College, Chicago, Illinois.
Potential Financial Conflicts of Interest: None disclosed.
Requests for Single Reprints: Brendan M. Reilly, MD, Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612; e-mail, firstname.lastname@example.org.
Current Author Addresses: Dr. Reilly: Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612.
Dr. Evans: Cook County (Stroger) Hospital, Administration Building, Room 1600, 1900 West Polk Street, Chicago, IL 60612.
Clinical prediction rules, sometimes called clinical decision rules, have proliferated in recent years. However, very few have undergone formal impact analysis, the standard of evidence to assess their impact on patient care. Without impact analysis, clinicians cannot know whether using a prediction rule will be beneficial or harmful. This paper reviews standards of evidence for developing and evaluating prediction rules; important differences between prediction rules and decision rules; how to assess the potential clinical impact of a prediction rule before translating it into a decision rule; methodologic issues critical to successful impact analysis, including defining outcome measures and estimating sample size; the importance of close collaboration between clinical investigators and practicing clinicians before, during, and after impact analysis; and the need to measure both efficacy and effectiveness when analyzing a decision rule's clinical impact. These considerations should inform future development, evaluation, and use of all clinical prediction or decision rules.
BP = blood pressure; ECG = electrocardiography; MI = myocardial infarction. Adapted from reference .
Adapted from reference .
Please read the other comments before posting. Contributors must reveal any conflict
Comments are moderated and will appear on the site at the discretion of The American
College of Physicians editorial staff. Please be sure your email address is
updated in your account, otherwise the American College of Physicians will not be
able to contact you about your comment.
* = Required Field
Disclosure of Any Conflicts of Interest*
(applies to the past 5 years and foreseeable future) Indicate any potential conflicts
of interest of each author below, including specific financial interests and relationships
and affiliations relevant to the subject matter or materials discussed in the manuscript
(eg, employment/affiliation, grants or funding, consultancies, honoraria, speakers
bureau, stock ownership or options, expert testimony, royalties, donation of medical
equipment, or patents filed, received, or pending). If all authors have none, check
"No potential conflicts or relevant financial interests" in the box below. Please
also indicate any funding received in support of this work. The information will
be posted with your response.
Reilly and Evans (1) have provided a very useful discussion of how to study the impact of clinical prediction/decision rules in actual practice. The majority of such rules have not received such evaluations, and this work will hopefully encourage researchers to begin to fill in this gap. We have three specific comments about the article.
First, the authors mention in the introduction that "How frequently these and other prediction rules are being used in clinical practice is not known". While this is certainly true of the vast majority of rules, a fair bit is now known about the rules developed in Ottawa (2-5). Surveys of nationally representative samples of Canadian emergency physicians have shown that the Ottawa Ankle Rules are known to virtually all and reportedly used by almost 90% of Canadian emergency physicians (2;4), while international surveys shows that this figure is dramatically different for other countries. For example, in the U.S. the vast majority were aware of this rule, but only 31% of emergency physicians reported using it(3). This kind of data has been collected on the Ottawa Knee Rules (2), the Canadian C-Spine Rule (5), and is currently being collected for the Canadian CT-Head Rule. Note that while self-reports of rule use tells us whether the physicians feel they make use of the rule, they cannot tell us how the rule is used; i.e. whether physicians are using the rule correctly, using it in all the circumstances for which it would be appropriate to do so, etc. To address these issues,other techniques must be employed (4).
Second, Reilly and Evans stress that clinical sensibility of the rule needs to be considered; any rule that does not make clinical sense will be unlikely to be widely used. Yet it is important to remember that clinical sensibility is to some extent idiosyncratic. Interviewing physicians about any specific rule is almost certain to elicit a wide variety of indicators for which it would make clinical sense, at least for some, to include in the rule. If there is general consensus that inclusion of a non-rule indicator would improve the safety or efficiency of the rule, then we agree that evaluating whether inclusion of the indicator into a modified rule may be warranted, even at the relatively late stage of the impact analysis. However, if this new indicator results in a more effective rule, this new rule must then be re-validated on a sample larger than is typical for most impact analyses. Whenever possible, consultation with physicians on all of the potentially useful indicators should be done prior to the derivation of the rule, and not left until the impact analysis.
Even when all appropriate factors have been considered during rule derivation, some physicians during the later impact analysis will inevitably feel that the rule could be made safer, more efficient, or more clinically sensible if some of the other indicators were included. Indeed, we have shown that factors known to add no more predictive value over the rule are still considered by a large proportion of physicians purporting to use a rule (4). We have argued that making it clear what factors were included in the derivation of the rule, but were excluded from the rule because they do not add anything, might help improve the perceived clinical sensibility of a rule, and perhaps the level of appropriate use.
Finally, Reilly and Evans correctly argue that addressing typical barriers to knowledge translation is a critical component to assessing impact of a rule. Again, we argue that such issues should be considered well before the stage of impact analysis. For example, as the authors suggest, physicians are often more interested in sensitivity than specifity. Indeed, surveys suggest that many physicians would only consider using a rule that had extremely high sensitivity (2). It is for this reason that the Ottawa rules have typically adopted extremely high sensitivity thresholds. While it is clear that reducing the threshold sensitivity would improve the specificity of the rule, our findings suggest that that increase in specificity would come at the cost of fewer physicians using the rule. In order to increase the chances of producing a rule that becomes widely used, such barriers should be considered well before the impact analysis stage, and ideally very early in the development of the rule.
1. Reilly B,.Evans JS. Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine 2006;144:201-9.
2. Graham ID, Stiell IG, Laupacis A, O'Connor AM, Wells GA. Emergency physicians' attitudes toward and use of clinical decision rules for radiography. Academic Emergency Medicine 1998;5:134-40.
3. Graham ID, Stiell IG, Laupacis A, McAuley L, Howell M, Clancy M et al. Awareness and use of the Ottawa Ankle and Knee Rules in 5 countries: Can publication alone be enough to change practice? Annals of Emergency Medicine 2001;37:259-66.
4. Brehaut JC, Stiell I, Graham I, and Visentin L. Clinical decision rules 'in real world': How a widely disseminated rule is used in everyday practice. Academic Emergency Medicine in press. 2005.
5. Brehaut, JC, Stiell, I., and Graham, I. Will a new clinical decision rule be widely used? The case of the Canadian C-Spine Rule (In Press). Acad Emerg Med . 2005.
The In the Clinic® slide sets are owned and copyrighted by the American College
of Physicians (ACP). All text, graphics, trademarks, and other intellectual property
incorporated into the slide sets remain the sole and exclusive property of the ACP.
The slide sets may be used only by the person who downloads or purchases them and
only for the purpose of presenting them during not-for-profit educational activities.
Users may incorporate the entire slide set or selected individual slides into their
own teaching presentations but may not alter the content of the slides in any way
or remove the ACP copyright notice. Users may make print copies for use as hand-outs
for the audience the user is personally addressing but may not otherwise reproduce
or distribute the slides by any means or media, including but not limited to sending
them as e-mail attachments, posting them on Internet or Intranet sites, publishing
them in meeting proceedings, or making them available for sale or distribution in
any unauthorized form, without the express written permission of the ACP. Unauthorized
use of the In the Clinic slide sets will constitute copyright infringement.
to gain full access to the content and tools.
Learn more about subscription options