The full content of Annals is available to subscribers

Subscribe/Learn More  >
Academia and the Profession |

Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions

Brendan M. Reilly, MD; and Arthur T. Evans, MD, MPH
[+] Article, Author, and Disclosure Information

From Cook County (Stroger) Hospital and Rush Medical College, Chicago, Illinois.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Brendan M. Reilly, MD, Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612; e-mail, breilly@cchil.org.

Current Author Addresses: Dr. Reilly: Cook County (Stroger) Hospital, Administration Building, Room 1528, 1900 West Polk Street, Chicago, IL 60612.

Dr. Evans: Cook County (Stroger) Hospital, Administration Building, Room 1600, 1900 West Polk Street, Chicago, IL 60612.

Ann Intern Med. 2006;144(3):201-209. doi:10.7326/0003-4819-144-3-200602070-00009
Text Size: A A A

Clinical prediction rules, sometimes called clinical decision rules, have proliferated in recent years. However, very few have undergone formal impact analysis, the standard of evidence to assess their impact on patient care. Without impact analysis, clinicians cannot know whether using a prediction rule will be beneficial or harmful. This paper reviews standards of evidence for developing and evaluating prediction rules; important differences between prediction rules and decision rules; how to assess the potential clinical impact of a prediction rule before translating it into a decision rule; methodologic issues critical to successful impact analysis, including defining outcome measures and estimating sample size; the importance of close collaboration between clinical investigators and practicing clinicians before, during, and after impact analysis; and the need to measure both efficacy and effectiveness when analyzing a decision rule's clinical impact. These considerations should inform future development, evaluation, and use of all clinical prediction or decision rules.


Grahic Jump Location
Figure 1.
Goldman and colleagues' clinical prediction rule for major cardiac complications for patients with chest pain.4

BP = blood pressure; ECG = electrocardiography; MI = myocardial infarction. Adapted from reference .

Grahic Jump Location
Grahic Jump Location
Figure 2.
Ottawa Ankle Rule.5

Adapted from reference .

Grahic Jump Location




CME Activities are only available to ACP members and Individual Annals subscribers. If you are a member or a subscriber please sign in. Otherwise please become a member or subscribe to Annals.
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Clinical sensibility and barriers to KT should be considered prior to impact analysis
Posted on February 23, 2006
Jamie C. Brehaut
Ottawa Health Research Institute, Ottawa Hospital, University of Ottawa
Conflict of Interest: None Declared

Reilly and Evans (1) have provided a very useful discussion of how to study the impact of clinical prediction/decision rules in actual practice. The majority of such rules have not received such evaluations, and this work will hopefully encourage researchers to begin to fill in this gap. We have three specific comments about the article.

First, the authors mention in the introduction that "How frequently these and other prediction rules are being used in clinical practice is not known". While this is certainly true of the vast majority of rules, a fair bit is now known about the rules developed in Ottawa (2-5). Surveys of nationally representative samples of Canadian emergency physicians have shown that the Ottawa Ankle Rules are known to virtually all and reportedly used by almost 90% of Canadian emergency physicians (2;4), while international surveys shows that this figure is dramatically different for other countries. For example, in the U.S. the vast majority were aware of this rule, but only 31% of emergency physicians reported using it(3). This kind of data has been collected on the Ottawa Knee Rules (2), the Canadian C-Spine Rule (5), and is currently being collected for the Canadian CT-Head Rule. Note that while self-reports of rule use tells us whether the physicians feel they make use of the rule, they cannot tell us how the rule is used; i.e. whether physicians are using the rule correctly, using it in all the circumstances for which it would be appropriate to do so, etc. To address these issues,other techniques must be employed (4).

Second, Reilly and Evans stress that clinical sensibility of the rule needs to be considered; any rule that does not make clinical sense will be unlikely to be widely used. Yet it is important to remember that clinical sensibility is to some extent idiosyncratic. Interviewing physicians about any specific rule is almost certain to elicit a wide variety of indicators for which it would make clinical sense, at least for some, to include in the rule. If there is general consensus that inclusion of a non-rule indicator would improve the safety or efficiency of the rule, then we agree that evaluating whether inclusion of the indicator into a modified rule may be warranted, even at the relatively late stage of the impact analysis. However, if this new indicator results in a more effective rule, this new rule must then be re-validated on a sample larger than is typical for most impact analyses. Whenever possible, consultation with physicians on all of the potentially useful indicators should be done prior to the derivation of the rule, and not left until the impact analysis.

Even when all appropriate factors have been considered during rule derivation, some physicians during the later impact analysis will inevitably feel that the rule could be made safer, more efficient, or more clinically sensible if some of the other indicators were included. Indeed, we have shown that factors known to add no more predictive value over the rule are still considered by a large proportion of physicians purporting to use a rule (4). We have argued that making it clear what factors were included in the derivation of the rule, but were excluded from the rule because they do not add anything, might help improve the perceived clinical sensibility of a rule, and perhaps the level of appropriate use.

Finally, Reilly and Evans correctly argue that addressing typical barriers to knowledge translation is a critical component to assessing impact of a rule. Again, we argue that such issues should be considered well before the stage of impact analysis. For example, as the authors suggest, physicians are often more interested in sensitivity than specifity. Indeed, surveys suggest that many physicians would only consider using a rule that had extremely high sensitivity (2). It is for this reason that the Ottawa rules have typically adopted extremely high sensitivity thresholds. While it is clear that reducing the threshold sensitivity would improve the specificity of the rule, our findings suggest that that increase in specificity would come at the cost of fewer physicians using the rule. In order to increase the chances of producing a rule that becomes widely used, such barriers should be considered well before the impact analysis stage, and ideally very early in the development of the rule.

Reference List

1. Reilly B,.Evans JS. Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine 2006;144:201-9.

2. Graham ID, Stiell IG, Laupacis A, O'Connor AM, Wells GA. Emergency physicians' attitudes toward and use of clinical decision rules for radiography. Academic Emergency Medicine 1998;5:134-40.

3. Graham ID, Stiell IG, Laupacis A, McAuley L, Howell M, Clancy M et al. Awareness and use of the Ottawa Ankle and Knee Rules in 5 countries: Can publication alone be enough to change practice? Annals of Emergency Medicine 2001;37:259-66.

4. Brehaut JC, Stiell I, Graham I, and Visentin L. Clinical decision rules 'in real world': How a widely disseminated rule is used in everyday practice. Academic Emergency Medicine in press. 2005.

5. Brehaut, JC, Stiell, I., and Graham, I. Will a new clinical decision rule be widely used? The case of the Canadian C-Spine Rule (In Press). Acad Emerg Med . 2005.

Conflict of Interest:

None declared

Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.