The full content of Annals is available to subscribers

Subscribe/Learn More  >
Research and Reporting Methods |

Potential Bias of Instrumental Variable Analyses for Observational Comparative Effectiveness ResearchPotential Bias of Instrumental Variable Analyses for Observational CER

Laura Faden Garabedian, PhD; Paula Chu, MS; Sengwee Toh, ScD; Alan M. Zaslavsky, PhD; and Stephen B. Soumerai, ScD
[+] Article, Author, and Disclosure Information

From Harvard Medical School, Harvard Pilgrim Health Care Institute, and Harvard University, Boston, Massachusetts.

Note: This project was awarded first place at the 2012 Association for Public Policy Analysis and Management Fall Research Conference Poster Awards.

Disclaimer: The authors of this article are responsible for its content. Statements in the article should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Acknowledgment: The authors thank Sebastian Schneeweiss, MD, ScD, and John Seeger, PharmD, DrPH, both from the Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, for their insightful feedback on the study design and manuscript and Elizabeth Garry, MPH, from the Division of Pharmacoepidemiology and Pharmacoeconomics, for providing helpful administrative support throughout the project.

Financial Support: This work was supported as a subcontract from the Brigham and Women's Hospital DEcIDE Method Center under contract 290-05-0016I from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program. Dr. Soumerai received grant support from the Centers for Disease Control and Prevention's Natural Experiments for Translation in Diabetes.

Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-1887.

Requests for Single Reprints: Laura Faden Garabedian, PhD, Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, 133 Brookline Avenue, 6th Floor, Boston, MA 02215; e-mail, laura.garabedian@post.harvard.edu.

Current Author Addresses: Drs. Garabedian, Toh, and Soumerai: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, 133 Brookline Avenue, 6th Floor, Boston, MA 02215.

Ms. Chu: Harvard PhD Program in Health Policy, 14 Story, 4th Floor, Cambridge, MA 02138.

Dr. Zaslavsky: Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115.

Author Contributions: Conception and design: L.F. Garabedian, S. Toh, A.M. Zaslavsky, S.B. Soumerai.

Analysis and interpretation of the data: L.F. Garabedian, P. Chu, S. Toh, A.M. Zaslavsky, S.B. Soumerai.

Drafting of the article: L.F. Garabedian.

Critical revision of the article for important intellectual content: L.F. Garabedian, P. Chu, S. Toh, A.M. Zaslavsky, S.B. Soumerai.

Final approval of the article: L.F. Garabedian, P. Chu, S. Toh, A.M. Zaslavsky, S.B. Soumerai.

Statistical expertise: L.F. Garabedian, P. Chu, S. Toh, A.M. Zaslavsky, S.B. Soumerai.

Obtaining of funding: S.B. Soumerai.

Collection and assembly of data: L.F. Garabedian, P. Chu.

Ann Intern Med. 2014;161(2):131-138. doi:10.7326/M13-1887
Text Size: A A A

Instrumental variable analysis is an increasingly popular method in comparative effectiveness research (CER). In theory, the instrument controls for unobserved and observed patient characteristics that affect the outcome. However, the results of instrumental variable analyses in observational settings may be biased if the instrument and outcome are related through an unadjusted third variable: an “instrument–outcome confounder.”The authors identified published CER studies that used instrumental variable analysis and searched the literature for potential confounders of the most common instrument–outcome pairs. Of the 187 studies identified, 114 used 1 or more of the 4 most common instrument categories: distance to facility, regional variation, facility variation, and physician variation. Of these, 65 used mortality as an outcome. Potential unadjusted instrument–outcome confounders were observed in all studies, including patient race, socioeconomic status, clinical risk factors, health status, and urban or rural residency; facility and procedure volume; and co-occurring treatments. Only 4 (6%) instrumental variable CER studies considered potential instrument–outcome confounders outside the study data. Many effect estimates may be biased by the failure to adjust for instrument–outcome confounding. The authors caution against overreliance on instrumental variable studies for CER.


Grahic Jump Location
Figure 1.

Instrumental variable assumptions.

The instrumental variable method substitutes actual random assignment to treatment with an instrument, a variable that predicts treatment assignment but is not related to all other factors that influence the outcome. This method relies on 5 critical assumptions (see text). Instrumental variable estimates of causal effects may be biased if a third variable, an instrument–outcome confounder, has an effect on both the instrument and the outcome (violating the ignorable treatment assumption) or mediates an effect of the instrument on the outcome (violating the exclusion restriction).

Grahic Jump Location
Grahic Jump Location
Figure 2.

Instrumental variable comparative effectiveness research studies (n = 187), by year of publication.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Instrumental Variable Analysis for Observational Comparative Effectiveness Research: The Paired Availability Design
Posted on July 24, 2014
Stuart G. Baker, Karen S. Lindeman
National Institutes of Health, Johns Hopkins Medical Institutions
Conflict of Interest: None Declared
The recent article by Garabedian and colleagues (1) discussed potential bias in the instrumental variable analysis of observational studies. However it did not mention an important type of instrumental variable approach with observational data, namely the paired availability design (PAD) (2-4). Under PAD, time period is the instrumental variable, and the availability of new treatment for patients changes over time periods.
For a fixed increase in time of availability in one time period versus the other, PAD makes the key assumptions discussed by Garabedian and colleagues, namely (i) the effect of treatment is always in same direction and (ii) the instrument affects outcome only through treatment. For a randomly occurring increase in time of availability in one time period versus the other, PAD substitutes for (i) another consequence of the more fundamental assumption of no change in treatment preferences over time (4).
Because PAD does not involve an adjustment for baseline variables, it has the advantage of simplifying data collection. However investigators must then be careful to reduce bias in the study design. The key potential bias is the possibility that time affects outcome directly and not through treatment. To mitigate the potential for this bias, we recommend using short time periods, keeping all other aspects of patient management constant over time, keeping staff constant over time, and analyzing only geographically or institutionally isolated medical centers (which reduces the possibility of temporal changes in demographics of the eligible population). Also the use of multiple medical centers averages the effect of random changes over time.
In 1994, we introduced PAD with a proposed study of the effect of epidural analgesia on the probability of Cesarean section (2). After data became available, we compared the results of PAD to results from a meta-analysis of randomized trials as well results from a multivariate adjustment for possible confounders (4). Despite the aforementioned potential biases, our application of PAD to obstetric data gave qualitatively similar results as the meta-analysis of randomized trials, but substantially differed from results using the multivariate adjustment. We believe the multivariate adjustment was biased by the omission of a key unmeasured confounder.
All methods of analyzing observational data require assumption. In some applications, we believe that PAD offers more reasonable assumptions (and hence less possibility of bias) than other methods.

1. Garabedian LF, Chu P, Toh S, Zaslavsky AM, Soumerai SB. Potential bias of instrumental variable analyses for observational comparative effectiveness research. Ann Int Med. 2014;161(2):131-138.
2. Baker SG, Lindeman KS. The paired availability design: a proposal for evaluating epidural analgesia during labor. Stat Med. 1994;13:2269-2278.
3. Baker SG, Lindeman KL Kramer BS The paired availability design for historical controls. BMC Med Res Meth. 2001; 1:9.
4. Baker SG and Lindeman KL. Revisiting a discrepant result: a propensity score analysis, the paired availability design for historical controls, and a meta-analysis of randomized trials, J Causal Inference. 2013; 1:51–82.

Author's Resonse
Posted on September 22, 2014
Laura F. Garabedian, MPH, PhD, Alan M. Zaslavsky, PhD, Stephen B. Soumerai, ScD
Harvard Medical School
Conflict of Interest: None Declared
Baker and Lindeman question our omission of time period as a potential instrumental variable for comparative effectiveness research (CER). Our study focused on the four most commonly used instruments in CER. However, we did acknowledge the potential for time-based instruments: “The assumption that the instrument is only related to the outcome through the treatment may apply best to specific, focused, plausibly exogenous interventions or events such as natural experiments (e.g., the Oregon Medicaid lottery [1]) or changes in policy or technology [2]).” The paired availability design (PAD) studies cited by Baker and Lindeman did not explicitly employ instrumental variable analysis, so they were not included in our systematic review.

We agree that the introduction of a medical intervention, technology or a policy change affecting treatment uptake might be a valid instrument. Nonetheless, such analyses rely on key assumptions, many of which Baker and Lindeman consider in their PAD studies, that often are not satisfied in practice (3, 4). In particular, changes in patient characteristics or other aspects of treatment that occur at the same time as the treatment of interest could be instrument-outcome confounders and bias the instrumental variable estimate.

A truly exogenous and isolated policy satisfies these assumptions (2, 4). Similarly, a randomized instrument, such as the Oregon Medicaid lottery (1), protects against instrument-outcome confounding. However, policy changes and randomized encouragement instruments may be weak predictors of the treatment if they do not sufficiently change provider or patient behavior. A weak instrument will inflate any residual bias (4).

In addition, time-based instrumental variable analyses, including PAD, rely on a pre-post comparison and do not account for trends that may bias results; this is particularly problematic for interventions that manifest over long periods of time. Interrupted time series methods have been developed to account for many of the biases that plague temporally identified interventions (5) and may be more appropriate than instrumental variable analyses.

1. Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP, et al. The Oregon experiment--effects of Medicaid on clinical outcomes. NEJM. 2013;368(18):1713–1722.

2. Angrist JD, Krueger AB. Instrumental Variables And The Search For Identification: From Supply And Demand To Natural Experiments. Journal of Economic Perspectives. 2001;15(4):69-85.

3. Baker SG, Lindeman KS and Kramer BS. The paired availability design for historical controls. BMC Research Methodology. 2001; 1:9.

4. Brookhart MA, Rassen JA and Schneiweiss S. Instrumental variable methods in comparative safety and effectiveness research. Pharmacoepidemiol Drug Saf. 2010; 19(6):537-554.

5. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27(4):299-309.

Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Journal Club
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.