Benjamin D. Sommers, MD, PhD; Sharon K. Long, PhD; Katherine Baicker, PhD
Presented in part at the 36th Annual Meeting of the Society of General Internal Medicine in Denver, Colorado, on 26 April 2013 and the Annual Research Meeting of AcademyHealth in Baltimore, Maryland, on 24 June 2013.
Disclaimer: Dr. Sommers is an advisor in the Office of the Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services. However, this paper was written in Dr. Sommers’ capacity as a Harvard employee and does not represent the views of the U.S. Department of Health and Human Services.
Acknowledgment: The authors thank James Ware at the Harvard School of Public Health for thoughtful advice on our statistical analysis; Jacob Robbins and Sarah Gordon, Research Assistants at the Harvard School of Public Health, for their work on this project; and Katherine Hempstead at the Robert Wood Johnson Foundation for helpful suggestions related to health care–amenable mortality.
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-2275.
Reproducible Research Statement: Study protocol and data set: Not available. Statistical code: Available from Dr. Sommers (e-mail, firstname.lastname@example.org).
Requests for Single Reprints: Benjamin D. Sommers, MD, PhD, Department of Health Policy and Management, Harvard School of Public Health, Kresge Building, Room 406, 677 Huntington Avenue, Boston, MA 02115; e-mail, email@example.com.
Current Author Addresses: Drs. Sommers and Baicker: Department of Health Policy and Management, Harvard School of Public Health, Kresge Building, Room 406, 677 Huntington Avenue, Boston, MA 02115.
Dr. Long: Health Policy Center, Urban Institute, 2100 M Street NW, Washington, DC 20037.
Author Contributions: Conception and design: B.D. Sommers, S.K. Long, K. Baicker.
Analysis and interpretation of the data: B.D. Sommers, S.K. Long, K. Baicker.
Drafting of the article: B.D. Sommers, S.K. Long, K. Baicker.
Critical revision of the article for important intellectual content: B.D. Sommers, S.K. Long, K. Baicker.
Final approval of the article: B.D. Sommers, S.K. Long, K. Baicker.
Statistical expertise: B.D. Sommers, S.K. Long, K. Baicker.
Collection and assembly of data: B.D. Sommers.
Sommers B., Long S., Baicker K.; Changes in Mortality After Massachusetts Health Care Reform: A Quasi-experimental Study. Ann Intern Med. 2014;160:585-593. doi: 10.7326/M13-2275
Download citation file:
Published: Ann Intern Med. 2014;160(9):585-593.
The Massachusetts 2006 health care reform has been called a model for the Affordable Care Act. The law attained near-universal insurance coverage and increased access to care. Its effect on population health is less clear.
To determine whether the Massachusetts reform was associated with changes in all-cause mortality and mortality from causes amenable to health care.
Comparison of mortality rates before and after reform in Massachusetts versus a control group with similar demographics and economic conditions.
Changes in mortality rates for adults in Massachusetts counties from 2001 to 2005 (prereform) and 2007 to 2010 (postreform) were compared with changes in a propensity score–defined control group of counties in other states.
Adults aged 20 to 64 years in Massachusetts and control group counties.
Annual county-level all-cause mortality in age-, sex-, and race-specific cells (n = 146 825) from the Centers for Disease Control and Prevention's Compressed Mortality File. Secondary outcomes were deaths from causes amenable to health care, insurance coverage, access to care, and self-reported health.
Reform in Massachusetts was associated with a significant decrease in all-cause mortality compared with the control group (−2.9%; P = 0.003, or an absolute decrease of 8.2 deaths per 100 000 adults). Deaths from causes amenable to health care also significantly decreased (−4.5%; P < 0.001). Changes were larger in counties with lower household incomes and higher prereform uninsured rates. Secondary analyses showed significant gains in coverage, access to care, and self-reported health. The number needed to treat was approximately 830 adults gaining health insurance to prevent 1 death per year.
Nonrandomized design subject to unmeasured confounders. Massachusetts results may not generalize to other states.
Health reform in Massachusetts was associated with significant reductions in all-cause mortality and deaths from causes amenable to health care.
John Tayu Lee, Christopher Millett
Imperial College London
June 24, 2014
Sommers et al (2014)1 examined the impact of Massachusetts 2006 health care reform on socioeconomic disparities in mortality. Using sophisticated methods, their results suggested this policy has protective effect on all-cause mortality as well as mortality for conditions that are most susceptible to health care utilisation. Perhaps most importantly, they found the effects were larger for those from lower incomes and uninsured by conducting subgroup analyses. An alternative approach to model the differential impact of the policy is to use interrupted times series with interaction terms between socioeconomic groups and three main variables in ITS model: time trend, dummy variable for policy, and continuous variable for duration of exposure. This model specification would allow patients in different income groups to have different baseline time trend, level change, and slope change from the reference group. This study design will model on the whole sample which means it reduces the risk of inefficient sample size when conducting subgroup analysis. The use of interrupted times series is becoming common in the evaluation of health interventions, but as far as we know, very few studies that use this method to examine policy effect on health inequalities by including interaction terms. REFERENCE1. Sommers BD, Long SK, Baicker K. Changes in Mortality After Massachusetts Health Care Reform: A Quasi-experimental Study. Annals of Internal Medicine 2014; 160(9): 585-93.
Robert Kaestner, PhD
Univeristy of Illinois
February 3, 2015
In a recent paper, Sommers et al. evaluated whether health reform in 2006 in Massachusetts, which resulted in a decrease in the proportion of persons without health insurance, affected mortality and concluded that health reform did in fact decrease mortality. (1) If true, this is an important finding, as there is relatively little evidence of a causal link between health insurance coverage and mortality. Moreover, Massachusetts reform was the forerunner of the reform contained in the Affordable Care Act. The significance of the published finding is reflected in the widespread media attention the article received. (2) One potential problem with the analysis was that inferences were based on an approach that was unlikely to be valid. The natural experiment that was the basis of the analysis was characterized by only one treatment state among 47 states and sample sizes in the states that were quite uneven in terms of the number of observations per state. In these settings, common approaches to inference have been shown to significantly over reject the null hypothesis. (3-5) I assessed the extent of the potential inference problem by calculating p-values for estimates in the article using randomization inference methods. (6) The p-values derived from this method were very large and ranged from 0.277 to 0.783. These p-values indicate that none of the estimates reported in the article are statistically significant at commonly accepted levels of significance. Moreover, given the relatively large magnitudes of the estimates that were reported, the large p-values associated with these estimates suggest that the analysis lacked adequate statistical power to detect plausible effect sizes. In sum, my analysis indicates that the question of whether Massachusetts health reform affected mortality remains unanswered. Robert Kaestner, Ph.D. Institute of Government and Public Affairs References1. Sommers, Benjamin D., Sharon K. Long, and Katherine Baicker. 2014. “Changes in Mortality After Massachusetts Health Care Reform: A Quasi-Experimental Study.” Annals of Internal Medicine 160 (9): 585–93. doi:10.7326/M13-2275.2. Tavernise, Sabrina. “Mortality Drop Seen to Follow ’06 Health Law,” The New York Times, May 5 2014. http://www.nytimes.com/2014/05/06/health/death-rate-fell-in-massachusetts-after-health-care-overhaul.html?_r=0, last accessed January 30, 2015. (A version of this article appears in print on May 6, 2014, on page A16 of the New York edition with the headline: Mortality Drop Follows Massachusetts Health Law.)3. Cameron, A. Colin, Jonah B. Gelbach, and Douglas L. Miller. 2008. “Bootstrap-Based Improvements for Inference with Clustered Errors.” The Review of Economics and Statistics 90 (3): 414–27. 4. Conley, Timothy G., and Christopher R. Taber. 2011. “Inferences with “Difference in Differences” with a Small Number of Policy Changes.” The Review of Economics and Statistics 93 (1): 113-125. 5. MacKinnon, James G., and Matthew D. Webb. 2014. Wild Bootstrap Inference for Wildly Different Cluster Sizes. Working Paper 1314. Queen’s University, Department of Economics. https://ideas.repec.org/p/qed/wpaper/1314.html, last accessed January 30, 2015.6. Rosenbaum, Paul R. 2002. “Covariance Adjustment in Randomized Experiments and Observational Studies.” Statistical Science 17 (3): 286–327. doi:10.1214/ss/1042727942.
Benjamin D. Sommers, Sharon K. Long, and Katherine Baicker
Harvard School of Public Health (BDS, KB) and the Urban Institute (SKL)
March 19, 2015
Conflict of Interest:
Dr. Sommers is an advisor in the Office of the Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services. However, this paper was written in Dr. Sommers’ capacity as a Harvard employee and does not represent the views of the<br/>U.S. Department of Health and Human Services. We received no funding for this work.
Authors' Response to Dr. Kaestner's Comment
We appreciate the concerns raised by Dr. Kaestner in his comment. Testing the robustness of our findings to alternative appropriate estimation techniques is a critical way to test the reliability of our quasi-experimental approach. Lacking sufficient detail on the analysis conducted by Dr. Kaestner, we cannot evaluate the soundness of his approach or the validity of his results. However, we believe his concern about uneven cluster size driving spurious results is addressed by the state-level analysis presented in Appendix Table 4 of our paper, described in detail in the on-line Supplemental Appendix.(1) We analyzed the study’s 47 states using annual state race-age-sex mortality rates. The analysis used a sample with an identical number of observations per state cluster – 288 race-sex-age yearly mortality rates – in 43 of the 47 states, while the other four states had some empty race-sex-age cells, leading to state samples ranging from 241-285. In that analysis, which avoids Dr. Kaestner’s primary methodological concern, we found large and statistically significant reductions in unadjusted all-cause mortality, as well as adjusted and unadjusted heath-care amenable mortality. Thus, the key results of the paper do not appear to be driven by uneven cluster sizes. More generally, our paper builds on numerous studies of Massachusetts health reform that have taken similar approaches with state-level clustering, using well-established methods for these sorts of analyses.(2,3) Our results remained statistically significant (Appendix Table 4) when using county-level clustering, which has been used by other researchers in county-based analyses of Massachusetts health reform such as ours.(4,5)Of course, all statistical estimation techniques have strengths and weaknesses, and the standard errors in quasi-experimental approaches are often sensitive to assumptions. This concern motivated the multiple approaches taken in the Appendix. We were reassured by the robustness of the results across approaches. We agree that great care must be taken with statistical inference in designs such as ours, but our analysis indicates that our use of comprehensive mortality statistics capturing every death in the study population of nearly 50 million people over nine years generated more than sufficient power to detect clinically meaningful mortality changes. REFERENCES1. Sommers BD, Long SK, Baicker K. Changes in Mortality After Massachusetts Health Care Reform: A Quasi-experimental Study. Ann Intern Med. May 6 2014;160(9):585-593.2. Courtemanche CJ, Zapata D. Does universal coverage improve health? The Massachusetts experience. J Policy Anal Manage. Winter 2014;33(1):36-69.3. Van Der Wees PJ, Zaslavsky AM, Ayanian JZ. Improvements in health status after Massachusetts health care reform. Milbank Q. Dec 2013;91(4):663-689.4. Miller S. The effect of insurance on emergency room visits: An analysis of the 2006 Massachusetts health reform. J Pub Econ. 2012;96(11-12):893-908.5. Hanchate AD, Lasser KE, Kapoor A, et al. Massachusetts reform and disparities in inpatient care utilization. Med Care. Jul 2012;50(7):569-577.
to gain full access to the content and tools.
Learn more about subscription options.
Register Now for a free account.
Healthcare Delivery and Policy.
Results provided by:
Copyright © 2016 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use
This PDF is available to Subscribers Only