Methods for Evaluating Natural Experiments in Obesity: A Systematic ReviewFREE
Submit a Comment
Contributors must reveal any conflict of interest. Comments are moderated. Please see our information for authorsregarding comments on an Annals publication.
Abstract
Background:
Given the obesity pandemic, rigorous methodological approaches, including natural experiments, are needed.
Purpose:
To identify studies that report effects of programs, policies, or built environment changes on obesity prevention and control and to describe their methods.
Data Sources:
PubMed, CINAHL, PsycINFO, and EconLit (January 2000 to August 2017).
Study Selection:
Natural experiments and experimental studies evaluating a program, policy, or built environment change in U.S. or non-U.S. populations by using measures of obesity or obesity-related health behaviors.
Data Extraction:
2 reviewers serially extracted data on study design, population characteristics, data sources and linkages, measures, and analytic methods and independently evaluated risk of bias.
Data Synthesis:
294 studies (188 U.S., 106 non-U.S.) were identified, including 156 natural experiments (53%), 118 experimental studies (40%), and 20 (7%) with unclear study design. Studies used 106 (71 U.S., 35 non-U.S.) data systems; 37% of the U.S. data systems were linked to another data source. For outcomes, 112 studies reported childhood weight and 32 adult weight; 152 had physical activity and 148 had dietary measures. For analysis, natural experiments most commonly used cross-sectional comparisons of exposed and unexposed groups (n = 55 [35%]). Most natural experiments had a high risk of bias, and 63% had weak handling of withdrawals and dropouts.
Limitation:
Outcomes restricted to obesity measures and health behaviors; inconsistent or unclear descriptions of natural experiment designs; and imperfect methods for assessing risk of bias in natural experiments.
Conclusion:
Many methodologically diverse natural experiments and experimental studies were identified that reported effects of U.S. and non-U.S. programs, policies, or built environment changes on obesity prevention and control. The findings reinforce the need for methodological and analytic advances that would strengthen evaluations of obesity prevention and control initiatives.
Primary Funding Source:
National Institutes of Health, Office of Disease Prevention, and Agency for Healthcare Research and Quality. (PROSPERO: CRD42017055750)
Obesity is a worldwide health problem: Globally, an estimated 1.9 billion adults are overweight or obese (1–3). Because the drivers of the obesity pandemic are complex, effective solutions to prevent and control obesity must extend beyond focusing on the individual to address the neighborhood context, as well as the social, cultural, and political background unique to regions and countries (2, 4, 5). To have the greatest effect, the Institute of Medicine recommended a “systems approach” targeting changes in 5 critical environments: physical activity, foods and beverages, messages, health care and work, and schools (6). Since then, many localities have enacted population-level policies to reduce obesity, including a sugar-sweetened beverage tax in Berkeley, California (7); calorie-labeling regulations in New York City (8–10); and support for building supermarkets in low-income neighborhoods (11). In addition, many school systems have implemented programs to boost children's fruit and vegetable consumption and increase outdoor time (12, 13). Evaluations of these policies and programs are enhancing our ability to adapt, scale, and disseminate those found to be effective (14).
Challenges in assessing these interventions stem from the complexity of the obesity problem, which tends to thwart interventions that focus on a single factor—or even several factors—and from a lack of standards for designing and evaluating interventions (2). Evaluations must take advantage of existing data sources; link policy, program, or transportation data to health data, such as electronic health records (EHRs); and follow populations over time. Furthermore, randomized controlled trials (RCTs) of obesity interventions are not always feasible or appropriate.
The purpose of this systematic review was to identify studies reporting the effects of programs, policies, or built environment changes on obesity prevention and control, and to describe their methods to better understand the population-based data sources, data linkages, and methodological and analytic approaches.
Methods
We addressed 6 key questions (KQs), formulated with input from experts, to inform the National Institutes of Health Pathways to Prevention Workshop Methods for Evaluating Natural Experiments in Obesity (5 to 6 December 2017). The full evidence report contains additional details (15).
This article reports on 5 of the 6 KQ topics: population-based data sources (KQ1) and data linkages (KQ2); obesity measures, dietary and physical activity behaviors, and other outcomes (KQ3); experimental and nonexperimental methods (KQ4); and risk of bias (KQ5).
Data Sources and Search Strategy
We searched PubMed, CINAHL, PsycINFO, and EconLit (1 January 2000 to 21 August 2017). This time frame was selected to encompass studies after the 2001 publication of The Surgeon General's Call to Action to Prevent and Decrease Overweight and Obesity (16).
Study Selection
Abstracts and full-text articles were screened by 2 trained team members; a third member adjudicated disagreements. We included natural experiments and experimental study designs that evaluated programs, policies, or built environment changes in U.S. or non-U.S. populations of all ages and that reported obesity measures for adults (body weight or body mass index [BMI]) or children (BMI Z score or percentile), or obesity-related individual health behaviors (dietary and physical activity). Studies were excluded if they had fewer than 100 participants, lacked a comparison or unexposed group or pre–post comparison, or were performed in a country that ranked below “very high” on the 2016 United Nations Human Development Index.
Data Extraction
Trained researchers serially abstracted the study's setting (such as school, community, or workplace); participants' demographic characteristics; details of the policy, program, or built environment changes (including name or bill number, original goal, governing body, year of enactment, implementation and completion, and target environment [for example, food and beverage environment]); data sources and their linkages; study design; and analytic methods.
Data Synthesis
We applied the United Kingdom Medical Research Council's definition of a natural experiment as a study in which the researchers do not have control of the intervention assignment, versus an experimental study or other nonexperimental study design in which the intervention assignment is not clear (17).
KQ1 and KQ2
We applied previously developed criteria to assess whether a data source was a data system—that is, whether it was found on the Web; was digitally accessible; was sharable; and contained outcomes or exposures of interest. We described data linkage methods between U.S. data systems.
KQ3
We described the types of measures to assess adult body weight or BMI, childhood BMI (Z score and percentile), and individual dietary behaviors in terms of total daily caloric intake, nutrients related to obesity (vegetable, fruit, or fiber intake), frequency of sugar-sweetened beverage or fast food intake, and individual physical activity behavior (both type and quantity of activity). We assessed additional outcomes, such as the food or physical activity environment.
KQ4
We described the types of study designs and analytic methods, as well as their frequency of use, by study design.
Two reviewers independently evaluated each study's risk of bias using 6 domains from the Effective Public Health Practice Project (EPHPP) quality assessment tool: selection bias, study design, confounding, blinding, data collection, and withdrawals and dropouts (18). Studies received domain-specific ratings (strong, moderate, or weak) according to the EPHPP algorithm (18). Each study also received a global risk-of-bias rating: “strong” if no domain was rated weak, “moderate” if only 1 domain was rated weak, or “weak” if 2 or more domains were rated weak. We developed additional study design–specific risk-of-bias questions for the nonexperimental designs used in natural experiments, such as interrupted time series (15).
Role of the Funding Source
The National Institutes of Health, Office of Disease Prevention, funded the review through an interagency agreement with the Agency for Healthcare Research and Quality (AHRQ). Neither organization had a role in study selection, assessment, or synthesis.
Results
Study Characteristics
We identified 26 316 unique citations, of which 294 studies (reported in 312 articles) were eligible for inclusion (Figure 1). The published report lists all 294 included, as well as excluded, articles (15). Most studies were conducted in the United States (n = 188), followed by Canada, the United Kingdom, and Italy. More than half the studies (n = 156 [53%]) met the criteria for natural experiments, 40% (n = 118) were experimental studies or had another study design, and 7% (n = 20) had an uncertain classification. Table 1 breaks down the number of studies according to study method; program, policy, or built environment goals; and target settings. Most studies, regardless of design, targeted children and schools and addressed changes in the physical activity and food and beverage environments. Of the 188 U.S. studies, we identified 139 unique policy or program evaluations. Examples were food and beverage policies, such as sugar-sweetened beverage bans (19); competitive food (20) and calorie-labeling laws (9, 10); federal programs, such as Women, Infants, and Children and the Supplemental Nutrition Assistance Program (21–23); and physical activity policies (24–26).
Figure 1. Evidence search and selection.
HDI = Human Development Index; KQ = key question.
* Sum of excluded abstracts exceeds 25 093 because reviewers were not required to agree on reasons for exclusion.
† Sum of excluded articles exceeds 911 because reviewers were not required to agree on reasons for exclusion.
‡ 294 studies (on a unique intervention, applied to a unique population) were identified; 312 individual articles reported on these 294 studies.
Population-Based Data Sources and Their Linkage Methods
All 294 studies reported using 1 or more population-based data sources. Ninety-three studies used a total of 169 data sources that were shareable (26 were used for primary data collection and 143 were secondary data sources). One hundred six unique data sources (71 U.S. and 35 non-U.S.) met the criteria for a data system (96 in natural experiments) (Appendix Table 1).
Thirty-nine percent of the 71 U.S. data systems were originally designed for an administrative purpose, 31% were for public health operations, and 57% had national coverage. Figure 2 characterizes the U.S. data systems with regard to their availability on the Web, the accessibility of their data, and the provision of a data dictionary. Although most data systems (73%) had a dedicated Web presence, few provided detailed information (such as data elements or data quality reports). For example, the NHANES (National Health and Nutrition Examination Survey) provides detailed information about data quality issues and potential analytic pitfalls.
Figure 2. Characterization of the 71 U.S. data systems in terms of availability (Web page), data accessibility, and provision of a data dictionary, among the 294 data sources.
For a data source to be considered a data system, it must exist on the Web, be available and accessible in digital format for use by other researchers (that is, sharable), and collect 1 or more outcomes of interest related to obesity, as defined in this project. Values in the light-green bars are mutually exclusive.
Of the 71 U.S. data systems, 26 (37%) were linked to a secondary data source or system other than a primary data source. Studies that linked their data systems to several external data systems used either an individual-level key (10 studies [for example, patient identifiers]) or a geographic allocation (16 studies [for example, a patient residing in a specific county, thus mapping the county specifications from other data sources for that individual]).
Measurement of Dietary and Physical Activity Behaviors in Children and Adults
Weight and BMI
One hundred twelve studies reported childhood weight and 32 reported adult weight outcomes. Table 2 displays the methods for measuring obesity in adults and children, by study design. In children, the most common measure was BMI Z score in both natural experiments (n = 19) and experimental studies (n = 5). Twenty-six of the 46 natural experiments used secondary data sources containing childhood weight and height measured by trained staff, such as the National Survey of Children's Health (27), Early Childhood Longitudinal Study—Kindergarten Cohort (28), and School Health Policies and Programs Study (29).
Thirty-two of 294 studies reported adult body weight (n = 6) and BMI (n = 31) outcomes (17 natural experiments, 13 experimental studies, and 2 other studies). Most studies reporting weight measures in adults were conducted in community (n = 15) or worksite (n = 10) settings. Of the 17 natural experiments, 11 used self-reported data, 6 used direct measurement by trained staff, and none used data from the EHR.
Diet and Physical Activity Behaviors
One hundred forty-eight studies reported dietary behavioral outcomes in terms of changes in intakes of fruits and vegetables (n = 147), sugar-sweetened beverages (n = 54), total daily calories (n = 17), fast food (n = 16), or fiber (n = 12) (Appendix Table 2). In children, the most frequently used diet assessment method was 24-hour recall (n = 31); for adults, the most frequent method was a food-frequency questionnaire (n = 26), such as the one from the Behavioral Risk Factor Surveillance System (BRFSS), which was used in 4 natural experiments (30–33).
One hundred fifty-two studies reported physical activity; most (n = 89) were done in the school (vs. community) setting. All 42 studies with a goal of changing the parks and recreation or transportation environment measured physical activity. Thirty-two of the 106 studies in children used electronic monitoring, such as accelerometers, whereas only 10 studies in adults did so (Appendix Table 3).
Thirty-seven studies (26 natural experiments) reported on additional outcomes, such as food purchasing behavior (n = 17), physical environment (n = 8), commuting (n = 4), and food environment (n = 6).
Experimental and Natural Experiment Study Designs and Analytic Methods
Table 3 summarizes the study designs used. The most common analytic approach in 156 natural experiments was cross-sectional comparison of exposed and unexposed groups (n = 55 [35%]), followed by pre–post analyses, with the preintervention period serving as the control for the postintervention period (n = 48 [31%]), and difference-in-differences approaches looking at changes over time compared with an external control group (n = 45 [29%]). All pre–post analyses measured variables at a single time point before intervention, and 80% had a single measure after intervention. A few natural experiments used other nonexperimental designs, including 4 instrumental variable approaches, 1 regression discontinuity approach, and 4 interrupted time series analyses with more than 2 time points before and after intervention.
Of the 118 experimental studies (40%), 74 were RCTs and 44 were controlled clinical trials. The unit of intervention allocation was most often at the organization (66%) or community (23%) level, and 93% of the studies used individual-level analyses.
Risks of Bias in Studies
Figure 3 shows risk-of-bias ratings for the 156 natural experiments. The domains most likely to be rated as strong (that is, low risk of bias) were data collection methods and confounding, but still only 74 (47%) were rated as such for data collection and 69 (44%) for confounding. Forty-three natural experiments (28%) were rated strong on selection bias; 98 studies (63%) were rated “very likely” that the study sample represented the target population, but only 40 (25%) reported that 80% to 100% of selected persons agreed to participate. Ninety-nine natural experiments (64%) were rated as weak (high risk of bias) in the domain of withdrawals and dropouts. For 30% of studies, withdrawals and dropouts were not reported or not enough information was given to determine how attrition was handled.
Figure 3. Risk of bias for natural experiment studies using the EPHPP tool (n = 156).
EPHPP = Effective Public Health Practice Project.
* Studies were given a “strong” global rating if no domains had a “weak” rating, a “moderate” global rating if a single domain had a weak rating, and a weak global rating if 2 or more domains had a weak rating.
Fifty-five percent of experimental studies were rated as having a strong study design and 61% as adequately addressing confounding. However, most were rated as moderate or weak in the areas of blinding, selection bias, and handling withdrawals and dropouts.
Discussion
To address the obesity pandemic, researchers, policymakers, and funders recognize the need for natural experiments to efficiently and practically identify successful programs and policies that can be disseminated widely and scaled to communities and schools (3, 34). Our systematic review identified many methodologically diverse natural experiments (n = 156) and data sources (n = 106) that have been used to estimate the effect of programs, policies, or built environment changes on obesity prevention and control. Natural experiments, like other obesity study designs, lacked consistent measures of diet and physical activity, and few assessed neighborhood or school co-benefits or harms associated with the policies and programs. Studies often had substantial risk of bias, particularly because of their handling of withdrawals and dropouts, weak study design, and management of confounding. Our findings reinforce the need for methodological and analytic advances, including reporting standards that would strengthen the evaluation of efforts to improve obesity prevention and control.
Although we identified 106 unique data sources (Appendix Table 1), most researchers used large, publicly available national surveys, such as NHANES, BRFSS, and the Youth Risk Behavioral Surveillance System, or state-level data, most commonly from California, New York, Massachusetts, Minnesota, Pennsylvania, and Texas. The Patient-Centered Outcomes Research Institute established the National Patient-Centered Clinical Research Network (PCORNet), with 13 Clinical Data Research and 20 Patient-Powered Research Networks to link patient data longitudinally across organizations as well as large U.S. health systems (35). Although the infrastructure is still under development and testing, PCORNet exemplifies how EHR data might be used to evaluate obesity prevention policies and programs, especially through links to other public health data sources. Community Commons is an example of a Web portal with the infrastructure to house and organize community-level data. It enables data sharing and interactive mapping functionality (36) for community evaluators and researchers (36, 37). Another resource for investigators is the National Collaborative on Childhood Obesity Research (NCCOR), which has focused on childhood obesity prevention data sources and measures. Expansion of NCCOR to include adults, data sources beyond health, and data dictionaries and guides to performing data linkages at the individual and geographic levels might help obesity researchers evaluate natural experiments and programs. Only one third of the studies linked their data to other sources, despite the potential for data linkages to expand researchers' ability to assess the effect of geography or neighborhood characteristics on outcomes of interest. Barriers to accessing and sharing EHR data include ethical and privacy concerns; retrieving commercial data related to the food system poses additional hurdles (38).
Although RCTs are considered the gold standard for reducing risk of bias, they are challenging to implement because of their high cost, and random assignment of participants or communities to policies or programs often is not feasible (39). To improve the applicability and validity of evaluations, obesity researchers might consider innovative trial designs that would allow randomization, such as stepped wedge or waitlist control designs. Although obesity policy evaluations are particularly amenable to natural experiments, most had substantial risk of bias. Advancing the validity and trustworthiness of future natural experiments should involve greater use of multiple comparison groups as well as sensitivity analyses to assess the robustness of results to inclusion or exclusion of variables in the models.
Finally, we expected to find greater use of nonexperimental designs and analytic approaches, such as propensity scores, instrumental variables, regression discontinuity, and interrupted time series, to improve comparison group selection and the ability to make causal inferences. The Medical Research Council recommends several pre–post measures to improve the design of natural experiments, especially if a control group is not available, as in the pre–post designs (17). However, most studies were cross-sectional (35%), were pre–post (31%) with single pre–post time points, or used a difference-in-differences design (29%). Depending on the research question, the type of natural experiment, and the stability of the outcome of interest, adding several time points or comparison groups should improve study validity but may have cost or time implications for researchers. Additional comparisons may be made by using synthetic control populations, regression discontinuity, or propensity score approaches.
We note some limitations to our systematic review. First, we excluded several types of studies: those that assessed associations between perceived or measured home, school, or neighborhood environment and outcomes (40) but without a clear program, policy, or environment change; studies without 1 of our main outcomes, such as those assessing the effect of menu labeling on caloric information awareness (41); and parks and transportation usage analyses that reported observed counts of people without individual-level change in physical activity (42). Second, the definition of which methods and designs constitute a natural experiment continues to evolve (17). In applying the Medical Research Council's definition, we found that few studies self-identified as natural experiments, studies used many different (but often poorly defined) terms for reporting their design (such as quasi-experiments), and some did not clearly report the role of the investigator in intervention assignment, making it challenging to classify these studies. Third, because no risk-of-bias assessment tool exists specifically for natural experiments, we relied on a previously developed instrument. Such tools as the EPHPP have been criticized for being overly rigorous and failing to account for feasibility, implementation, and future scalability, which are important considerations for natural experiments. In addition, we applied the EPHPP's definition of selection bias, which assesses the extent to which study participants are likely to represent the target population (18) and is more relevant to public health studies but is not the textbook definition of selection bias (43).
The implications from this review emphasize the need to enhance obesity researchers' access to information sources, including commercial data (such as information from food retailers and mapping data on mobile phones). The results also indicate a need for better infrastructure for existing data sources, with detailed codebooks and practical tutorials for performing data linkages. Investments must be made in communities that are collecting ongoing health and behavioral information and linking to data sources at the health care, community, school, and public health levels. Good examples of community-based data collection are Michigan's Project Healthy Schools (44) and Shape Up Somerville (Massachusetts) (45). The former was created through collaboration between the University of Michigan and local community organizations, including public schools, to assess the long-term effect of a wellness program in schools (44). Enhancing the validity and trustworthiness of future research will require collaboration between nonexperimental design methodologists and obesity content researchers to broaden the study design toolbox for natural experiment studies. To improve the rigor, consistency, and transparency regarding risk of bias in obesity natural experiments, the field needs standard reporting guidelines. Finally, given the public health importance and complexity of obesity, the field would benefit from a clearinghouse of high-quality, evidence-based obesity prevention and control studies, held to rigorous methodological standards, to provide best-practices or “ready-to-scale” evidence for researchers, policymakers, and practitioners. An example from the education field is the What Works Clearinghouse, which applies criteria to review existing research on different educational programs, products, practices, and policies to inform educators (46).
References
1.
World Health Organization. Global Action Plan for the Prevention and Control of Noncommunicable Diseases 2013-2020. Geneva: World Health Organization; 2013.
2.
Swinburn BA, Sacks G, Hall KD, McPherson K, Finegood DT, Moodie ML, et al. The global obesity pandemic: shaped by global drivers and local environments. Lancet. 2011;378:804-14. [PMID: 21872749] doi: 10.1016/S0140-6736(11)60813-1
3.
NCD Risk Factor Collaboration (NCD-RisC). Worldwide trends in body-mass index, underweight, overweight, and obesity from 1975 to 2016: a pooled analysis of 2416 population-based measurement studies in 128·9 million children, adolescents, and adults. Lancet. 2017;390:2627-42. [PMID: 29029897] doi: 10.1016/S0140-6736(17)32129-3
4.
Gortmaker SL, Swinburn BA, Levy D, Carter R, Mabry PL, Finegood DT, et al. Changing the future of obesity: science, policy, and action. Lancet. 2011;378:838-47. [PMID: 21872752] doi: 10.1016/S0140-6736(11)60815-5
5.
Hammond RA. Complex systems modeling for obesity research. Prev Chronic Dis. 2009;6:A97. [PMID: 19527598]
6.
Committee on Evaluating Progress of Obesity Prevention Effort; Food and Nutrition Board; Institute of Medicine. Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress. Washington, DC: National Academies Pr; 2013.
7.
Falbe J, Thompson HR, Becker CM, Rojas N, McCulloch CE, Madsen KA. Impact of the berkeley excise tax on sugar-sweetened beverage consumption. Am J Public Health. 2016;106:1865-71. [PMID: 27552267] doi: 10.2105/AJPH.2016.303362
8.
Elbel B, Kersh R, Brescoll VL, Dixon LB. Calorie labeling and food choices: a first look at the effects on low-income people in New York City. Health Aff (Millwood). 2009;28:w1110-21. [PMID: 19808705] doi: 10.1377/hlthaff.28.6.w1110
9.
Restrepo BJ. Calorie labeling in chain restaurants and body weight: evidence from New York. Health Econ. 2017;26:1191-1209. [PMID: 27451966] doi: 10.1002/hec.3389
10.
Vadiveloo MK, Dixon LB, Elbel B. Consumer purchasing patterns in response to calorie labeling legislation in New York City. Int J Behav Nutr Phys Act. 2011;8:51. [PMID: 21619632] doi: 10.1186/1479-5868-8-51
11.
Dubowitz T, Ghosh-Dastidar M, Cohen DA, Beckman R, Steiner ED, Hunter GP, et al. Diet and perceptions change with supermarket introduction in a food desert, but not because of supermarket use. Health Aff (Millwood). 2015;34:1858-68. [PMID: 26526243] doi: 10.1377/hlthaff.2015.0667
12.
Bere E, Veierød MB, Bjelland M, Klepp KI. Free school fruit—sustained effect 1 year later. Health Educ Res. 2006;21:268-75. [PMID: 16219630]
13.
Bere E, Veierød MB, Klepp KI. The Norwegian school fruit programme: evaluating paid vs. no-cost subscriptions. Prev Med. 2005;41:463-70. [PMID: 15917042]
14.
Sisnowski J, Street JM, Merlin T. Improving food environments and tackling obesity: a realist systematic review of the policy success of regulatory interventions targeting population nutrition. PLoS One. 2017;12:e0182581. [PMID: 28783757] doi: 10.1371/journal.pone.0182581
15.
Bennett WL, Cheskin LJ, Wilson RF, Zhang A, Tseng E, Shogbesan O, et al. Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review. Comparative Effectiveness Review no. 204. (Prepared by the Johns Hopkins University Evidence-based Practice Center under contract no. 290-2012-00007-I.) AHRQ publication no. 18-EHC006-EF. Rockville: Agency for Healthcare Research and Quality; 2017.
16.
Department of Health and Human Services. The Surgeon General's Call to Action to Prevent and Decrease Overweight and Obesity. Rockville: Department of Health and Human Services; 2001.
17.
Medical Research Council. Using natural experiments to evaluate population health interventions. 2010. Accessed at www.mrc.ac.uk/documents/pdf/natural-experiments-to-evaluate-population-health-interventions on 18 May 2017.
18.
EPHPP: Quality Assessment Tool for Quantitative Studies. 2009. Accessed at Effective Public Health Practice Project at https://merst.ca/wp-content/uploads/2018/02/quality-assessment-tool_2010.pdf on 5 April 2018.
19.
Taber DR, Chriqui JF, Powell LM, Chaloupka FJ. Banning all sugar-sweetened beverages in middle schools: reduction of in-school access and purchasing but not overall consumption. Arch Pediatr Adolesc Med. 2012;166:256-62. [PMID: 22064875] doi: 10.1001/archpediatrics.2011.200
20.
Wright K, Norris K, NewmanGiger J, Suro Z. Improving healthy dietary behaviors, nutrition knowledge, and self-efficacy among underserved school children with parent and community involvement. Child Obes. 2012;8:347-56. [PMID: 22867074] doi: 10.1089/chi.2012.0045
21.
Liu J, Kuo T, Jiang L, Robles B, Whaley SE. Food and drink consumption among 1-5-year-old Los Angeles County children from households receiving dual SNAP and WIC v. only WIC benefits. Public Health Nutr. 2017;20:2478-85. [PMID: 27609603] doi: 10.1017/S1368980016002329
22.
Tester JM, Leung CW, Crawford PB. Revised WIC food package and children's diet quality. Pediatrics. 2016;137. [PMID: 27244804] doi: 10.1542/peds.2015-3557
23.
Sekhobo JP, Edmunds LS, Dalenius K, Jernigan J, Davis CF, Giddings M, et al. Neighborhood disparities in prevalence of childhood obesity among low-income children before and after implementation of New York City child care regulations. Prev Chronic Dis. 2014;11:E181. [PMID: 25321632] doi: 10.5888/pcd11.140152
24.
Slater S, Chriqui J, Chaloupka FJ, Johnston L. Joint use policies: are they related to adolescent behavior? Prev Med. 2014;69 Suppl 1:S37-43. [PMID: 25199731] doi: 10.1016/j.ypmed.2014.08.032
25.
Kim J. Are physical education-related state policies and schools' physical education requirement related to children's physical activity and obesity? J Sch Health. 2012;82:268-76. [PMID: 22568462] doi: 10.1111/j.1746-1561.2012.00697.x
26.
Sabia JJ, Nguyen TT, Rosenberg O. High school physical education requirements and youth body weight: new evidence from the YRBS. Health Econ. 2017;26:1291-1306. [PMID: 27576770] doi: 10.1002/hec.3399
27.
Gee KA. School-based body mass index screening and parental notification in late adolescence: evidence from Arkansas's Act 1220. J Adolesc Health. 2015;57:270-6. [PMID: 26115907] doi: 10.1016/j.jadohealth.2015.05.007
28.
Capogrossi K, You W. The influence of school nutrition programs on the weight of low-income children: a treatment effect analysis. Health Econ. 2017;26:980-1000. [PMID: 27381591] doi: 10.1002/hec.3378
29.
Taber DR, Stevens J, Evenson KR, Ward DS, Poole C, Maciejewski ML, et al. State policies targeting junk food in schools: racial/ethnic differences in the effect of policy change on soda consumption. Am J Public Health. 2011;101:1769-75. [PMID: 21778484] doi: 10.2105/AJPH.2011.300221
30.
Sadler RC, Gilliland JA, Arku G. A food retail-based intervention on food security and consumption. Int J Environ Res Public Health. 2013;10:3325-46. [PMID: 23921626] doi: 10.3390/ijerph10083325
31.
Miewald C, Holben D, Hall P. Role of a food box program in fruit and vegetable consumption and food security. Can J Diet Pract Res. 2012;73:59-65. [PMID: 22668838]
32.
Cummins S, Flint E, Matthews SA. New neighborhood grocery store increased awareness of food access but did not alter dietary habits or obesity. Health Aff (Millwood). 2014;33:283-91. [PMID: 24493772] doi: 10.1377/hlthaff.2013.0512
33.
Barnidge EK, Hipp PR, Estlund A, Duggan K, Barnhart KJ, Brownson RC. Association between community garden participation and fruit and vegetable consumption in rural Missouri. Int J Behav Nutr Phys Act. 2013;10:128. [PMID: 24252563] doi: 10.1186/1479-5868-10-128
34.
Economos CD, Hammond RA. Designing effective and sustainable multifaceted interventions for obesity prevention and healthy communities. Obesity (Silver Spring). 2017;25:1155-6. [PMID: 28653500] doi: 10.1002/oby.21893
35.
Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc. 2014;21:578-82. [PMID: 24821743] doi: 10.1136/amiajnl-2014-002747
36.
Community Commons. Community Commons & IP3. Accessed at www.communitycommons.org on 18 December 2017.
37.
Fulcher CL. A unifying public good Web site for healthy, sustainable, and livable communities. Natl Civ Rev. 2014;103:9-12.
38.
Jones KK, Zenk SN, Tarlov E, Powell LM, Matthews SA, Horoi I. A step-by-step approach to improve data quality when using commercial business lists to characterize retail food environments. BMC Res Notes. 2017;10:35. [PMID: 28061798] doi: 10.1186/s13104-016-2355-1
39.
Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Community Health. 2012;66:1182-6. [PMID: 22577181] doi: 10.1136/jech-2011-200375
40.
Wong SH, Huang WY, Cerin E, Gao Y, Lai PC, Burnett A. Home and neighbourhood environment: association with children's physical activity and obesity-related dietary behaviour. Hong Kong Med J. 2016;22 Suppl 6:43-7. [PMID: 27807318]
41.
Chen R, Smyser M, Chan N, Ta M, Saelens BE, Krieger J. Changes in awareness and use of calorie information after mandatory menu labeling in restaurants in King County, Washington. Am J Public Health. 2015;105:546-53. [PMID: 25602868] doi: 10.2105/AJPH.2014.302262
42.
Thom G, Lean M. Is there an optimal diet for weight management and metabolic health? Gastroenterology. 2017;152:1739-51. [PMID: 28214525] doi: 10.1053/j.gastro.2017.01.056
43.
Porta M. A Dictionary of Epidemiology. 6th ed. New York: Oxford Univ Pr; 2014.
44.
Corriveau N, Eagle T, Jiang Q, Rogers R, Gurm R, Aaronson S, et al. Sustained benefit over four-year follow-up of Michigan's Project Healthy Schools. Am J Public Health. 2015;105:e19-25. [PMID: 26469650] doi: 10.2105/AJPH.2015.302835
45.
Folta SC, Kuder JF, Goldberg JP, Hyatt RR, Must A, Naumova EN, et al. Changes in diet and physical activity resulting from the Shape Up Somerville community intervention. BMC Pediatr. 2013;13:157. [PMID: 24093936] doi: 10.1186/1471-2431-13-157
46.
Institute of Education Sciences. What Works Clearinghouse: preview of regression discontinuity design standards. 2015. Accessed at https://ies.ed.gov/ncee/wwc/Document/258 on 29 January 2017.
47.
Bauhoff S. The effect of school district nutrition policies on dietary intake and overweight: a synthetic control approach. Econ Hum Biol. 2014;12:45-55. [PMID: 23891422] doi: 10.1016/j.ehb.2013.06.001
48.
Branas CC, Cheney RA, MacDonald JM, Tam VW, Jackson TD, Ten Have TR. A difference-in-differences analysis of health, safety, and greening vacant urban space. Am J Epidemiol. 2011;174:1296-306. [PMID: 22079788] doi: 10.1093/aje/kwr273
49.
Camacho-Rivera M, Rosenbaum E, Yama C, Chambers E. Low-income housing rental assistance, perceptions of neighborhood food environment, and dietary patterns among latino adults: the AHOME study. J Racial Ethn Health Disparities. 2017;4:346-53. [PMID: 27129854] doi: 10.1007/s40615-016-0234-z
50.
Chen C, Chou S-Y, Thornton RJ. The effect of household technology on weight and health outcomes among Chinese adults: evidence from China's ‘Home Appliances Going to the Countryside' policy. J Hum Cap. 2015;9:364-401.
51.
Cleland V, Dwyer T, Blizzard L, Venn A. The provision of compulsory school physical activity: associations with physical activity, fitness and overweight in childhood and twenty years later. Int J Behav Nutr Phys Act. 2008;5:14. [PMID: 18312621] doi: 10.1186/1479-5868-5-14
52.
Coffield JE, Metos JM, Utz RL, Waitzman NJ. A multivariate analysis of federally mandated school wellness policies on adolescent obesity. J Adolesc Health. 2011;49:363-70. [PMID: 21939866] doi: 10.1016/j.jadohealth.2011.01.010
53.
Cradock AL, McHugh A, Mont-Ferguson H, Grant L, Barrett JL, Wang YC, et al. Effect of school district policy change on consumption of sugar-sweetened beverages among high school students, Boston, Massachusetts, 2004-2006. Prev Chronic Dis. 2011;8:A74. [PMID: 21672398]
54.
Cummins S, Petticrew M, Higgins C, Findlay A, Sparks L. Large scale food retailing as an intervention for diet and health: quasi-experimental evaluation of a natural experiment. J Epidemiol Community Health. 2005;59:1035-40. [PMID: 16286490]
55.
Cummins S, Findlay A, Petticrew CHM, Sparks L, Thomson H. Reducing inequalities in health and diet: findings from a study on the impact of a food retail development. Environ Plan A. 2008;40:402-22.
56.
Datar A, Nicosia N. The effect of state competitive food and beverage regulations on childhood overweight and obesity. J Adolesc Health. 2017;60:520-7. [PMID: 27836530] doi: 10.1016/j.jadohealth.2016.09.003
57.
Fitzpatrick C, Datta GD, Henderson M, Gray-Donald K, Kestens Y, Barnett TA. School food environments associated with adiposity in Canadian children. Int J Obes (Lond). 2017;41:1005-10. [PMID: 28186100] doi: 10.1038/ijo.2017.39
58.
Frongillo EA, Fawcett SB, Ritchie LD, SoniaArteaga S, Loria CM, Pate RR, et al. Community policies and programs to prevent obesity and child adiposity. Am J Prev Med. 2017;53:576-83. [PMID: 28688728] doi: 10.1016/j.amepre.2017.05.006
59.
Giles-Corti B, Bull F, Knuiman M, McCormack G, Van Niel K, Timperio A, et al. The influence of urban design on neighbourhood walking following residential relocation: longitudinal results from the RESIDE study. Soc Sci Med. 2013;77:20-30. [PMID: 23206559] doi: 10.1016/j.socscimed.2012.10.016
60.
Hennessy E, Oh A, Agurs-Collins T, Chriqui JF, Mâsse LC, Moser RP, et al. State-level school competitive food and beverage laws are associated with children's weight status. J Sch Health. 2014;84:609-16. [PMID: 25117896] doi: 10.1111/josh.12181
61.
Hobin E, So J, Rosella L, Comte M, Manske S, McGavock J. Trajectories of objectively measured physical activity among secondary students in Canada in the context of a province-wide physical education policy: a longitudinal analysis. J Obes. 2014;2014:958645. [PMID: 24672714] doi: 10.1155/2014/958645
62.
Hobin E, Erickson T, Comte M, Zuo F, Pasha S, Murnaghan D, et al. Examining the impact of a province-wide physical education policy on secondary students' physical activity as a natural experiment. Int J Behav Nutr Phys Act. 2017;14:98. [PMID: 28724390] doi: 10.1186/s12966-017-0550-7
63.
Hoelscher D, Ory M, Dowdy D, Miao J, Atteberry H, Nichols D, et al. Effects of funding allocation for Safe Routes to School programs on active commuting to school and related behavioral, knowledge, and psychosocial outcomes: results from the Texas Childhood Obesity Prevention Policy Evaluation (T-COPPE) study. Environ Behav. 2016;48:210-29.
64.
Howlett E, Davis C, Burton S. From food desert to food oasis: the potential influence of food retailers on childhood obesity rates. J Bus Ethics. 2016;139:215-24.
65.
Hu Y, van Lenthe FJ, Judge K, Lahelma E, Costa G, de Gelder R, et al. Did the English strategy reduce inequalities in health? A difference-in-difference analysis comparing England with three other European countries. BMC Public Health. 2016;16:865. [PMID: 27558269] doi: 10.1186/s12889-016-3505-z
66.
Hughes RJ, Edwards KL, Clarke GP, Evans CE, Cade JE, Ransley JK. Childhood consumption of fruit and vegetables across England: a study of 2306 6-7-year-olds in 2007. Br J Nutr. 2012;108:733-42. [PMID: 22321148] doi: 10.1017/S0007114511005939
67.
Hunter S, Leatherdale ST, Storey K, Carson V. A quasi-experimental examination of how school-based physical activity changes impact secondary school student moderate- to vigorous- intensity physical activity over time in the COMPASS study. Int J Behav Nutr Phys Act. 2016;13:86. [PMID: 27473113] doi: 10.1186/s12966-016-0411-9
68.
Jia P, Li M, Xue H, Lu L, Xu F, Wang Y. School environment and policies, child eating behavior and overweight/obesity in urban China: the childhood obesity study in China megacities. Int J Obes (Lond). 2017;41:813-9. [PMID: 28074059] doi: 10.1038/ijo.2017.2
69.
Just DR, Wansink B. School lunch debit card payment systems are associated with lower nutrition and higher calories. Obesity (Silver Spring). 2014;22:24-6. [PMID: 23929600] doi: 10.1002/oby.20591
70.
Kern E, Chan NL, Fleming DW, Krieger JW; Centers for Disease Control and Prevention (CDC). Declines in student obesity prevalence associated with a prevention initiative—King County, Washington, 2012. MMWR Morb Mortal Wkly Rep. 2014;63:155-7. [PMID: 24553199]
71.
Kubik MY, Lytle LA, Story M. Schoolwide food practices are associated with body mass index in middle school students. Arch Pediatr Adolesc Med. 2005;159:1111-4. [PMID: 16330732]
72.
Lachapelle U, Frank LD. Transit and health: mode of transport, employer-sponsored public transit pass programs, and physical activity. J Public Health Policy. 2009;30 Suppl 1:S73-94. [PMID: 19190584] doi: 10.1057/jphp.2008.52
73.
MacDonald JM, Stokes RJ, Cohen DA, Kofner A, Ridgeway GK. The effect of light rail transit on body mass index and physical activity. Am J Prev Med. 2010;39:105-12. [PMID: 20621257] doi: 10.1016/j.amepre.2010.03.016
74.
Madsen KA. School-based body mass index screening and parent notification: a statewide natural experiment. Arch Pediatr Adolesc Med. 2011;165:987-92. [PMID: 21727262] doi: 10.1001/archpediatrics.2011.127
75.
Malakellis M, Hoare E, Sanigorski A, Crooks N, Allender S, Nichols M, et al. School-based systems change for obesity prevention in adolescents: outcomes of the Australian Capital Territory ‘It's Your Move!'. Aust N Z J Public Health. 2017;41:490-6. [PMID: 28749562] doi: 10.1111/1753-6405.12696
76.
Mâsse LC, de Niet-Fitzgerald JE, Watts AW, Naylor PJ, Saewyc EM. Associations between the school food environment, student consumption and body mass index of Canadian adolescents. Int J Behav Nutr Phys Act. 2014;11:29. [PMID: 24666770] doi: 10.1186/1479-5868-11-29
77.
Miller HJ, Tribby CP, Brown BB, Smith KR, Werner CM, Wolf J, et al. Public transit generates new physical activity: evidence from individual GPS and accelerometer data before and after light rail construction in a neighborhood of Salt Lake City, Utah, USA. Health Place. 2015;36:8-17. [PMID: 26340643] doi: 10.1016/j.healthplace.2015.08.005
78.
Nanney MS, MacLehose RF, Kubik MY, Davey CS, O'Connell MJ, Grannon KY, et al. School obesity prevention policies and practices in minnesota and student outcomes: a longitudinal cohort study. Am J Prev Med. 2016;51:656-63. [PMID: 27320703] doi: 10.1016/j.amepre.2016.05.008
79.
Nguyen BT, Powell LM. Supplemental nutrition assistance program participation and sugar-sweetened beverage consumption, overall and by source. Prev Med. 2015;81:82-6. [PMID: 26303370] doi: 10.1016/j.ypmed.2015.08.003
80.
Olsho LE, Payne GH, Walker DK, Baronberg S, Jernigan J, Abrami A. Impacts of a farmers' market incentive programme on fruit and vegetable access, purchase and consumption. Public Health Nutr. 2015;18:2712-21. [PMID: 25919225] doi: 10.1017/S1368980015001056
81.
Parsons WG, Garcia GM, Hoffman PK. Evaluating school wellness policy in curbing childhood obesity in Anchorage, Alaska. J Sch Nurs. 2014;30:324-31. [PMID: 24316497] doi: 10.1177/1059840513513155
82.
Peterson KE, Spadano-Gasbarro JL, Greaney ML, Austin SB, Mezgebu S, Hunt AT, et al. Three-year improvements in weight status and weight-related behaviors in middle school students: the Healthy Choices Study. PLoS One. 2015;10:e0134470. [PMID: 26295837] doi: 10.1371/journal.pone.0134470
83.
Powell LM, Chriqui J, Chaloupka FJ. Associations between state-level soda taxes and adolescent body mass index. J Adolesc Health. 2009;45:S57-63. [PMID: 19699437] doi: 10.1016/j.jadohealth.2009.03.003
84.
Quigg R, Reeder AI, Gray A, Holt A, Waters D. The effectiveness of a community playground intervention. J Urban Health. 2012;89:171-84. [PMID: 21959697] doi: 10.1007/s11524-011-9622-1
85.
Riis J, Grason H, Strobino D, Ahmed S, Minkovitz C. State school policies and youth obesity. Matern Child Health J. 2012;16 Suppl 1:S111-8. [PMID: 22527761] doi: 10.1007/s10995-012-1000-4
86.
Robles B, Montes CE, Nobari TZ, Wang MC, Kuo T. Dietary behaviors among public health center clients with electronic benefit transfer access at farmers' markets. J Acad Nutr Diet. 2017;117:58-68. [PMID: 27618576] doi: 10.1016/j.jand.2016.07.012
87.
Schwartz AE, Leardo M, Aneja S, Elbel B. Effect of a school-based water intervention on child body mass index and obesity. JAMA Pediatr. 2016;170:220-6. [PMID: 26784336] doi: 10.1001/jamapediatrics.2015.3778
88.
Sturm R, Powell LM, Chriqui JF, Chaloupka FJ. Soda taxes, soft drink consumption, and children's body mass index. Health Aff (Millwood). 2010;29:1052-8. [PMID: 20360173] doi: 10.1377/hlthaff.2009.0061
89.
Sturm R, Hattori A. Diet and obesity in Los Angeles County 2007-2012: is there a measurable effect of the 2008 “fast-food ban”? Soc Sci Med. 2015;133:205-11. [PMID: 25779774] doi: 10.1016/j.socscimed.2015.03.004
90.
Taber DR, Chriqui JF, Perna FM, Powell LM, Chaloupka FJ. Weight status among adolescents in states that govern competitive food nutrition content. Pediatrics. 2012;130:437-44. [PMID: 22891223] doi: 10.1542/peds.2011-3353
91.
Toussaint LL, Housholder K, Janssen K, Mansfield A, Lynch BA. Slowing BMI growth trajectories in elementary school-aged children: the northeast iowa food and fitness initiative. Fam Community Health. 2017;40:192-7. [PMID: 28525438] doi: 10.1097/FCH.0000000000000151
92.
Utter J, Denny S, Dyson B. School gardens and adolescent nutrition and BMI: results from a national, multilevel study. Prev Med. 2016;83:1-4. [PMID: 26657347] doi: 10.1016/j.ypmed.2015.11.022
93.
von Hippel PT, Bradbury WK. The effects of school physical education grants on obesity, fitness, and academic achievement. Prev Med. 2015;78:44-51. [PMID: 26163396] doi: 10.1016/j.ypmed.2015.06.011
94.
Webb E, Netuveli G, Millett C. Free bus passes, use of public transport and obesity among older people in England. J Epidemiol Community Health. 2012;66:176-80. [PMID: 21911850] doi: 10.1136/jech.2011.133165
95.
Wells L, Nelson M. The National School Fruit Scheme produces short-term but not longer-term increases in fruit consumption in primary school children. Br J Nutr. 2005;93:537-42. [PMID: 15946417]
96.
Woodward-Lopez G, Gosliner W, Samuels SE, Craypo L, Kao J, Crawford PB. Lessons learned from evaluations of California's statewide school nutrition standards. Am J Public Health. 2010;100:2137-45. [PMID: 20864696] doi: 10.2105/AJPH.2010.193490
97.
Ludwig J, Sanbonmatsu L, Gennetian L, Adam E, Duncan GJ, Katz LF, et al. Neighborhoods, obesity, and diabetes—a randomized social experiment. N Engl J Med. 2011;365:1509-19. [PMID: 22010917] doi: 10.1056/NEJMsa1103216
98.
Perry CL, Bishop DB, Taylor GL, Davis M, Story M, Gray C, et al. A randomized school trial of environmental strategies to encourage fruit and vegetable consumption among children. Health Educ Behav. 2004;31:65-76. [PMID: 14768658]
99.
Naylor PJ, Macdonald HM, Zebedee JA, Reed KE, McKay HA. Lessons learned from Action Schools! BC—an ‘active school' model to promote physical activity in elementary schools. J Sci Med Sport. 2006;9:413-23. [PMID: 16884957]
100.
Pope M. Preventing weight gain in children who are school age and African-American. Pediatr Phys Ther. 2016;28:207-16. [PMID: 26914717] doi: 10.1097/PEP.0000000000000243
101.
Rush E, McLennan S, Obolonkin V, Vandal AC, Hamlin M, Simmons D, et al. Project Energize: whole-region primary school nutrition and physical activity programme; evaluation of body size and fitness 5 years after the randomised controlled trial. Br J Nutr. 2014;111:363-71. [PMID: 23867069] doi: 10.1017/S0007114513002316
102.
Lorentzen C, Ommundsen Y, Jenum AK, Holme I. The “Romsås in Motion” community intervention: mediating effects of psychosocial factors on forward transition in the stages of change in physical activity. Health Educ Behav. 2009;36:348-65. [PMID: 18065570]
103.
Evans CE, Ransley JK, Christian MS, Greenwood DC, Thomas JD, Cade JE. A cluster-randomised controlled trial of a school-based fruit and vegetable intervention: Project Tomato. Public Health Nutr. 2013;16:1073-81. [PMID: 23237386] doi: 10.1017/S1368980012005290
104.
Cheadle A, Rauzon S, Spring R, Schwartz PM, Gee S, Gonzalez E, et al. Kaiser permanente's community health initiative in northern California: evaluation findings and lessons learned. Am J Health Promot. 2012;27:e59-68. [PMID: 23113787] doi: 10.4278/ajhp.111222-QUAN-462
105.
de Silva-Sanigorski A, Elea D, Bell C, Kremer P, Carpenter L, Nichols M, et al. Obesity prevention in the family day care setting: impact of the Romp & Chomp intervention on opportunities for children's physical activity and healthy eating. Child Care Health Dev. 2011;37:385-93. [PMID: 21276039] doi: 10.1111/j.1365-2214.2010.01205.x
Information & Authors
Information
Published In
Annals of Internal Medicine
Volume 168 • Number 11 • 5 June 2018
Pages: 791 - 800
History
Published online: 1 May 2018
Published in issue: 5 June 2018
Keywords
- Age groups
- Behavior
- Beverages
- Body weight
- Children
- Computer and information sciences
- Diagnostic medicine
- Diet
- Exercise
- Food
- Health care
- Information technology
- Nutrition
- Obesity
- Population statistics
- Prevention, policy, and public health
- Psychiatry and mental health
- Psychology
- Research and reporting methods
- Signs and symptoms
- Sports and exercise medicine
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. For an editable text file, please select Medlars format which will download as a .txt file. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
Download article citation data for:
Methods for Evaluating Natural Experiments in Obesity: A Systematic Review. Ann Intern Med.2018;168:791-800. [Epub 1 May 2018]. doi:10.7326/M18-0309
Methods for Evaluating Natural Experiments in Obesity: A Systematic Review. Ann Intern Med.2018;168:791-800. [Epub 1 May 2018]. doi:10.7326/M18-0309
View More
Get Access
Login Options:
Purchase
You will be redirected to acponline.org to sign-in to Annals to complete your purchase.
Create your Free Account
You will be redirected to acponline.org to create an account that will provide access to Annals.
Comments
0 Comments