Evaluate the Impact of 21st Century Community Learning Centers

Over the last decade the 21st Century Community Learning Center program (21st CCLC) funded by the U.S. Department of Education has served over 100,000 Texas students annually in after-school centers. Approximately 779 local Texas 21st CCLC programs served students from over 800 schools during the 2012-13, and/or 2013-14 academic years. Statewide evaluations of the impact of 21st CCLCs, which typically show small or mixed academic impacts of the program in aggregate, may serve to obscure the success of specific centers. To explore this, Westat examined individual grade-level impacts for the 779 local programs which served students during the 2012-13 and/or 2013-14 academic years. Propensity-score matching was used to create comparison groups of non-21st CCLC students for each grade-level, for each of the centers, for each of three outcomes: school attendance, STAAR Mathematics, and STAAR Reading. The distribution of effect sizes for each STAAR outcome centered on approximately d = ≈ -0.02 (no impact), whereas the median impact for attendance was small (d = 0.14). These findings are similar to a recent statewide evaluation of Texas 21st CCLC where students were pooled together across all centers. i However, in this study, individual centers were identified with both large negative and positive impacts. Westat identified several centers with large positive impacts that replicated in both academic years examined. Subsequent interviews completed with some of these centers revealed robust academic tutoring and supports in place. Results show that a large-scale analytic approach can help identify individual centers that are implementing promising practices that may be worthy of more rigorous evaluation and scaling. The 21st Century Community Learning Center program (21st CCLC) funded by the U.S. Department of Education has served over 100,000 Texas students in after-school programs annually for the last several years. Approximately 779 local Texas 21st CLC programs served students from over 800 schools during the 2012-13, and/or 2013-14 academic years. Federal funding for these programs totaled approximately $101.6 million dollars in Texas in 2013.ii The programs represent an important component in Texas’ education landscape. Each program offers a locally determined mix of student supports, ranging from sports and arts activities to academically focused tutoring. Annual program evaluations are required, but are typically not focused on causal impacts; over 95% of the local 21st Century CCLC program evaluations had not used a comparison group of non-21st CCLC students to evaluate impacts. Westat/Edvance was the Texas Education Agency’s (TEAs) technical assistance provider for the 21st CLCs during this period. Westat evaluated each of the 779 CLCs individually by conducting a matched-comparison evaluation. Propensity score matching was used to create a control-group for each of the sites using data from the UT ERC. The impact of student participation in a CLC was estimated separately for attendance, and mathematics and reading assessment (STAAR) outcomes. When aggregated across sites, the overall impact statewide was similar to those found in other evaluations.i However evaluation of impacts over two consecutive years revealed a number of Texas 21st CCLC programs with positive academic outcomes that repeated suggesting real impacts and not simply single events, observed possibly due to chance. Education Research Center www.texaserc.utexas.edu POLICY BRIEF Page | 2 Meta-analysis of after-school programs have identified either small or inconsistent impacts on student academic outcomes.iii,iv,v This has led somevi to call for an end to funding of the entire 21st CCLC program by Congress as being ineffective. However, large scale evaluations are designed to estimate aggregate outcomes and have ignored likely variability in impacts across specific 21st CCLC centers. These may represent programs with effective student support models that could be scaled in the future. In Texas, 21st CCLC programs have extensive latitude (within general guidelines) to provide different services aligned to the needs of their local schools and their populations. For example one program may provide afterschool mathematics tutoring, another may emphasize student engagement through sports activities. Each program therefore represents a natural experiment, where program offerings are varied across varied student populations and contexts. This evaluation explored impacts across this large pool of experimental conditions to identify programs where an intensive review of the program’s offerings may result in It is likely that the influence of 21st CCLC programming on student academic outcomes varies across sites, and that sites with particularly effective academically-oriented programming may exhibit higher student academic, as compared to students.