This quality-outcomes study was designed to both (a) describe performance in Seattle Public Schools (SPS) summer learning programs in ways that are useful to staff and (b) provide evaluative evidence (i.e., validity) for an instructional model that includes challenging academic content and responsive instructional practices.
Results from this study were mainly positive yet partially ambiguous. Summer program offerings were well-attended and characterized by high-quality instructional practices, with a majority of students increasing their literacy and math skills during the program. Findings about the association between exposure to more responsiveness instruction (e.g., quality) and academic skill change were mixed.
Results include:
Positive academic skill change was found in the raw data, including for academically at-risk students. Positive change on the academic performance measures used during the summer program was found for 73% of students, and positive change on the academic achievement tests was found for 74% of students from the 2015 to 2016 school year. Standardized effect sizes for the full sample ranged from medium to large (i.e., dz = .56 – .95) across the seven academic skill measures.
Attendance was regular, and instructional responsiveness was consistently high. Summer program attendance for 21 or more days (out of a total possible 27 days) was observed for 77% of students. Analysis of instructional responsiveness using the Summer Learning PQA revealed three profiles of instructional responsiveness at the point of service: high, medium, and low quality. However, compared to other urban samples, the “low” SPS profile is not very low.
Students in SPS summer programs had similar rates of skill change across profiles of instructional responsiveness in the most rigorous models for 3rd and 4th grade students (N = 535); that is, there was insufficient evidence in support of the hypothesized pattern of differential skill change across profiles of instructional quality. However, these results should be interpreted with caution due to the absence of a true low-quality instructional practices subgroup in the sample. Less statistically rigorous but more theoretically well-specified models for the entire K-4 sample (N = 1060) revealed a positive association between instructional quality and academic skill change, despite the lack of a true low-quality subgroup.
Analyses of academically at-risk students revealed similarly mixed results. In the more statistically rigorous models with grades 3-4, students who entered SPS summer programs below proficient on academic achievement tests for the prior school year (2015-16) showed similar rates of academic skill change across profiles of instruction. In the theoretically well-specified models, academically at-risk students showed greater changes in academic skills in summer programs with higher-quality instructional practices.