Since 2007, the Oklahoma State Department of Education has operated a quality improvement system (QIS) for its approximately 100 federally-funded 21st Century Community Learning Centers (OK 21CCLC) afterschool programs with the explicit purpose of improving the performance of these service providers. This report draws upon data from 23 performance measures collected annually over multiple annual program cycles to present findings for reliability, validity, performance change, and effect of intervention fidelity on performance change. These analyses were conducted as part of an ongoing effort to: (a) evaluate over-time change in performance that is the central purpose of the QIS and (b) improve the accuracy and usefulness of performance data available to individual organizations that participate in the QIS.
In general, our findings indicate that the Oklahoma Afterschool Improvement Process is performing in accordance with its purposes: using accurate performance data to incentivize improvement in the quality of services.
Findings for the reliability and validity of the measures include:
- All of the 23 measures demonstrated acceptable levels of reliability.
- There is evidence for construct validity at each time point and factorial invariance across time points.
Findings for performance improvement include:
- Nearly all measures incrementally improved during a four year (2010-2013) period, while a subset demonstrated statistically significant growth.
- For nearly all measures, lower-performing sites at the baseline year (2010-2011) improved most. A subset of models demonstrated statistically significant effects.
- The indicator with the largest increase over four years was Targeting At-Risk Students, suggesting that even though the students served became more challenging, service quality was also generally improving.
Findings for intervention fidelity include:
- Higher fidelity of YPQI implementation is positively associated with growth on nearly all performance measures at over half of all year-to-year time increments, in line with the YPQI theory of change
This report is supplement to a series of annual reports submitted to the Oklahoma State Department of Education over eight years. These reports provide the unadjusted information that was used in the models described in this report. The supplement to the annual performance report for the 2013-14 program year (Sniegowski, Gersh, Smith, & Garner, 2015) provides the unadjusted means and descriptive statistics for all of the items and scales in the study.