Quality in the Out-of-School Time Sector: Insights from the Youth PQA Validation Study

From the Introduction:

Over the last decade the High/Scope Educational Research Foundation has developed and validated an observational assessment instrument for out-of-school time (OST) programs, the Youth Program Quality Assessment, and several methodologies for its use. This experience has been instrumental in shaping our ideas about what program quality is, how it works, and how OST organizations can consistently produce it. There is much discussion in the field of youth development about the nature and effects of program quality—even arguably a rough consensus on program practices and elements that define quality in youth development settings. However, there is less guidance available regarding the relative importance of specific quality practices, how to know if a program is producing them well enough, and perhaps most importantly, how these elements of quality can be intentionally applied to improve OST settings. This article attempts to join what we have learned about program quality in OST programs—what counts and how best to see it—to a framework that describes organizational structure and change. We hope that this effort to will inform setting-level intervention, improvement and accountability work in the OST field.

After collecting hundreds of structured observations in a wide variety of youth work settings, we frame the issue like this: a high quality program provides youth with access to key experiences that advance adaptive, developmental and learning outcomes. However, OST organizations frequently miss opportunities to provide these key experiences for the youth who attend them. This is an area of systemic underperformance because these missed opportunities occur due to existing structures, practices and policies across the OST sector. The areas of underperformance can be identified, described and assessed through two setting-level constructs: quality at the point of service (POS quality) and quality of the professional learning community (PLC quality). POS occurs where youth, staff, and resources come together, and POS quality involves both (1) the delivery of key developmental experiences and (2) the level of access participating youth have to these experiences. In high quality programs, the PLC exists primarily to build and sustain the POS. PLC quality, as we construe it, is primarily focused on (1) the role of supervisors as human resource mangers and (2) the creation of knowledge management systems that facilitate the translation of program data/information into plans for action related to POS quality. Our ideas about the elements of quality that make up POS and PLC constructs are not new. What is important is how these setting-level constructs allow us to see quality more clearly and in ways that are linked to structure and change dynamics in OST organizations.

In [this paper] we examine in turn, a diagram about how the PLC and POS settings occur in organizations, a theory of dynamics that influence these settings (make quality higher or lower), and review the contents and psychometric characteristics of our primary POS quality measure, the Youth Program Quality Assessment (Youth PQA). With these pieces in hand, we then move to the task of defining the empirical context of POS and PLC quality across a wide range of OST settings.