Original Validation of the Youth Program Quality Assessment (Youth PQA)

Summary

The Youth Program Quality Assessment (PQA) is an assessment of best practices in afterschool programs, community organizations, schools, summer programs, and other places where you have fun, work, and learn with adults. The Youth PQA creates understanding and accountability focused on the point of service — where youth and adults come together to coproduce developmental experiences. The ultimate purposes of the Youth PQA are empowering staff to envision optimal programming and building motivation of youth to participate and engage. As an approach to assessment at the systems level, the Youth PQA links accountability to equity by focusing on access to high-quality learning environments for all youth who enroll. As a research tool, the Youth PQA improves measurement of instructional process in places where young people learn.

The Youth PQA consists of seven sections or subscales, each bearing on one dimension of program quality critical for positive youth development: safe environment, supportive environment, interaction, engagement, youth-centered policies and practices, high expectations, and access. Administration of the Youth PQA employs direct observation of youth program activities for its first four sections and a structured interview with a program director for its remaining three sections. The instrument can be used by outside observers to produce the most precise data or as a program self-assessment directed toward generation of rich conversations among staff.

The Youth PQA Validation Study was a 4-year effort to develop and validate a tool to assess program quality in youth settings. Through the process of instrument development, dozens of expert practitioners and researchers were brought together to provide input on the tool. In total, the validation study encompassed 59 organizations in Michigan and more than 300 Youth PQA observations and interviews conducted in programs serving 1,635 youth. Most of these youth programs were afterschool programs that met weekly or daily over several months. The average age of youth in the sample was 14 years, and more than half were attending programs in an urban context.

The Youth PQA Validation Study employed multiple, independent data sources, including interviews with program administrators, observations in youth work settings, surveys of program youth, expert opinions, and verified reports of staff training. The study’s primary concurrent measure of program quality was the Youth Survey from Youth Development Strategies, Inc. All Youth Survey data were independently collected and prepared for analysis by Youth Development Strategies, Inc.

In general, findings from the study demonstrate that the Youth PQA is a valid, reliable, and highly usable measure of youth program quality. Principle findings include:

1. The Youth PQA measurement rubrics are well calibrated for use in a wide range of youth serving organizations. Average scores fall near the center of the scale and are spread across all five scale points.
2. Pairs of data collectors were able to achieve acceptable levels of inter-rater reliability on most of the Youth PQA’s measurement constructs.
3. The Youth PQA scales subscales are reliable measures of several dimensions of quality. Key subscales demonstrated acceptable levels of internal consistency in two samples.
4. The Youth PQA can be used to assess specific components of programs and is not just a single global quality rating. In repeated factor analyses on two waves of Youth PQA data, the subscales were validated as separate, distinguishable constructs.
5. Youth PQA quality ratings reflect youth reports about their own experiences in the same program offerings. Youth PQA scores demonstrate concurrent validity through significant positive correlation with aligned scores from the YDSI Youth Survey.
6. The Youth PQA measures dimensions of quality that are related to positive outcomes for youth such as youth sense of challenge and growth from the youth program. Youth PQA scores demonstrate predictive validity in multivariate and multilevel models of the data, controlling for youth background variables.
7. Staff in 21st Century afterschool programs find the instrument to have face validity, to be applicable to their current work, and to be a foundation for purposeful change in their programs.