A Vision for Using an Argument-Based Framework for Validity Applied to a Comprehensive System of Assessments for English Learners in Secondary Grades

By Margaret Heritage, Caroline Wylie, Molly Faulkner-Bond, and Aída Walqui

Part 7: Using a Community of Practice Approach to Evaluate the Validity Argument for Classroom-Based Assessment

For high-stakes assessment uses such as accountability, graduation, or certification validity, evidence is often gathered through a series of rigorous studies that may involve convening large groups of subject matter experts, or by conducting complex statistical analyses. The outcomes of these efforts are typically written into a technical report, which may be posted publicly or shared with governing bodies such as the federal government or a certifying organization. Most testing programs also retain a committee of technical advisors who review their validation plans and results to provide feedback throughout the process.

However, because the CAS Framework centers classroom-based assessment at its core, a different approach to the collection and evaluation of validity evaluation is needed. Rather than being undertaken by external subject matter experts or psychometricians, we propose that an evaluation of validity evidence for classroom-based assessment is conducted in a teachers’ community of practice (COP).

The accumulation and evaluation of validity evidence is an iterative and educative process. Repeated cycles of assessment review provide opportunities for teachers to increase their assessment literacy; to deepen their knowledge of simultaneous language and content development and their understanding of quality in the context of classroom assessment; to improve their analysis, interpretation, and application of assessment information to support student learning; to make improvements to an assessment question, task, or activity for future use; and to enhance the quality of newly developed assessment questions, tasks, or activities.

Community of Practice refers to a group of teachers within a school who meet regularly to tackle a particular problem of practice and find solutions.

Educative refers to the teacher learning that arises from engaging in with peers in discussions about classroom assessment, standards, and pedagogy.

This increased role is not intended to place undue burden on teachers who will need time and support to engage in this work. Teachers are not expected to address all their assessments simultaneously. Rather, the framework lays out a roadmap for changing assessments in the system to maximally benefit English Learners’ development. Implementing the framework should be thought of as a long-term undertaking. The COP envisioned here might replace (but function similarly to) current practices such as data discussions or meetings in which teacher teams meet to review assessment data and plan instruction.

Sources of Evidence for Validity Evaluation

A variety of evidence sources is needed for the validity evaluation in the COP to determine the strength of the evidence in support of an assessment’s use.

Evidence in the CAS Framework falls in one of six categories, generally:

Documentation related to assessment development and/or administration of the assessment (e.g., learning goals presented to students, directions for portfolio selection)
Individual teacher reflection on specific aspects of the assessment (e.g., teacher reflection on whether the questions, tasks, and activities are accessible to the range of students’ zone of proximal development (ZPD) present within the class)
Peer observation of assessment processes or supporting processes (e.g., how effectively teachers communicate learning goals to students, observation and analysis of student-to-student or student-to-teacher interactions)
Student feedback about assessment (e.g., survey or interviews)
Peer feedback on an assessment claim (e.g., peer review and feedback on the alignment between the breadth and depth of cognitive complexity and language usage represented by the unit goals and the evidence selected by students)

In the sections that follow, the types of supporting evidence are described more specifically in the context of the related claims.

To download a brief overview of Using a Community of Practice Approach to Evaluate the Validity Argument
for Classroom-Based Assessment click here.