Proposition Specific Protocols: The first protocol guides COP members through three stages of reviewing and evaluating evidence for a specific proposition: (1) the Initial Planning stage will identify the targeted proposition, ensure common understanding, and identify the types of relevant evidence that may need to be collected for the evaluation of the validity argument; (2) the validity review is the core work of the COP to review evidence for propositions’ claims and identify strengths and areas to improve; (3) during the Action Review COP members examine whether recommended changes were made or to reflect on how the discussion from the Validity Review impacted ongoing assessment development.

 

The COP members may decide to apply this protocol initially to some, none or all of the propositions associated with a specific type of assessment, depending on their familiarity with each of the propositions for the assessment on which they are focused. For example, the COP members may decide that they first wanted to better understand what it might mean to develop assessments that allow students to use multiple modalities. The group could use the protocol for Classroom Summative Proposition 2 and bring examples of assessments that either use multiple modalities, or that members think could be modified to incorporate multiple modalities. These discussions can help teachers to develop assessment literacy in collaborative ways. Other aspects of shared assessments are not discussed at this stage because the group elected to focus on a specific aspect.

 

Assessment Specific Protocols: Once the COP members have developed a level of familiarity with each proposition and its claims and evidence, they can use the protocol for the particular assessment (e.g., classroom summative). This protocol addresses all five propositions and supports a discussion about the chain of evidence that starts with the unit goals and ends with how the summative unit assessment will be used. This protocol will support COP discussions as the members articulate the chain of evidence for a specific assessment. The COP may need to take some time to decide what of the possible forms of evidence they are going to collect. For instance, will they survey students? Will they engage in peer teacher review? Careful planning will be needed prior to collecting evidence for review.

 

When the evidence has been amassed, the purpose of the discussion is to identify places where the evidence might not support a specific claim, or where an entire proposition cannot be supported. The COP members can then decide either to discuss how to improve the assessment, or decide when that work could be done. It is important to capture sufficiently detailed notes about the decisions that could assist future revisions if they are not going to be made immediately.

 

COP Discussions Using Both Types of Protocols: The entire COP discussion is intended to be formative: the logic model and propositions describe the ideal nature and conditions of an assessment. The evidence describes the current status of the assessment, and the COP discussions examine the gap between intentions and reality. Lessons learned from a review of one assessment can be applied to future assessments. The COP members might decide to focus their efforts on revising an upcoming assessment and then examine it in terms of the validity evidence to determine whether the revisions helped them get closer to the ideal model as described by the propositions. The COP might also decide to return to the elaborated protocol because they identified a specific proposition as an area of weakness across their assessments and want to work on that area in a targeted way.

 

It is important to note that it would not be practical or even advisable that a COP engage in this level of review for every classroom assessment. An in-depth review of one or two per semester may be sufficient, if individual and COP time is also spent on revising other assessments that may have the same weaknesses.