A Vision for Using an Argument-Based Framework for Validity Applied to a Comprehensive System of Assessments for English Learners in Secondary Grades

By Margaret Heritage, Caroline Wylie, Molly Faulkner-Bond, and Aída Walqui

Part 5: The CAS Framework

The CAS Framework is intended to reflect the nature of the learning contexts and the students’ backgrounds in assessment to support equal opportunities to learn and to achieve for secondary English Learners. The CAS Framework is based on the following assessment principles:

Focuses on the learner and learning: Assessment provides insights into each student’s thinking, skills, and language development. Assessment is aligned to high-quality classroom learning, consistent with Table 1, provides all students with the opportunity to show where they are in their learning through multiple modalities, and reflects meaningful, worthwhile tasks that challenge the upper reaches of students’ language competence and conceptual understanding.
Emphasizes rigorous learning: Assessment focuses on the concepts, knowledge, language, and analytical practices inherent in academic content standards. Assessment reflects high-quality classroom learning experiences characterized by apprenticeship, interaction, and scaffolding within the students’ zone of proximal development (ZPD) that promote deep, interconnected understandings and the language to express them.
Produces actionable information for the user: Information is asset- and future-oriented, focusing on what students can do in terms of content and language as well as highlighting areas of need and potential growth. Information yielded is tractable for teaching and learning.
Supports metacognition and self-regulation: Assessment provides information that supports the ongoing development of students’ metacognitive thinking about their learning (both their thinking processes and their language use), their achievement, and their approaches to learning, which in turn enables them to proactively orient their actions to achieving goals.
Promotes self-efficacy and learner identity: Assessment is designed with multiple entry points so that all students are able to show what they know and what they can do with language, giving students a sense of accomplishment and helping them enhance their feelings of self-efficacy and build positive learner identities.

Overview of Framework

Just as a one-size-fits all pedagogy does not meet the learning needs of every English Learner, no single assessment can accomplish all assessment purposes for English Learners. The CAS comprises a range of assessments (formative assessment, end-of-unit classroom assessment, end-of-year classroom assessment, and student portfolios). The CAS’s aim is to provide decision-makers with the information they need to support English Learners in the secondary grades to engage in challenging and rigorous content learning required to meet state standards. Table 1 shows the potential users, purposes, and proposed methods of assessment within the CAS Framework.

Table 1. Assessments and Their Uses in a CAS

Three Properties of the System

To optimize the credibility and utility of the resulting information, an assessment system should exhibit three properties: comprehensiveness, coherence, and continuity (National Research Council [NRC], 2001).

Comprehensiveness means that a range of approaches are used to provide a variety of evidence to support educational decision- making (NRC, 2001). Coherence relative to the framework has two dimensions: (1) the alignment among learning goals, instruction, and assessment so that all three are moving in the same direction (horizontal coherence), and (2) the extent to which learning goals, instruction, and assessment are continually intertwined over time to promote student progress (developmental coherence) (Herman, 2010). Continuity refers to the need for the system to assess student progress over time, “akin more to a video recording than to snapshots provided by a system of on-demand tests” (NRC, 2001, p. 257).

The CAS Framework is designed to reflect these three properties. For example, the framework supports comprehensiveness by including assessments with different grain-sizes and purposes, by emphasizing horizontal coherence among learning goals, instruction, and assessment approaches, along with the role of learning trajectories to support vertical coherence, and by attending to continuity with the inclusion of the collection and review of evidence in a student portfolio.

Soundness of Assessment Information

There are two important concepts that concern the soundness of the information from any assessment for decision-making: validity and reliability.

Validity

Does the assessment information lead to sound interpretations, decisions, or actions that enhance learning for secondary-grade English Learners? (Moss, Girard, & Haniford, 2006.). The answer to this question depends on the validity of the inferences made from test scores for a given purpose. The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014) refers to validity as the “degree to which evidence and theory support the interpretation of test scores for proposed uses of tests” (p. 11), and validation as a “process of constructing and evaluating arguments for and against the intended interpretation of test scores and their relevance to the proposed uses” (p. 11). Significant efforts are generally expended to collect validity evidence for assessments that are used to inform high-stakes decisions such as certification or placement decisions. Certainly, assessments that have significant consequences for students should have a high degree of validity (i.e., they require more evidence) to support their use. However, as we shall see in the CAS Framework, validity is important for all assessment uses, including the use of questions teachers ask during formative assessment.

Validity is not a property of the assessment but of the inferences made from assessment results and the extent to which they justify the use of an assessment for a specific purpose. For instance, scores from a mathematics assessment may have a strong degree of validity for mathematics problem solving, but a weak degree of validity for students’ computational fluency. In formative assessment, for example, a concern about validity would lead a teacher to explore whether the questions, tasks, and activities that she uses to elicit evidence of student thinking are aligned with the learning goals of the lesson, and whether they generate student responses that provide insights into their conceptual understanding, their use of analytical practices, and their language development.

Reliability, a necessary but not sufficient condition for validity, refers to the consistency of assessment results across settings, students, and users. Appendix A provides a discussion of reliability in relation to the assessments in the CAS Framework.

Since the CAS represents a reimagining of what is possible, not a reordering of what currently is, there are types of assessment (e.g., district-administered, off-the-shelf assessments) that are not explicitly included in the framework. In the reimagined system, such instruments would not be necessary.