Part 9: Validity of the System

​

In addition to collecting and evaluating evidence for the validity of interpretations and uses of the information yielded by the individual assessments, judgments about how well the assessments work together as a system to accomplish systemic goals will also be necessary. Common goals for an assessment system might include (1) supporting all students to master specific skills and knowledge deemed valuable and necessary for success after graduation, (2) minimizing achievement gaps or differences among groups who are believed to have the same underlying ability distribution, and (3) meeting external requirements or standards related to accountability or equity, for example. Specific additional and central goals for this CAS are for English Learners in secondary grades to have rigorous and engaging learning opportunities, and for teachers of these students to participate with skill and confidence in the practice of designing, interpreting, and validating assessments that provide meaningful information about English Learners’ knowledge, skills, and powerful use of language.

 

The CAS Framework lays out a roadmap for changing assessments in the system, and it is unlikely that all assessments will simultaneously undergo a validation process. An important component of this long-term undertaking is that as the validity evidence for each assessment is evaluated, the entire system should be kept in review in terms of the degree to which it exhibits the three properties of an assessment system: comprehensiveness, coherence, and continuity (3Cs) discussed at the beginning of the document. Such a review may result in changes to the assessments therein, in order to ensure that they all emanate from the same set of goals related to language and content learning, communicate a shared vision of what is important for students to know and be able to do with content and language, and push teaching and learning in a common direction so that the demands of understanding concepts, practices and relationships are not privileged above the demands of linguistic resources, nor vice versa.

 

Some questions to guide a review of the CAS in relation to the 3Cs include,

 

• Are all the assessments aligned to the complementary learning goals (of different grain sizes depending on the assessment)?

• Do the assessments reflect the same model of teaching content and analytic learning and the development of language resources needed to learn that content to English Learners?

• Is this model of teaching clear and consistent at all levels (e.g., formative assessment to classroom summative, to external assessment)

• Do the assessments at all levels reflect the same conceptualization of how students learn content and language simultaneously?

• Do the assessments provide various ways for English Learners to demonstrate their competence in content and language?

• Are the assessments linked conceptually so that change over time can be observed and interpreted?

 

These questions are not intended as a complete set but rather as starting points to consider system validity.

 

The key point about the system is that combination of assessments should provide mutually complementary views of student learning that together reinforce important goals and teaching and learning practices while strengthening the validity of the system as a whole (Herman, 2010).

 

Policy Implications

 

Implementing this inverted assessment system has policy implications, which are outside the scope of this document, but will need to be attended to, nevertheless. These include addressing the current system-wide distortions caused by the punitive use of accountability results for teachers and schools, and also fundamentally changing the nature of accountability. A common argument for heavily privileging large-scale standardized assessments is their relative cost and reliability compared to more person-oriented approaches such as those represented in the framework. While it is true that machine-scored items may be cheaper to purchase, score, and administer than portfolios, it is also true that such instruments have a direct – and often negative – impact on what teachers do in the classroom and what English Learners have an opportunity to learn. Multiple choice assessments, portfolios, and any kind of assessment, do not operate in a vacuum; they are administered and used in the larger context of teaching and learning, and their costs and benefits should be evaluated in that larger context. The CAS Framework is intended to show the many benefits of implementing an assessment system that prioritizes quality teaching and learning that will lead to successful outcomes for every English Learner.