The National Academies Press

Currently Skimming:

Pages 50-70

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 50... ... 50 4 Quality Standards for Performance Assessments Standards for educational achievement have been developed that de-lineate the values and desired outcomes of educational programs inways that are both transparent to stakeholders and provide guidance for curriculum development, instruction, and assessment. In addition, as described in Chapter 3, the measurement profession has developed a set of standards for the quality control of educational assessments. Read the entire page →
From page 51... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 51 poses, two of which -- accountability and instruction -- are particularly relevant to this report. As noted by several participants at the workshop, these two purposes are not always compatible, as they are concerned with different kinds of decisions and with collecting different kinds of information. Read the entire page →
From page 52... ... 52 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION of useful performance assessments for the purpose of accountability across programs and across states because that is what the National Reporting System (NRS) requires. Read the entire page →
From page 53... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 53 to achieve these standards. Thus, in any specific assessment situation, there are inevitable trade-offs in allocating resources so as to optimize the desired balance among the qualities. Read the entire page →
From page 54... ... 54 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION • What reliability issues are of concern in this assessment? • What are the potential sources and kinds of error in this assessment? Read the entire page →
From page 55... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 55 Identifying Potential Sources and Kinds of Error in Performance Assessment Because most performance assessments include several different facets of measurement (e.g., tasks, forms, raters, occasions) , a logical analysis of the potential sources of inconsistency or measurement error should be made in order to ascertain the kinds of data that need to be collected. Read the entire page →
From page 56... ... 56 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION will be averaged out across students. Thus, when decisions about programs are based on group average scores, higher levels of reliability can be expected than would be typically obtained from the individual scores upon which the group averages are based. Read the entire page →
From page 57... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 57 Estimating the Reliability of Scores There is a wide range of well-defined approaches to estimating the reliability of assessments, both for individuals and for groups; these are discussed in general in the Standards, while detailed procedures can be found in measurement textbooks (e.g., Crocker and Algina, 1986; Linn et al., 1999; Nitko, 2001) Read the entire page →
From page 58... ... 58 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION procedures, clear and understandable scoring procedures and criteria, and sufficient and effective training and monitoring of raters. The training of raters may have an additional benefit -- it may tie in with professional development for teachers in adult education programs. Read the entire page →
From page 59... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 59 for supporting all kinds of claims or for supporting a given claim for all times, situations, and groups of test takers. The Standards discusses the following sources of evidence that support a validation argument: • Evidence based on test content. Read the entire page →
From page 60... ... 60 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION ment can also be collected in this way. In this context, for example, accountability requirements may well impede program functioning, or they may conflict with client goals. Read the entire page →
From page 61... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 61 version of the test he or she receives. Equating is carried out routinely for new versions of large-scale standardized assessments. Read the entire page →
From page 62... ... 62 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION perts, common standards, and exemplars of performance that are aligned to these standards. Social moderation replaces the statistical and measurement requirements of the previous approaches with consensus among experts on common standards and on exemplars of performance. Read the entire page →
From page 63... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 63 Linn (1993) provides examples of uses of social moderation that are relevant to the context of accountability assessment in adult education, while Mislevy (1995) Read the entire page →
From page 64... ... 64 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION ment. First, the NRS is essentially an ordinal scale2 that breaks up what is, in fact, a continuum of proficiency into six levels that are not necessarily evenly spaced. Read the entire page →
From page 65... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 65 An additional concern is that the kinds of performance assessments that might be envisioned may be even less sensitive to tracking small developmental increments than some assessments already being used. Performance assessment tasks tend to be more cognitively demanding, educationally relevant, and authentic to real-life situations, which means they are not usually designed to focus either on small increments or on the component skills and abilities that may contribute to successful performance on the task as a whole. Read the entire page →
From page 66... ... 66 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION tical to try to obtain a random sample of adults to attend adult education classes. Second, if the adult education classes included students who were randomly selected rather than people who had chosen to take the classes, there would be major consequences for the ways in which the adult education classes were taught. Read the entire page →
From page 67... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 67 Equitable Treatment in the Testing Process All test takers should be given a comparable opportunity to demonstrate their level on the skills and knowledge measured by the assessment (NRC, 1999b) Read the entire page →
From page 68... ... 68 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION ing both types of low scores as if they mean the same thing is fundamentally unfair. In the context of adult literacy, where there are extreme variations in the amount of time individual students attend class (e.g., 31 hours per student per year in the 10 states with the lowest average and up to 106 hours per student among the 10 states with the highest average) Read the entire page →
From page 69... ... QUALITY STANDARDS FOR PERFORMANCE ASSESSMENTS 69 ment, the assessment can be said to be practical or feasible. Practicality concerns the adequacy of resources and how these are allocated in the design, development, and use of assessments. Read the entire page →
From page 70... ... 70 PERFORMANCE ASSESSMENTS FOR ADULT EDUCATION mance levels. Bickerton noted that it could take up to double the 150 hours mentioned above to complete one NRS level for students who, on average, are receiving instruction for a total of just 66 to 86 hours (DOEd, 2001c) Read the entire page →

From page 50...

... 50 4 Quality Standards for Performance Assessments Standards for educational achievement have been developed that de-lineate the values and desired outcomes of educational programs inways that are both transparent to stakeholders and provide guidance for curriculum development, instruction, and assessment. In addition, as described in Chapter 3, the measurement profession has developed a set of standards for the quality control of educational assessments.