National Academies Press: OpenBook
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

NATIONAL RESEARCH COUNCIL

CENTER FOR EDUCATION

2101 Constitution Avenue Washington, DC 20418

(202) 334-3087 FAX: (202) 334-2210

June 5, 2000

Norma Cantu

Assistant Secretary for the Office for Civil Rights

U.S. Department of Education Mary E. Switzer Building 330 C Street, SW Washington, DC 20202

Dear Ms. Cantu:

This letter report responds to your office's request for a review and critique of “The Use of Tests When Making High-Stakes Decisions for Students: A Resource Guide for Educators and Policymakers” (hereafter referred to as the Guide). The Guide has been revised substantially since the Board on Testing and Assessment (BOTA) issued an earlier letter report about it, in June 1996, but its purpose remains unchanged. As you note in your transmittal letter for the Guide, it is intended to provide “information about the professional standards relating to the use of tests for high-stakes purposes, the relevant federal laws that apply to such practices, and references that can help shape educationally sound and legally sufficient testing practices.”

We note that the Guide was initiated because the Office for Civil Rights (OCR) identified the issue of fairness in testing and assessment as a high priority and because of the ever-increasing number of complaints involving test fairness that your investigators are called on to review each year. The OCR has solicited comment on the Guide by arranging a series of public meetings around the country and by conferring with professional societies, test publishers, scholars, education officials, and others in order to prepare a document that serves its stated purpose of assisting those responsible for complying with federal law regarding fairness in testing. The Guide has gone through a number of drafts in response to the comments the OCR has received; this letter report addresses the draft dated April 2000.

We begin our review by commending you and your staff both for the substantial improvements in the Guide and for the thorough process through which you have sought comment on its form and content from the public and from interested professionals and scholars. We appreciate the opportunity to review this version and hope that our comments will be helpful as you continue to work on this document.

The National Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering to serve government and other organizations

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

In our review, we defined our task as an analysis of the extent to which the April draft of the Guide comports with professional standards. As part of its review of the document, BOTA arranged a hearing (in January 2000) through which it could seek more information on the views of parties who had commented on earlier drafts. BOTA also reviewed the version of the Guide supplied in January and the comments made about it in order to fully understand the opinions of those who responded to earlier drafts. As we and many of the participants in our January hearing noted, the Guide has been much improved.

This report comments on three aspects of the Guide: its clarity, technical issues, and legal issues.

Clarity

In general, the Guide reads well, and its structure has been improved over previous drafts. Our principal comment on its clarity relates to the use of the Standards for Educational and Psychological Testing (American Educational Research Association, 1999). The Guide relies heavily on the authority of the Standards, and doing so is clearly warranted. However, the role of the Standards is not properly explained and, in some cases, relevant standards are omitted from discussion or paraphrased in the text in a way that slightly changes their meaning.

We recommend two ways to correct these problems. First, early in the text, the Guide should explain why the Standards are viewed as the primary authority on technical issues and how they are used to support the points that are made in the Guide. Such an explanation would be particularly useful to readers who may not be familiar with the Standards and their provenance. In particular, the Guide should explain that the Standards were developed and revised by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education through a process that involved the participation of hundreds of testing professionals and thousands of pages of written comment from both professionals and the public. The current edition reflects the experience gained from many years of wide use of previous versions of the Standards in the testing community.

A related point is that the Guide's use of other sources should be explained. In particular, a distinction should be made between reliance on the Standards, which are judged to represent the primary statement of professional consensus regarding educational testing, and other documents cited, such as High Stakes: Testing for Tracking, Promotion, and Graduation (National Research Council, 1999), which are used to help explain or elaborate principles that are stated in the Standards.

Second, whenever the Guide cites one of the standards to support a point, that standard should be reproduced in the text or in a footnote, only omitting from the quote any text that is unrelated to the Standards' recommendations on that point. Many of the references to the Standards in the current Guide are paraphrased or modified, and in some cases the meaning has been slightly changed. An example of the type of error that can occur when a standard is paraphrased or quoted incompletely appears on page 3 of the Guide. The sentence associated with footnote 5 and footnote 6 reads: “The Joint Standards

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

indicate that a high-stakes decision generally should not be made on the basis of a single score. The use of multiple measures generally enhances the validity of the decision.” In the Standards, the cited standard, 13.7, reads in full: “In educational settings, a decision or characterization that will have major impact on a student should not be made on the basis of a single test score. Other relevant information should be taken into account if it will enhance the overall validity of the decision” [italics added]. The supporting text for this standard continues: “Any assessment data used in making decisions are evaluated in terms of validity, reliability, and relevance to the specific needs of the students.” The Guide's discussion of the standard omits the qualifying statement (italicized above) and the call for evaluation of the data for their intended use. In many cases, the standards themselves include detail and qualifications that are important to proper interpretation. The supporting text also frequently provides explanation, helpful examples, and practical advice for the important, difficult work suggested. Care should be taken to make sure the meaning of quoted standards is not changed as they are discussed in the Guide.

Other examples are on pages 25-28 and 32-33, where the Standards are cited many times, but not all of the footnoted text consists of direct quotes. Some of the modified text slightly changes the meaning of the Standards; perhaps more important, there is the possibility of confusing readers as to the authority for the points made in these sections. Wherever the Guide is adopting the recommendations of the Standards, that should be clear. If the Guide is expanding on or otherwise deviating from what is called for in the Standards, that should also be made clear.

Technical Issues

The Guide undertakes the considerable challenge of conveying to lay readers some of the important technical considerations that relate to fairness in testing. It is surely true that some members of the Guide's target audience will be unfamiliar with the technical language used in the Standards, and the Guide provides important explanation, elaboration, and emphasis of key ideas for many of these topics.

In general, the Guide succeeds in laying out the relevant measurement issues, but several specific points either are not entirely correct or are outdated. We are aware that for much of the time that the Guide was being drafted and revised the 1999 version of the Standards was not available, and we believe that some of the Guide text should be revised in accordance with the new Standards. We offer specific suggestions here for improving the discussion of several points and in doing so provide guidance for further possible improvements.

The Guide refers in several places to the importance of students having the opportunity to learn the material covered in tests whose results will be used in high-stakes decisions and does cite standard 13.5, which explicitly addresses that point. Because this is such an important point, and because determining in a particular instance what constitutes sufficient opportunity to learn could be difficult, we recommend that the Guide reproduce the language of standard 13.5 in full and reflect the discussion of this issue in chapter 13 and chapter 7 of the Standards (particularly the text on p. 76). We also recommend that the first Debra P. case be discussed in this context. The points made in

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

the District Court's first opinion about “punishing victims for deficits created by an inferior educational environment” (474 F.Supp at 257) and in the Court of Appeal's opinion about “present effects of past intentional discrimination” (644 F 2d at 407) are especially appropriate.

Several points related to the concept of validity need clarification in the Guide. The definition of validity in the first sentence of section A on page 19 is inadequate. We recommend a direct quote from the beginning of the chapter on validity or from the glossary of the Standards. We note that the references to construct validity, content evidence of validity, criterion evidence of validity, and predictive evidence of validity are inconsistent with the current language of the Standards, which now refer to the different types of evidence on which validity decisions rest (p. 174). We believe the Guide should reiterate current thinking about the various sources of evidence for evaluating proposed interpretations of test scores as described in the Standards. We also recommend that the idea that validity can be generalized beyond a specific sample of items, students, and settings should be addressed in this part of the Guide; a brief discussion of the conditions under which validity evidence from one application might generalize to another should also be included in this section.

The references on page 24 to “stability and instability” should more properly read “consistency and inconsistency.” Moreover, this section should more clearly convey the point that reliability varies both in degree and with the nature of the generalization that is made across test forms, across raters, across populations, etc.

The Guide's discussion of “Fairness in Validity” (starting on p. 25) is not consistent with the treatment of this topic in the Standards, which describes varying views of fairness. The Standards discuss fairness in test use and as a property of tests. We recommend that this section be revised to be more consistent with chapter 7 (particularly pages 74-80) of the Standards.

The discussion of inclusion and accommodation for English-language learners in the current version of the Guide is too brief: it provides little detail regarding the challenges of ensuring fairness in testing for this group, the possible effects of inclusions and accommodations on the validity of tests scores, and currently unanswered questions about native-language literacy and knowledge testing for students not proficient in English. Furthermore, we note that the Guide strays from its stated purpose and scope (on p. 50) in its recommendation on inclusion. Early on, the Guide defines high-stakes decisions as those with important consequences for individual students and indicates that student (not classroom or school or jurisdiction) decisions are its purview. The paragraph on page 50 and the accompanying footnote refer to “statewide assessments used to hold schools and school districts accountable for student performance.” It should be noted that inclusion rules for testing in program evaluation need not be identical to those used for making high-stakes student-level decisions.

Inclusion of limited-English-proficient students in a test used to make high-stakes decisions about individual students is psychometrically defensible if the assessment can be shown to be a valid and reliable indicator of targeted skills and knowledge, and if both the student 's English language proficiency and the degree to which English fluency is a central component of the skills and knowledge being tested are taken into account. Accommodations of existing assessments or alternative assessments may improve the validity and reliability of assessments and increase the level of participation of those

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

students in assessments. We recommend that both this section and the section on students with disabilities in the Guide be more closely tied to the portions of the Standards that address these issues (chapter 9 and chapter 10). We also recommend that this section be expanded or, if it remains brief, that it include qualifications similar to those outlined above. In either case, readers should be directed to specific sources of further detail on these questions.

The Guide currently treats admissions testing for higher education very briefly. More useful, perhaps, might be a statement that the general principles of sound educational measurement apply equally to tests used in K-12 education and to postsecondary and other admissions testing. However, the Guide should note that more detailed guidance on differences in the appropriate use of tests for prediction of postsecondary performance should be sought from other sources (which should be listed) or will be provided by OCR at a future date.

Finally, many of the definitions in the measurement glossary are confusing or incomplete. For example, the definition of “inferences of test scores” is “Meaning or interpretation of test results, as constrained by the validity evidence that has been accumulated for particular purposes and populations of students.” A more useful definition might address the information a test is designed to yield about the test taker's knowledge and skills in a particular domain. We recommend that the glossary be reviewed to ensure that it is both clear to lay readers and in accord with current use of the terms by measurement professionals. The glossary of the Standards will be a valuable resource for this review.

Legal Issues

The Guide's presentation of the laws that relate to fairness in testing is thorough and, for the most part, accurate. We believe, however, that it may not be as helpful to lay readers as it could be. It would be very helpful to non-lawyers, for example, if the relationships among the various relevant laws and regulations were clearly spelled out, and if the laws that most directly involve application of the standards were highlighted.

A related point is that OCR's authority to issue the Guide, which was questioned during discussion of previous drafts; is not presented as clearly as it might be. Lay readers in particular may be confused on this point. Although this issue is certainly addressed in the current draft, we recommend that the cover letter and introduction be reviewed to be sure that they both contain consistent, straightforward language explaining OCR 's legal responsibility to enforce civil rights laws and the specific mechanisms through which it does so.

We recognize both the difficulties of the task OCR has undertaken and the inevitable effects on texts when input is received from a wide variety of experts and interested parties. In this letter report no attempt was made to supply a line-by-line critique of the Guide. We have set forth several examples where revision of the current text would be helpful. Other examples might also be found. We encourage additional revisions where appropriate to further clarify the Guide and to ensure that it remains consonant with the Standards for Educational and Psychological Testing. We reiterate

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×

that the Guide will serve a very important function both in OCR's work of enforcing civil rights laws as they pertain to educational testing and in OCR's efforts to reach out to educational institutions and jurisdictions to assist them in devising testing strategies that comport with the law. We thank you for this opportunity to assist in your efforts to make the Guide accurate and useful.

Sincerely yours,

Robert L. Linn,

Chair

Board on Testing and Assessment

cc: Scott Palmer, Deputy Assistant Secretary for the Office for Civil Rights, U.S. Department of Education

Attachment: Board roster

Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 1
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 2
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 3
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 4
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 5
Suggested Citation:"Letter Report." National Research Council. 2000. Letter Report to the Office for Civil Rights, Department of Education. Washington, DC: The National Academies Press. doi: 10.17226/9880.
×
Page 6
Next: Board on Testing and Assessment »
Letter Report to the Office for Civil Rights, Department of Education Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!