National Academies Press: OpenBook

Evaluation of the Voluntary National Tests: Phase 1 (1999)

Chapter: 5 Inclusion and Accommodation

« Previous: 4 VNT Pilot and Field Test Plans
Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×

5
Inclusion and Accommodation

In the November 1997 legislation that established the National Assessment Governing Board's (NAGB) responsibility for the development of the Voluntary National Tests (VNT), Congress required NAGB to make four determinations. The third of these is “whether the test development process and test items take into account the needs of disadvantaged, limited English proficient and disabled students” (P.L. 105-78: Sec. 307 (b) (3)). The same legislation called on the National Research Council (NRC) “to evaluate whether the test items address the needs of disadvantaged, limited English proficient, and disabled students.”

There are two key challenges to testing students with disabilities or limited English proficiency. The first challenge is to establish effective procedures for identifying and screening such students, so they can appropriately be included in assessment programs. Federal law and state and local policy increasingly demand participation of these special populations in all education activities, both as a means of establishing the educational needs and progress of individual students and for purposes of system accountability. The second challenge is to identify and provide necessary accommodations (e.g., large-print type, extended time) to students with special needs while maintaining comparable validity of test performance with that in the general population (see National Research Council, 1997, 1999b). That is, any accommodation should alter only the conditions of assessment without otherwise affecting the measurement of performance. This issue is growing in importance, along with the number of students with disabilities or with limited English proficiency. Students with disabilities now comprise 12.3 percent of all students in elementary and secondary school, and students with limited English proficiency are 5.5 percent of all students.

Findings

While NAGB has just approved a set of “principles” for the inclusion and accommodation of students with disabilities and limited English proficiency in the pilot test of the VNT (National Assessment Governing Board, 1998c), there has as yet been little developmental work on the project to

Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×

address the special needs of these populations. The 430 participants in cognitive laboratory sessions included students with disabilities (32 in reading and 19 in math) and limited English proficiency (11 in reading and 12 in math), but their numbers were too small to provide substantial or reliable information about the participation of such students in the VNT.

In addition to the statement of principles, the American Institutes for Research (AIR) has two planning documents: “Revised Inclusion and Accommodations Work Plan” (May 8, 1998), which was approved by NAGB at its May meeting, and “Background Paper Reviewing Laws and Regulations, Current Practice, and Research Relevant to Inclusion and Accommodations for Students with Disabilities” (July 23, 1998). AIR has also developed lists of organizations with particular interest in the educational and testing needs of students with disabilities or limited English proficiency for NAGB's use in holding public hearings about inclusion and accommodation. We understand that a parallel background paper on inclusion and accommodation of students with limited English proficiency will be presented to NAGB at its November 1998 meeting. As noted above, we know of no VNT development activities or plans specifically aimed at the needs of disadvantaged students or those with limited English proficiency, but such students did participate in one development activity—the cognitive labs.

Because of the compressed schedule in the early phases of VNT development, along with the desire to achieve close correspondence between the VNT and NAEP, the NAGB plans and the AIR background paper on students with disabilities both focus on recent NAEP practices for inclusion and accommodation, rather than taking a broader, more proactive stance. We believe the federal government has an important leadership role to play in subsidizing and demonstrating valid efforts to include these populations.

The procedures discussed in the draft documents are intended to increase participation and provide valid assessments for all students, but they essentially involve retrofitting established assessment instruments and procedures to special populations of students; another approach would be to design and develop assessments from the beginning that are accessible to and valid for all students.

From its beginning around 1970 and through the middle 1990s, NAEP assessments were carried out without accommodation of any kind. Procedures for “inclusion” usually focused on the exclusion of some students from the assessments, rather than on universal participation. Since the mid-1990s, as the growth of special student populations and the importance of their participation in large-scale assessments have increasingly been recognized, NAEP has experimented with new, more inclusive participation and accommodation policies, which are recapitulated with reference to the VNT in the NAGB principles and AIR planning documents. For example, in NAGB's draft principles, and depending on the test and population in question, accommodations for the pilot test may include large-print booklets, extended time, small-group administration, one-on-one administration, a scribe or computer to record answers, reading a test aloud by an examiner, other format or equipment modifications, or a bilingual dictionary if it is normally allowed by the school. However, the success of these policies in increasing participation is not yet established, nor have their effects on test performance and score comparability been validated.

Unless extensive development work is done with students with disabilities and with limited English proficiency, it would be unreasonable to expect that the VNT will be valid for use with these student populations. Both of these populations are heterogeneous, e.g., in primary language, level of proficiency in English, and specific type of disability. Moreover, they differ from the majority of students, not only in ways that affect test-taking directly—e.g., those that can be accommodated through additional time or assistive devices—but also in styles of learning and in levels of motivation or anxiety. Such differences are very likely to reduce the validity and comparability of test performance.

The Committee on Appropriate Test Use has identified two important ways in which inclusion

Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×

and accommodation can be improved (National Research Council, 1999b). First, the focus should be on inclusion and accommodation issues throughout item and test development, so a test is designed from the ground up to be accessible and comparable among special populations. For example, the NRC report recommends oversampling of students with disabilities and with limited English proficiency in the course of pilot testing so there will be sufficient numbers of cases in major subgroups of these students to permit DIF analyses. Second, test developers should explore the use of new technologies, such as computer-based, adaptive testing for students who need extra time, which show promise of substantially reducing or eliminating irrelevant performance differentials between many students who require accommodation and other students. The NRC Committee on Appropriate Test Use recognized, however, that development work of this kind is just beginning, and there are presently few successful exemplars of it.

Conclusion

The statement of principles and the AIR planning documents provide a limited basis for evaluation of provisions for inclusion and accommodation in the VNT—and no specific basis to address the quality of item development relative to the needs of those students.

NAGB's desire to maintain correspondence between NAEP and the VNT has not precluded departure from current NAEP practices in other areas—for example, the use of intertextual items and gridded responses. In our judgment, a major opportunity for improved large-scale assessment is being lost in NAGB's conservative approach to inclusion and accommodation in the VNT.

Recommendation

5-1. NAGB should accelerate its plans and schedule for inclusion and accommodation of students with disabilities and limited English proficiency in order to increase the participation of both those student populations and to increase the comparability of VNT performance among student populations.

This recommendation requires prompt action because so much of the development work in the first round of the VNT has already been completed. We have already noted the modest attention to students with special needs in the cognitive laboratory sessions. In the pilot test, NAGB plans to identify students with disabilities and with limited English proficiency and with the types of accommodations that have been provided. However, there are no provisions in the design to ensure that there will be sufficient numbers of these students—such as students requiring specific types of accommodation—to support reliable DIF analyses. We think that it would be feasible to include larger numbers of such students in the pilot and field tests, for example, by increasing sampling fractions of such students within schools. Moreover, there appears to be no plan to translate the 8th-grade mathematics test into Spanish (or any other language), a decision that is likely to affect participation in the VNT by major school districts. There has been some discussion of a Spanish translation after the field test, but this would be too late for the item analyses needed to construct comparable English and Spanish forms.

Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×
Page 45
Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×
Page 46
Suggested Citation:"5 Inclusion and Accommodation." National Research Council. 1999. Evaluation of the Voluntary National Tests: Phase 1. Washington, DC: The National Academies Press. doi: 10.17226/6324.
×
Page 47
Next: 6 Reporting »
Evaluation of the Voluntary National Tests: Phase 1 Get This Book
×
Buy Paperback | $47.00 Buy Ebook | $37.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In his 1997 State of the Union address, President Clinton announced a federal initiative to develop tests of 4th-grade reading and 8th-grade mathematics that would provide reliable information about student performance at two key points in their educational careers. According to the U.S. Department of Education, the Voluntary National Tests (VNT) would create a catalyst for continued school improvement by focusing parental and community-wide attention on achievement and would become new tools to hold school systems accountable for their students' performance. The National Assessment Governing Board (NAGB) has responsibility for development of the VNT. Congress recognized that a testing program of the scale and magnitude of the VNT initiative raises many important technical questions and requires quality control throughout development and implementation. In P.L. 105-78, Congress called on the National Research Council (NRC) to evaluate a series of technical issues pertaining to the validity of test items, the validity of proposed links between the VNT and the National Assessment of Educational Progress (NAEP), plans for the accommodation and inclusion of students with disabilities and English-language learners, plans for reporting test information to parents and the public, and potential uses of the tests. This report covers phase 1 of the evaluation (November 1997-July 1998) and focuses on three principal issues: test specifications and frameworks; preliminary evidence of the quality of test items; and plans for the pilot and field test studies, for inclusion and accommodation, and for reporting VNT results.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!