National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

Executive Summary

Under the Immigration and Nationality Act, applicants for U.S. citizenship are tested on their knowledge of English and of U.S. history and government. These tests are designed and administered under the U.S. Citizenship and Immigration Services (USCIS), an agency in the U.S. Department of Homeland Security. In 2001, the agency began the process of redesigning the testing program due to concerns that the current testing procedure may not be sufficiently uniform, and that test content may not be appropriate to determine if applicants have a meaningful knowledge of English or of U.S. history and government.

As part of the redesign effort, USCIS has requested independent advice from the Board on Testing and Assessment (BOTA) of the National Research Council. BOTA formed an expert committee to provide the requested advice. This interim report is the committee’s response to the first part of its charge: to help optimize the validity, reliability, and fairness of the redesigned tests and administration procedures. The committee’s final report will respond to the second part of the charge by providing an assessment of the processes used to develop and evaluate the new testing program before its nationwide implementation.

THE COMMITTEE’S APPROACH

The naturalization tests are high-stakes tests for large numbers of people, so it is especially important that they meet high standards for validity, reliability, and fairness. When tests are used to allocate valued societal benefits, such as citizenship, they need to be part of an accountable process that can be traced directly from those designing and administering the tests to the citizenry and their elected representatives. Any changes in the redesigned tests will be controversial, so it is critical that the development process be open, transparent, and accountable.

The committee is concerned that the redesign program currently lacks both the necessary advisory structure for making important decisions and a coherent, overarching research and test development design that will lead to a scientifically and politically defensible testing program. Thus, the committee’s recommendations address two major sets of issues, under the headings of structure and process. By structure, we mean the mechanism to oversee the redesign effort and provide advice through boards or advisory committees, arranged by expertise and function. Not only would a sound structure bring in a variety of expert opinions to address complex issues, but it would also enhance the credibility of the project to stakeholders and the public. Process refers to the research design and sequence of test development activities, based on best practices in testing, to ensure that the resulting decisions about naturalization applicants are valid, reliable, and fair. While the exact nature of research and test development activities varies depending on a test’s purpose and the context in which it is developed, professional norms of good testing practice require that a series of general procedures be in place when developing high-stakes, standardized tests.

Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

MULTITIERED OVERSIGHT

Because of the need for both political accountability and technical soundness, many high-stakes testing programs use a multitiered system of advisory bodies. Such systems often include three types of advisers. First, an oversight committee—including a range of constituent group representatives, members of the public, and elected or appointed officials—typically reviews options and analyses presented by technical and content experts, solicits a wide range of stakeholder input, and then weighs those sources in making policy recommendations. Second, technical panels—consisting of experts in psychometrics and test development—ensure that the testing program follows sound testing practices, providing technical advice that is independent from the decisions made by the testing contractor. Third, content panels—consisting of experts on the content to be tested—develop the content frameworks that define the knowledge and skills to be tested. In Chapter 2, the committee makes a recommendation about an advisory structure for the redesign effort:

Recommendation 1: U.S. Citizenship and Immigration Services should put in place an advisory structure to advise the agency in making important decisions about the naturalization test redesign.

TEST DEVELOPMENT

Widely accepted standards for educational and psychological testing require that the testing program have a sound scientific basis as well as systematic documentation of the approach being followed. In Chapter 3, the committee makes a recommendation about a plan to guide the testing program:

Recommendation 2: A detailed plan for test development should be created with help from a technical advisory panel and review by an oversight committee. The research and test development plan should comply with testing standards and should include all of the necessary steps for developing a valid, reliable, and fair test.

The development steps include specifying the purpose of the test and the inferences to be drawn, developing the content frameworks, building the test specifications, creating potential test items and scoring rubrics, reviewing and piloting test items, evaluating the quality of items, assembling test forms, and setting cutscores (passing scores) if needed.

As USCIS and its testing contractor gear up for a second phase of pilot testing, the committee thinks that a number of the essential steps described above have not been taken for both the English and the history and government tests. The committee’s concerns are heightened by the fact that USCIS plans to make pass/fail decisions about applicants based on their performance during the second pilot test, before that test has been shown to be valid, reliable, and fair and before a formal procedure has been developed for setting cutscores or deciding how cutscores will be used. Further pilot testing should be postponed until the necessary earlier steps

Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

are taken and are carefully documented. Alternatives to making pass/fail decisions based on the second phase pilot test should be seriously considered.

CONTENT FRAMEWORKS

One of the critical early steps in the test development process is developing content frameworks that clearly define the knowledge and skills to be measured. Ultimate judgments of the validity of interpretations of test scores—for example, that an individual is qualified to become a U.S. citizen—rely on clear definitions of what the test is intended to measure and evidence that it does indeed measure what was intended. In Chapter 3, the committee makes a recommendation about the development of the content frameworks:

Recommendation 3: Work on developing the content frameworks (including publishing the history and government framework in the Federal Register) should cease until a clear, transparent, and publicly accountable process is defined and vetted with an oversight group.

SETTING STANDARDS

USCIS has indicated that as part of its efforts to standardize the naturalization process, it plans to set passing scores for the newly designed tests. When the results of the standard-setting process have highly significant consequences for large numbers of examinees, the process by which passing scores are determined should be clearly documented and defensible. In the test redesign plans reviewed by the committee, it is not clear how the final passing scores will be determined or by whom. Best practices suggest that this final policy decision should be made by agency officials in close consultation with an oversight committee. The decision should be informed by the results of the standard-setting procedure. In Chapter 3, the committee makes a recommendation about setting standards:

Recommendation 4: After a determination has been made about the various item formats that will be used on the redesigned test, USCIS and its testing contractor should develop a detailed plan for standard setting, with input from the technical advisory group and a final recommendation by the oversight committee.

Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 1
Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 2
Suggested Citation:"Executive Summary." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 3
Next: 1 Introduction »
Redesigning the U.S. Naturalization Tests: Interim Report Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!