National Academies Press: OpenBook

Redesigning the U.S. Naturalization Tests: Interim Report (2004)

Chapter: 2 Structure of the Redesign Program

« Previous: 1 Introduction
Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

2
Structure of the Redesign Program

Professional testing standards require that tests with important consequences for the individuals taking them should be valid, reliable, and fair. The criteria for reliability and validity are largely technical ones: Can the requisite skills and knowledge be measured with a certain degree of consistency? Does the test measure what it purports to measure, and are appropriate and justifiable inferences being drawn from the test results? In contrast, fairness is not only a technical standard but also a political and social one. It embodies the technical aspects of reliability and validity, and it also draws on societal conceptions of what constitutes equitable and fair treatment (National Research Council, 1999a).

Technical and political issues are also joined when tests are used to advance public policy purposes, as in the case of the naturalization tests. When tests are used in this way, they impose consequences on the larger polity, beyond those taking the test, because they are used to allocate valued societal benefits, such as citizenship (McDonnell, 2004). The naturalization tests are an instance of a nation using tests results to help decide which new members to admit, and thus the entire nation has a stake in it. Consequently, such tests need to be part of an accountable process that can be traced directly from those designing and administering the tests to the citizenry and their elected representatives (Gormley and Balla, 2004; Gruber, 1987). In instances in which test content and use are contested, it is especially critical that the development process be open, transparent, and accountable. Although some stakeholders may disagree with decisions about test content or use, a transparent and accountable process that also meets professional standards of good testing practice will increase the likelihood that stakeholders will view the outcome as legitimate.

MULTITIERED OVERSIGHT

Because of this need for political accountability and technical soundness, many high-stakes testing programs—whether for granting high school diplomas, admitting students to college, or licensing for professions—use a multitiered system of advisory bodies. These bodies provide oversight and guidance, both during the development process and on an ongoing basis once the test is implemented. As part of the documentation supporting the test, a test developer should describe the qualifications of members of its advisory bodies, how they were selected, and the extent to which they participated in the process (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999).

Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

The function and composition of such advisory bodies vary, depending on the purpose of the test and the specific circumstances of its administration and use. However, most testing systems include three types of advisers that differ in their role and composition:

  • a policy or oversight committee,

  • a technical panel, and

  • content panels.

The purpose of a policy or oversight committee is to ensure that policy decisions about such issues as the research and test development plan, content frameworks, passing scores, and test administration procedures take into consideration both expert judgments and public values and concerns. Such a committee typically reviews analyses and options presented by technical and content experts, solicits a wide range of stakeholder input through various public mechanisms, and then weighs those sources in making policy recommendations. In its oversight function, this type of committee often reviews data on the test’s impact once it is operational, recommending modifications when necessary. Such a committee usually includes a range of constituent group representatives and members of the public, as well as elected or appointed officials or their designees. The committee recognizes that it is difficult to determine what type of members can best represent the public, given the wide-ranging and varied public views held about immigrants and citizenship. Nevertheless, national and state testing programs routinely appoint public members, and USCIS should consider doing so as one way to bring the broader public interest into its deliberations.

Depending on the legal standing and composition of an oversight committee, its decisions may automatically become policy for a particular testing system, or they may be recommendations to an agency official or political body with authority to enact policy. Under either arrangement, however, responsibility for key decisions is expanded beyond the staff of the agency sponsoring the test and its testing contractor, thus giving the test greater legitimacy.

Technical panels consist of experts in psychometrics and test development who can provide advice that is independent from the guidance and decisions made by the testing contractor. The purpose of the technical panel is to ensure that the testing program follows sound testing practices. Its role may include reviewing and advising on the overall research and test development plan, as well as suggesting specific studies and pilot tests necessary to produce a valid, reliable, and fair test. Technical panels can also assist in interpreting data drawn from the pilot tests and research studies and their implications for test design. These experts can serve as on-call advisers to agency managers and the testing contractor.

Testing programs usually also have a content panel for each subject tested. Content panels consist of experts representing different theoretical positions or viewpoints on the content to be tested, since there are controversies in all subject areas about what constitutes proficiency in a given area. Such committees are often charged with developing the content framework that defines the constructs (the body of knowledge and skills) to be tested, helping to develop test specifications, and reviewing test items and forms to ensure that they are aligned with the frameworks—an important aspect of validity.

Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

Two Examples

Two examples of the type of multitiered advisory structure we have briefly described are outlined below to illustrate how these bodies operate. Such bodies are widely used in large-scale, high-stakes testing programs. If USCIS would like information about others, the committee can provide additional examples.

The SAT Advisory Structure

The SAT is the most widely used college admissions test. The scores of individual students on the SAT are used by college admissions officers as a factor in the admissions decision. The test is developed and administered by the College Board, a nonprofit organization with an advisory committee on research and development that oversees all of its programs.

The SAT is currently being revised, so the College Board has formed committees of experts in reading, mathematics, and writing to assist in redeveloping the test. The committee members are drawn mainly from the ranks of distinguished educators at the high school and college levels. The experts will make recommendations to the College Board on proposed changes to the SAT. The College Board has also established a psychometric advisory panel, a group of specialists in the areas of educational and psychological measurement. It will advise on the process of designing special studies and conducting field trials and pilot tests and will review analyses conducted by technical staff working on the test’s redesign.

Virginia’s High School Exit Exam Advisory Structure

Most states that require students to pass a test in order to receive a high school diploma (25 states currently have high school exit exams) also typically have advisory committee structures. Virginia uses these kinds of advisory bodies for its Standards of Learning (SOL) tests. In the late 1990s, after members of the public questioned various aspects of the test, the state board of education created the SOL Assessment Program Advisory Committee, consisting of 23 members drawn from parent-teacher organizations, district and school officials, teachers, and the Virginia School Boards Association. The advisory committee does an annual audit of the SOL program and provides recommendations to the state board of education. In turn, the state board of education can ask the advisory committee to investigate and comment on changes to the content of the tests. The board of education has also appointed a five-member panel of testing experts to advise it on technical matters related to the tests. In addition, a content committee was assembled to advise on a revision of the history portions of the exam.

Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

Importance of an Advisory Structure

The specific organization and composition of each testing program’s advisory structure differ, depending on the needs and politics specific to the program. But the common theme is that high-stakes testing programs build these structures to seek expert guidance, gain legitimacy, and provide a buffer if decisions or results are criticized. The committee recognizes that USCIS has already begun to build some of its own structure. A history and government panel was convened to develop a content framework, and USCIS meets regularly with selected stakeholders to get input into the redesign process. However, these sources of input have been identified and used unsystematically. In the committee’s judgment, too many of the test design decisions are being made by a small number of USCIS and MetriTech staff. Although many of the decisions may be appropriate, they lack the credibility of having gone through a more public, deliberative process, and they leave the agency open to criticism. In the committee’s view, the moderate investment of time and money required to implement the type of advisory structure proposed in this report will more than pay off in the future, by helping to avoid criticism and costly setbacks further down the road.

Role of the National Research Council Committee

In discussing advisory structures, an obvious question arises as to how a group such as our committee at the National Research Council (NRC) fits into the kind of arrangements we have outlined as characterizing many major testing programs across the country. Certainly the committee members represent the range of expertise typically found on technical and content panels, and the committee’s charge touches on some of the same issues covered by a multitiered advisory structure. But our committee’s function is quite different. Its focus and approach are consistent with the mandate of the NRC, an autonomous organization charged with providing scientifically based policy analysis to government agencies. For that reason, the committee’s analysis and recommendations must remain separate from the operational activities of the naturalization test redesign.

Although we can provide feedback related to some major aspects of the testing program, as we are doing in this report, we cannot provide the political legitimacy of a properly constituted oversight committee, the on-call and confidential advice of a properly constituted technical panel, or the broad and deep guidance of properly constituted content panels. Furthermore, NRC procedures require that committee recommendations be transmitted in formal, public reports to the sponsor that represent a consensus of all committee members, reflect conclusions based on the scientific research literature, and have undergone review by a separate set of independent experts. These procedures prohibit the NRC committee from providing “real-time” or private advice or from making recommendations that rest on political rather than scientific judgment. Finally, in order for the committee to fulfill the charge for the final report of providing an objective and credible overall assessment of the technical quality of the test redesign, we cannot become so involved with the decisions of the test redesign program that the final report would amount to an overall assessment of our own work. For all these reasons, the ability of the NRC committee to substitute for the usual advisory bodies is limited.

Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×

Recommendation 1: U.S. Citizenship and Immigration Services should put in place an advisory structure to advise the agency in making important decisions about the naturalization test redesign.

USCIS should form a policy or oversight committee, consisting of elected or appointed officials (or their designees), representatives of constituency groups, appropriate experts, and members of the public. The oversight committee should respond to technical and content analyses, solicit broad public input, and make recommendations about general policy directions, such as those related to the overall structure and administration of the tests, the content frameworks and passing scores, and possible alternatives to the tests. This group should be a standing committee that meets regularly during the design phase and as needed once the new tests become operational.

USCIS should establish a small technical panel consisting of experts in psychometrics, test development, and standard setting to provide both the agency and the testing contractor with specific, ongoing technical advice throughout the redesign process. Because the tests may need to be modified and new test forms created, the technical panel should be a standing one that meets as needed once the new tests become operational.

The oversight committee should evaluate the composition of the previously convened history and government content panel and decide if it encompasses all of the necessary categories of expertise. If needed, USCIS should modify that panel to form a new history and government content panel. The panel should guide the development of the content framework and test specifications for the history and government test, and review test items and forms to ensure they are aligned with the content framework and test specifications.

USCIS should form an English language content panel consisting of experts in language assessment, English as a second language, and adult education. The panel should guide the development of the content framework and test specifications for the English language test, and review test items and forms to ensure they are aligned with the content framework and test specifications.

These advisory bodies should not operate independently. Some coordination and collaboration will be needed. This can be achieved by having overlapping joint memberships on committees or by holding some joint meetings. The oversight committee should review the work of both the technical and content panels. In these ways, each panel’s work will be informed by the work of the others, so that the work is harmonious and synergistic.

Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 7
Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 8
Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 9
Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 10
Suggested Citation:"2 Structure of the Redesign Program." National Research Council. 2004. Redesigning the U.S. Naturalization Tests: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/11168.
×
Page 11
Next: 3 The Test Development Process »
Redesigning the U.S. Naturalization Tests: Interim Report Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!