A Principled Approach to Language Assessment Considerations for the U.S. Foreign Service Institute (2020) / Chapter Skim
Currently Skimming:
4 Possible Approaches for Assessing Language Proficiency in the FSI Context
4 Possible Approaches for Assessing Language Proficiency in the FSI Context
Pages 41-62
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 41... ...
. OVERVIEW Scope of Committee's Work The chapter's discussion of possible changes to the current FSI test is guided by the description of the considerations in language assessment development in Figure 1-1 (in Chapter 1)
|
From page 42... ...
. Rather than describing all possible approaches to assessment, the committee selected the changes to discuss on the basis of its knowledge of the research in language assessment and its understanding of FSI's context, target language use domain, and current test.
|
From page 43... ...
Such improvements might strengthen the assessment of aspects of the language proficiency construct that are already partly reflected on the current test. With respect to the first goal of broadening the construct measured by the test or the test's coverage of that construct, several possible changes are discussed below: scoring for listening comprehension on the speaking test, adding writing as a response mode for some reading or listening tasks, adding paired or group oral tests, including listening tasks with a range of language varieties and unscripted texts, incorporating language supports, adding a scenario-based assessment, and incorporating portfolios of work samples.
|
From page 44... ...
Obviously, it is not productive to standardize a language proficiency test in the service of the second goal in ways that prevent the test from assessing the aspects of language proficiency that are important in the target language use domain. With respect to the second goal of increasing the fairness and reliability of test scores, the discussion below covers the following possible changes: adding short assessment tasks administered by computer, using automated assessment of speaking, providing transparent scoring criteria, using additional scorers, and providing more detailed score reports.
|
From page 45... ...
Table 4-1 summarizes the possible changes to the FSI test that are discussed in the rest of this chapter, with the potential goals the committee explored that might motivate consideration of these changes and their additional instructional or practical considerations. As discussed above, the potential goals for change would need to emerge from an overall review of the current test and its context, using a principled approach to analyze how the current test could be strengthened to support the key claims about its validity.
|
From page 46... ...
46 A PRINCIPLED APPROACH TO LANGUAGE ASSESSMENT TABLE 4-1 Possible Changes to the FSI Test to Meet Potential Goals Potential Test Construct, Reliability and Fairness Potential Instructional and Possible Change Considerations Practical Considerations Using Multiple Measures • Better coverage of Foreign • Additional cost for Service language uses test development and • Greater reliability and administration fairness Scoring Listening on the • More systematic use of • Potential for positive Speaking Test listening information effect on instruction already generated by the • Additional complexity to test the scoring process • Possibility of increased measurement error Adding Target-Language • Coverage of Foreign • Potential for positive Writing as a Response Service language uses that effect on instruction Mode for Some Reading or involve writing • Extra cost for test Listening Tasks development and administration Adding Paired or Group • Better coverage of Foreign • Potential for positive Oral Tests Service language uses effect on instruction related to interactional • Cost and practical competence challenges of coordinating • Possibility of increased tests measurement error due to partner variability Using Recorded Listening • Potential for better • Potential for positive Tasks That Use a Range generalization of listening effect on instruction of Language Varieties and assessment to typical • Increased cost for Unscripted Texts range of Foreign Service test development and contexts administration Incorporating Language • Better coverage of Foreign • Minor modifications to Supports (such as dictionary Service language uses current test and translation apps) Adding a Scenario-Based • Better coverage of • Potential for positive Assessment complex Foreign Service effect on instruction language uses • Increased cost for test development and administration
|
From page 47... ...
POSSIBLE APPROACHES FOR ASSESSING LANGUAGE PROFICIENCY 47 TABLE 4-1 Continued Potential Test Construct, Reliability and Fairness Potential Instructional and Possible Change Considerations Practical Considerations Incorporating Portfolios of • Better coverage of Foreign • Difficult to standardize Work Samples Service language uses • Extra cost for • Potential for increased development of scoring overall reliability and criteria and procedures fairness by using multiple measures Adding Computer- • Better coverage and • Additional cost and Administered Tests Using reliability for Foreign administrative steps, Short Tasks in Reading and Service professional topics which may be prohibitive Listening for low-volume languages Using Automated Assessment • Potential to increase • Capabilities are limited of Speaking standardization but improving • Potential to decrease cost of test administration • Expensive to develop, so cost-effective only for high-volume tests Providing Transparent • Potential for greater • Minor modifications of Scoring Criteria reliability and fairness current test information procedures Using Additional Scorers • Potential for greater • Minor modification of reliability and fairness current test procedures • Additional cost Providing More Detailed • Better understanding of • Potential for positive Score Reports scores for all users of FSI effect on instruction test • Increased cost and time for score reporting
|
From page 48... ...
A number of the possible changes discussed below provide examples of additional measures that could be added to the current test to produce an overall testing program using multiple measures. SCORING LISTENING ON THE SPEAKING TEST The committee's statement of task specifically asks about the possibility of explicit assessments of listening comprehension that could be part of the FSI assessment.
|
From page 49... ...
For FSI, for example, a Foreign Service officer participating in a formal meeting might need to make an initial presentation and then respond to follow-up questions, a use of integrated skills echoed in the "work-related exchange" portion of the FSI speaking test. One point emphasized by the integrated skills literature is the need to consider scoring all the skills that are of interest in the target language.
|
From page 50... ...
, allowing raters to judge whether test takers are able to comprehend the other speaker and respond appropriately, are aware of other speakers' roles in a conversation, and are able to manage conversational turn-taking, repair conversational breakdowns, and co-construct topics. Paired oral tests resemble naturalistic conversation, mirror pair work that is common in communicatively oriented and task-based language classrooms, and can help in the measurement of interactional competence (Ducasse and Brown, 2009)
|
From page 51... ...
The paired task involves listening to audio recordings of two speakers providing different positions on a topic, followed by an opportunity for the test takers to summarize, discuss, and defend one of the positions. The scoring criteria include specific attention to interactional competence, with consideration to connecting one's own ideas to a partner's ideas, expanding on a partner's ideas, making relevant comments, taking turns appropriately, asking appropriate questions, disagreeing politely, and answering questions in an appropriate amount of time.
|
From page 52... ...
INCLUDING LISTENING TASKS WITH A RANGE OF LANGUAGE VARIETIES AND UNSCRIPTED TEXTS A review of the current FSI test using a principled approach would consider the extent to which the results generalize from the test situation to real-world circumstances. The current speaking test includes listening in the target language that is spoken by the tester in a relatively structured exchange, but daily exchanges will likely include a much wider variety of types of speech.
|
From page 53... ...
However, in addition to providing better coverage of the target language use domain in some contexts, it would also likely have beneficial effects on instruction to provide test takers with exposure to the relevant range of language varieties. INCORPORATING LANGUAGE SUPPORTS In many situations, real-world language proficiency often involves the use of language supports (Oh, 2019)
|
From page 54... ...
. Many language assessments give test takers a series of unrelated tasks, each focusing primarily on a single language skill, such as reading or listening, which do not reflect the complexity of language use in real-world goal-oriented activities.
|
From page 55... ...
. The overarching goal of the intermediate module is for test takers to make an oral pitch for an educational trip abroad to a selection committee on behalf of a virtual team, which requires a coherent series of interrelated subtasks involving multiple language skills that need to be performed on the path toward scenario completion ("pitch the trip")
|
From page 56... ...
For FSI, a portfolio could be used in addition to the current test, which would be an example of using multiple measures for decision making. Portfolios may help address concerns that some test takers may be able to pass the test but not actually be able to use the target language in their work, while others may be able to use the target language in their work but not pass the test.
|
From page 57... ...
For example, the current FSI test is intended to assess a test-taker's ability to understand and use professional-level vocabulary, discourse, and concepts in relation to a range of political, economic, and social topics that are relevant to the Foreign Service; however, only two or three reading texts are used for in-depth reading. A test using short assessment tasks in reading or listening could sample from a greater range of discourse and topics than can the limited number of reading passages sampled in the current test.
|
From page 58... ...
A simpler but conceptually related approach would make use of a "two-step" process. In this approach, a screener test would be used to estimate if test takers are likely at or above a threshold level of proficiency
|
From page 59... ...
Test takers below this threshold would not go on to take the full test, and the score on the screener would be their official score. For expedience and cost-effectiveness, the screener test could be computer administered and consist of questions with machine-scorable response formats.
|
From page 60... ...
PROVIDING TRANSPARENT SCORING CRITERIA Because of the subjective nature of scoring extended responses, such as those elicited by the FSI test, it is important that scorers be well trained to apply the criteria laid out in the scoring rubric and that the criteria clearly
|
From page 61... ...
. To help ensure the reliability, fairness, and transparency of the scoring process used in the FSI test -- as well as the perception of that reliability and fairness -- FSI should consider providing more information to the test takers and users about its scoring rubrics and procedures, as well as its scorer training processes.
|
From page 62... ...
If the review of FSI's testing program shows any potential concerns about its fairness, additional transparency about the reasons for individual scores can help address those concerns, as well as help identify aspects of the scoring process that may need to be improved. As with the comments above about transparent scoring, the provision of more detailed score reports could be part of an overall effort to develop a shared understanding about language assessment across all stakeholders across the State Department.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.