subject: Responses From Students to Questions Asked [print this page] Responses From Students to Questions Asked
The fact that this question requires students to write a response in English provides a good alternative explanation; specifically, that Tina's first language is not English, and her writing skills in English are poor. This also points to a third alternative explanation, that the individual who scored Tina's answer gave her a low score because of the poor quality of her writing. This potential problem with the scoring of English language learners' written responses to academic achievement test tasks is discussed in National Research Council (1997b), which points out that errors result from inaccurate and inconsistent scoring of openended performance-based measures. There is evidence the scorers may pay attention to linguistic features of performance unrelated to the content of the assessment" (p. 122). These additional alternative explanations and the rebuttal evidence are also presented in Figure 6-5.
To understand what constitutes an inappropriate or invalid accommodation, as well as to describe current accommodations or to design new accommodations that may be more appropriate for different groups of test-takers, a framework for systematically describing the characteristics of assessment tasks is needed. Bachman and Palmer (1996) present a framework of task characteristics that may be useful in describing the ways in which the characteristics and conditions of assessment tasks are altered for the purpose of accommodating students with disabilities and English language learners. This framework includes characteristics of the setting, the rubric, the input, and the expected response, as well as the relationships between input and response. Bachman and Palmer maintain that these characteristics can be used to describe existing assessment tasks as well as to provide a basis for designing new types of assessment tasks. By changing the specific characteristics of a particular task, test developers can create an entirely new task. Accommodations that are commonly provided for students with disabilities or English language learners can also be described in terms of alterations in task characteristics, as in Table 6-3.
Task characteristics can be altered in such a way that the plausibility of alternative explanations for test results is lessened. This can be illustrated with the example used earlier, in which there were three possible alternative explanations for Tina's poor performance on the reading assessment. To weaken the first alternative explanation, that Tina's poor performance is the result of her lack of familiarity with the cultural content of the reading passage, test developers would need to control a characteristic of the input in this case the content of the texts used in the assessment to ensure that the texts presented to test-takers are not so unfamiliar as to interfere with their success on the assessment. One way to achieve this is to include students with disabilities and English language learners in the pretesting of items, and possibly to pretest items using cognitive labs. Nevertheless, even with a very large bank of pretested texts, there will still be some probability that a particular test-taker will be unfamiliar with a given text, and this interaction effect will be very difficult to eliminate entirely. To weaken the second alternative explanation, that Tina's poor performance on reading assessment is the result of poor writing skills (which are not being assessed), either or both of two characteristics of the expected response-language or mode of response-could be altered.
If Tina can read and write reasonably well in her native language, she might be permitted to write her responses in her native language. If she cannot write well in her native language but is reasonably fluent orally in English, then she could be permitted to speak her answers to a scribe, who would write them for her. If she is also not fluent orally in English, then she could be permitted to speak her answers in her native language to a scribe. In this part, learning a foreign language needs a leaning tools, many students choose Rosetta Stone German and Rosetta Stone Hebrew to learn German and Hebrew.The appropriateness of these alterations of course depends on the targeted construct. These alterations in task characteristics are shown in Table 6-4. These different accommodations provide data that weaken the possible alternative arguments and thus strengthen the inferences that were intended to be made based on Tina's scores. To weaken the third alternative explanation, that Tina's poor reading performance is the result of the scorer's using the wrong scoring criteria, the test developers need to control two characteristics of the scoring process: the criteria used for scoring and the scorers themselves. Such corrections are handled most effectively through rigorous and repeated scorer training sessions. Figure 6-5 portrays the relationships between the alternative explanations, rebuttal data, and types of data needed to weaken the alternative explanations.
