Center on English Learning & Achievement
The Unseen Social and Cultural Substance of Written Responses in Mathematics
Vicky Kouba, Audrey Champagne, and Zaline Roy-Campbell, O. Cezikturk, M. Benschoten, S. Sherwood, and C. Ho
Students written responses to open-ended (extended constructed response) tasks show that student performance differs qualitatively based on such social and cultural factors as knowledge of valued forms of communication in mathematics, language and reading proficiency, knowledge of task-designers' implicit perspective, and type of logic. Real-world contexts also introduce greater opportunities for divergent, yet reasonable responses from students. Extended constructed response tasks are a viable means for meeting some of the challenges of equity.
Knowing how to participate in the socially constructed forms of mathematics and science communication is an expectation of the current standards-based reform in mathematics and science (Champagne & Kouba, 1997a). Morgan (1998) argues that this expectation is a hidden, but very present aspect of assessment. She states that in many assessment situations the learner "not only needs to understand a particular piece of school mathematics but also needs to know the forms of behaviour that will lead to recognition of this and how (and when) to display these forms of behaviour" (p. 4). The difficulty is not just that students do not heed directions to explain and justify, as Dossey, Mullis, and Jones (1993) state, but that students do not know what is implied by such directions. Morgan (1998) frames this issue as one of equity:
Our current research on mathematics and science literacy, as part of the Center on English Learning and Achievement (CELA), is an investigation of students written explanations in mathematics and science. One of our goals is to systematically examine student responses for clues on how culture and language may be part of "disadvantaged" students performance, especially in terms of mathematical behaviors valued within the larger mathematics community and culture.
As we examine student responses, we also must keep in mind Tates (1996) admonition that any delineation of expected behaviors, especially in testing situations, ought to be done from a multicultural and social reconstructionist philosophy that allows students to "solve problems from their lived reality" (p. 195).
History of the Task
One of the most difficult 1996 NAEP extended-response items for eighth-grade students was:
Less than 1 percent of the 1,615 eighth-grade students in the NAEP sample provided an extended or satisfactory response, 29 percent gave a partial response, 4 percent gave a minimal response, nearly 40 percent gave an incorrect response, and nearly one-third omitted the item (NCES, 1999; Kenney & Lindquist, in press).
Eighth-grade students responses in the NAEP sample were scored as follows:
Extended -- Correct response (Justification need not include table, but should account for all possible length-width combinations or demonstrate evidence that all combinations have been considered in the formulation of the explanation.)
Satisfactory -- Response that indicates that 9 x 9 square has maximum area (81 sq. ft.) or that another rectangle has maximum area but accompanying work contains a minor error OR Response contains all work for nine rectangles (widths 1 through 9), but maximum area is not indicated (or response indicates that a rectangle other than the 9 x 9 has the maximum area).
Partial -- Response shows at least 3 different
rectangles (dimensions and areas); may indicate that one of those rectangles has maximum
Minimal -- Response demonstrates a minimal understanding that area and perimeter formulas for rectangles are needed in the solution and may show a beginning attempt to organize the data and information. For example, this might be illustrated by generating one specific value for the length and width that fits the given information.
Incorrect any other incorrect answer. (NCES, 1999)
Kenney and Lindquist (in press) suggest that the low performance on this item could have been a result of a limited scoring guide. They felt that the NAEP scoring guide did not account for students who had worked on similar items and "just knew" that a square yields the maximum area for a quadrilateral with a fixed perimeter. Based on our research, we also believe that narrowly interpreted scoring guides on high-stakes tests may depress reports of performance (Kouba, 1999). However, our results suggested that the low performance in our sample seemed more a societal or cultural result of students lack of knowledge about the expected forms of literacy (see Champagne and Kouba, 1997b for more on forms and levels of literacy in science and mathematics).
Study and Results
Expected Forms of Responses
We were curious about what a detailed examination of a large sample of students responses to the dog yard item would reveal about students prior knowledge and understandings of expected forms of response. We administered the dog yard item to 315 eighth-grade students across three middle schools in an urban school district that is racially, linguistically, and economically diverse. The students in our sample took the item as the last one of their year-end science exam (a situation outside the environment of the mathematics classroom, much as the NAEP assessments were administered outside of the usual mathematics classroom routine).
We did not use the NAEP scoring guide, except to look at the extended and satisfactory responses. The students in our sample faired somewhat better than the NAEP group, but the results still were low with 2 percent scoring in the extended or satisfactory level. On the encouraging side, we did have a couple of explanations where students used the expected justification structure of showing all possible areas of rectangles and concluding that the square had the largest area. We also had responses that showed an ability to argue from a more abstract perspective. For example, two students gave the following responses:
Both of these responses have the structure of a justification, a stated conclusion with a warrant (because ), and both demonstrate an understanding of the task and pattern of change in areas as length and width are altered. Both also rely on a linguistic rather than a symbolic or diagramatic explanation.
But, what of the students who did not garner a rating of Extended or Satisfactory? Was poor performance more a lack of mathematical understanding and mathematical background, or a lack of understanding the societally determined expectations for presenting a convincing argument?
Kenney and Linquist (in press) suggested that middle school students have prior experience with determining the maximum area for figures with fixed perimeters, and thus know the mathematics necessary to solve the item. For our sample of students, we checked with the schools and found that most of the eighth-grade students had done problems that looked at maximum areas possible with fixed perimeters. We also have evidence in the students responses of prior knowledge and experience with such items. The latter of the two responses displayed above seems more a reiteration of an established conclusion than an explanation of a relationship discovered as a result of doing this particular dog yard item. We also had students who wrote,
The mention of a barn or house, and the suggestions of a circle also seem indicative of prior experience with similar types of items. Thus, students in our sample who answered:
may have clearly understood the mathematical relationships within the item, but not the testing expectations in terms of providing a convincing justification. Based on our qualitative analysis of the students responses, at least a third of the eighth-grade students had the requisite mathematical understanding. Thus, we strongly agree with Morgans (1998) recommendation that the expectations for form of response must become part of the daily instruction in mathematics. Equitable instruction and assessment require that students get systematic instruction in the valued forms of mathematical explanations.
Linguistic Concerns Related to Equity
We also found evidence of linguistic or mathematical reading comprehension difficulties such as reading "four sides" as "four equal sides" or reading "whole number lengths" as meaning "even lengths":
Some students also thought of right angles as "squared" angles, thus requiring Julie to make "a squared fenced in area for her dog." These linguistic difficulties led students to a correct shape and area, but an incorrect justification. Other students simply wrote that they could not make sense of some of the words in this task or did not understand what the task was about.
Although language poses difficulties for all students who struggle with reading, it is a particular concern for students for whom English is not their first language. In order to be a good problem solver, one must be proficient in the language in general as well as the technical and symbolic languages of mathematics. Therefore, the limited English proficient (LEP) child may be at a disadvantage, not because he or she does not possess the necessary skills to solve the problem but because of a lack of "accessibility" in the second language (Mestre, 1981). Failure to master formal discourse styles may interfere with students understanding of word problems (Cummins, 1991). Mathematics has its own specific forms of discourse. Therefore LEP students must master this language as well. Students are required to combine their linguistic, cognitive, and meta-cognitive development to successfully comprehend the reading. Thus, in addition to the mathematics skills that they need to solve the problem, students must simultaneously develop the requisite comprehension skills while encountering text that is culturally biased. (Fencing in dogs is not a universal cultural activity.)
A second linguistic dimension to the interpretation of the dog-yard task emerged from our consideration of the students responses. Six students suggested making the dog yard some shape other than a square or rectangle (e.g., circle, hexagon, pentagon, or irregular shape using the sides of the doghouse or of Julies house). As we discussed these responses, we kept trying to make sense of these students written comments to Julie that their shapes were "what would work best." Were the students just answering the last question, "What is the largest area that Julie can enclose with 36 feet of fencing?" Or were they thinking in some other way? English education members of our research group suggested thinking of inflection as a factor in students reading comprehension. Julie may want to make a rectangular shape and use fence for all sides of the enclosure, but the best solution is clearly something else. In other words, the students may have interpreted the task as one of convincing Julie that her wants were not the best solution.
Social and Cultural Concerns Related to Equity
About three percent of the eighth-grade students (11 students) gave answers that indicated a divergent, yet reasonable interpretation of the context. Two students said that the problem could not be solved because they didnt know the shape of Julies yard, i.e., " I really dont know how her yard looks and how her yard is measured;" or "[maybe] she cant do it because her yard is too small." These responses support the concerns that Tate (1996) raises; that is, that some items are constructed from a cultural or economic perspective quite different from that of many of the students (e.g., a White, middle class, or Eurocentric perspective). The dog yard task seems designed from the perspective of having a relatively large yard. This is contrary to the "lived reality" of many of the students in our study. The lived reality for many of the students in our sample, especially those from the lower socioeconomic groups, is that they live in urban apartments or brownstones which have tiny (4-foot by 4-foot) front yards and no backyards. And those urban buildings that do have backyards often have oddly shaped ones that would not allow for a 9-foot by 9-foot square dog yard. Thus, students who placed themselves entirely within the context from their perspective may have dealt in a literal and self-situated way with the direction to find the maximum area for Julies dog yard.
An equitable approach to preparing students to respond in reflective ways to items such as the dog yard task might be that teachers help students to identify and solve from multiple perspectives. Tate (1996) suggests that teachers employ multicultural and social reconstructionist approaches where students ultimately are expected to solve the same question from the perspective of different members of the class, school or society. This necessitates teaching students the scientific habit of mind of always considering alternative assumptions and always suggesting solutions from alternate assumptions.
Logic, Reality, and Equity
Some divergent responses initially might be perceived as just idiosyncratic differences in people. However, we view these as indicative of child rather than adult logic, (as in Piagets argument that the intellectual structures of children are not the same as those for adults). Childrens thoughts, references, and logic are embedded in the details of the reality of the context. Once the mathematics has been embedded in a context, children are less able than adults to see the context as just a vehicle for understanding the mathematics. Children are less able to extract the mathematics from the context. For some students, this leads to an inability to respond as expected, because they cannot "get past" a contextual detail that the adult may have never considered. For example, some students in our sample were concerned about having a gate so that "the dog would be able to get out" and suggesting fencing around the dog house, but keeping enough pieces of fence to make a gate. Another student, who seemed to have prior knowledge about appropriate shapes of pens for dogs said a 6 by 12 rectangle should be used because that was the largest area that we could have while at the same time giving the dog room to run. One student thought it couldnt be done with only 36 feet of fence and indicated some relationship between the height of the fence and the amount of fencing. As we discussed this response, we realized that the student seemed to be thinking that Julie had 36 board-feet of fence. Perhaps this student had experience buying lumber by the board-foot.
Although some of these types of attention to detail can be avoided by careful pilot testing and construction of tasks, there is no way to make a context free from alternative interpretations. We are brought again to the need to help students learn to provide multiple solutions from multiple assumptions (e.g., if we ignore the need for a gate, the answer is x; if we take into consideration the need for a gate, the answer is y).
Our work has brought us to the conclusion that the use of extended constructed response items in assessment offers a far better prognosis for equitable assessment than a return to multiple choice testing. The use of extended constructed responses opens the possibility to let students reveal their logic and argue from their lived realities (as well as from the realities of others). We see the requirement of written explanations or well constructed convincing arguments as a means to allow students to demonstrate what they know. We also see it as a means to change from a staid objective view of mathematics to what Tate (1996) argues for, mathematics "as a tool to guide social decision-making influenced by the values of those who use it in human affairs" (p. 187). The mathematics education community is well into understanding and grappling with the challenges of using extended-constructed tasks in testing and the concomitant cultural, social and political implications.
Champagne, A. B., & Kouba, V. L.: 1997a, Communication and reasoning in science literacy. In S. Sjoberg & E. Kallerud (Eds.), Science, Technology and Citizenship (pp. 75-92). Oslo, Norway: Norsk institutt for studier av forskning og utdanning (NIFU).
Champagne, A. B., & Kouba, V. L.: 1997b, Assessment in the service of education: Assessment of science literacy standards. In Y. Bum-Hong et al. (Eds.), Globalization of Science Education: Moving Toward Worldwide Science Education Standards (pp. 167-172). Seoul, Korea: Korean Educational Development Institute.
Cummins, J.: 1981, The role of primary language minority students. In California State Department of Education (Ed.), Schooling and Language Minority Students: A Theoretical Framework (pp. 3-49). Los Angeles: Evaluation, Dissemination and Assessment Center, California State University.
Dossey, J. A., Mullis, I. V. S., & Jones, C. O.: 1993, Can Students Do Mathematical Problem Solving? Results from Constructed-Response Questions in NAEPs 1992 Mathematics Assessment. Washington, DC: National Center for Education Statistics.
Kenney, P. A. & Lindquist, M. M.: in press, Students performance on thematically related NAEP tasks. In E. A. Silver & P. A. Kenney (Eds.), Results from the Seventh Mathematics Assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.
Kouba, V. L.: 1999. Multiple interpretations = more challenges. Mathematics Teaching in the Middle School, 5(4), 232-235.
Mestre, Jose P.: 1981. 'Predicting academic achievement among bilingual Hispanic college technical students.' Educational & Psychological Measurement, Vol. 41, pp. 1255-1264.
Morgan, C.: 1998, Assessment of mathematical behavior: A social perspective. In P. Gates (webmaster), First Mathematics Education and Society Conference Proceedings. Retrieved Oct 29, 1999 from World Wide Web: http://www.nottingham.ac.uk/csme/meas/papers/morgan.html
National Center for Education Statistics (NCES): 1999, The Nations Report Card: What Is NAEP [Sample Questions]. Retrieved October 20, 1999 from World Wide Web: http://nces.ed.gov/nationsreportcard/ITMRLS/INTRO.SHTML
Tate, W. F.: 1996, Mathematizing and the Democracy: The Need for an Education that is Multicultural and Social Reconstructionist. In C. A. Grant & M. L. Gomez (Eds.), Making Schooling Multicultural: Campus and Classroom. Englewood Cliffs, NJ: Merrill.
The Center on English Learning and Achievement