A group of ten graduate students of literature education and instructional technology, together with the project directors (the authors) and the directors of the Literature Center, developed criteria and procedures for evaluating the content of multimedia literature applications in terms of their inherent capacity to represent and support response-based pedagogies. Eight evaluative categories were established through a series of five focus group sessions. While responses within each category (except classroom usage, see below) included ratings on a ten-point scale for comparative purposes, reviews were essentially narrative in form to encourage the same kind of critical thinking about the hypermedia applications that we would hope teachers would encourage about literature. Within this framework, however, reviewers were asked to answer specific questions and look for particular features or kinds of features in each of the evaluative categories. These fell roughly into three groupings -- technical concerns, response-based considerations, and classroom issues.

It is entirely possible that a hypermedia literature application might be excellent from an instructional technology viewpoint, but deal with literature in a manner that was not at all response-based. To distinguish between the two, the first three evaluative categories considered technical concerns related to multimedia/hypermedia in general. They examined the general quality of programs without considering them from a response-based perspective and include (Swan & Meskill, 1995):

1. Content clarity is concerned with the general accuracy, completeness, and appropriateness of an application for the given population, and with whether the structure of a program and its use of hypermedia are appropriate to its content.
   

2. Technical quality is concerned with a program's user interface; in particular, with its navigational systems, its use of multimedia, and its ease of use. This category also asks whether an application's use of multimedia is intrinsic (serves to enhance content) or extrinsic (decorative), and whether or not it is aesthetically pleasing overall.
   

3. Use of technology is concerned with whether an application makes good use of multimedia/hypermedia technologies or if its content could be just as well or better presented using more conventional means. It is particularly concerned with the multimedia aspects of particular applications, but also looks for such uniquely computer-based functions as nonlinearity, internal coaching, construction tools, and student management.
   
   Response-based Considerations
   

   There is some reason to believe that a unique characteristic of the computing medium is its ability to represent cognitive processes in ways that support their internalization as habits of thought (Papert, 1983; Salomon, 1981; Swan & Black, 1993). Response-based considerations consider whether the formal aspects of hypermedia literature applications present literary works in ways that might support Langer's (1990) four stances. Grounded in response-based conceptions of knowledge, text, readers, and teachers, these include (Swan and Meskill, 1995):
   

4. What counts as knowledge? is concerned with whether a program represents knowledge as constructed or static, as evolving or canonical. In this category, reviewers were asked to consider whether a program is capable of incorporating students' responses to a work of literature, whether it includes multiple perspectives on that work, whether it promotes linkages between the text and students' experiences, and whether it encourages an analytic or an exploratory approach to literary understanding.
   

5. The role of the text is primarily concerned with the way meaning is represented in relationship to the text. In this category, evaluators were asked to consider whether multiple meanings or interpretations are provided, and whether or not a program makes some provision for students to develop their own interpretations of a work.
   

6. The role of the students considers the degree of student control over a program. It asks whether a program contains tools for student construction, whether and how a program validates students' responses to a literary work, and whether or not a program supports student discourse.
   

7. The role of the teacher is concerned with whether software design validates and supports the teacher's role as guide, facilitator, and responder. It considers, therefore, whether and how a program can be modified by a teacher, whether it includes teacher materials and/or internal management tools, and whether or not a program promotes student-teacher discourse and/or interaction.
   
   Classroom Issues
   

   Finally, how multimedia/hypermedia materials are used ultimately determines their effectiveness. Bad materials can be used well; good materials can be used poorly. Group members felt, therefore, that a category should be included that dealt with classroom usage.
   

8. Classroom usage, then, is concerned with how an hypermedia application might be used in a classroom to support literary understanding.
   

Evaluators were also asked to isolate specific features and multimedia/hypermedia tools that might support response-based teaching and learning. These were reduced to eleven general features that might reinforce response-based pedagogies as follows (Meskill & Swan, 1995):

1. Transparent navigation. Evaluators found that if it were not clear how users moved through an application, students and teachers became easily disoriented and frustrated. On the other hand, they noted that too much constraint, however transparent, can potentially inhibit and even drown out students' and teachers' voices.
   

2. Intertextuality and juxtaposition. Another desirable attribute for supporting response-based practice was found to be some mechanism through which a variety of media elements could be interrelated and/or juxtaposed to represent contrasts, similarities, and relationships between and among texts.
   

3. Facility to share responses. It was felt that one of the most powerful features of multimedia/hypermedia technology for supporting response-based literature teaching and learning was its potential capability to facilitate the sharing of student responses on-line. The medium, reviewers believed, could represent multiple threads of conversations around students' reading and writing experiences in ways that would be at best cumbersome in traditional paper-and-pen formats.
   

4. Facility to support non-text responses. The empowering aspect of adding visual and aural support to one's imaginings and understandings has long held appeal in the language arts classroom (Purves, Rogers & Soter, 1990). An aspect of response-based practice that evaluators felt multimedia might nicely complement, therefore, was the use of visual and aural media to illustrate and reflect student envisionment building.
   

5. Facility to make links. A key tenet of response-based approaches is that readers make connections between what they read and their own knowledge and experience. Making such connections might be encouraged and supported by hypermedia tools that allow for on-screen linking. Visual representations of student-constructed connections are valuable both in terms of the processes evoked in their construction and their role in shared discourse.
   

6. Support for envisionment building. The provision of tools with which students can create, edit, refine, and reinterpret representations of their personal envisionments using the full range of available media was deemed highly desirable from a response-based perspective. Reviewers saw clear benefits for both the public and collaborative use of such tools and their use by individuals to develop their own interpretations of texts being explored.
   

7. Access to multiple perspectives. Another key tenet of response-based approaches is the open-ended nature of text as regards individual interpretation. From a response-based perspective, then, a desirable feature is that no single authorial voice predominate, but rather that multiple voices, meanings, and interpretations be provided.
   

8. Support for discourse. An ideal role for multimedia in response-based classrooms is as a catalyst for discussion and socially mediated discovery. Differing points of view are a source of delight, and divergent imaginings are the optimal vehicle for discovery and growth among conversation participants. Multimedia and hypermedia programs, reviewers felt, ought to be designed to stimulate student-to-student and student-to-teacher discourse around literature.
   

9. Promotion of student ownership. Reviewers felt strongly that applications which presented only "canonized" knowledge and/or interpretations of text were antithetical to the goals and processes of response-based practice. Without explicit provision for student entry into textual worlds, multimedia/hypermedia technology can inhibit rather that induce imaginings. Such provision might include tools for students to annotate, extend, and build discourse threads of their own around a literary work.
   

10. Presentation of background knowledge. One of multimedia/hypermedia's strongest features is its capacity to store and display large amounts of textual, aural, and visual information. Evaluators believed that the technology was thus well-suited to the provision of large stores of supporting information that could be accessed by students, as needed, to fill in gaps in their experience, and to help stimulate and enhance student envisionment.
    

11. Facility to explore the author's craft. Hypermedia's capacity to store and display large amounts of textual, aural, and visual information also allows for craft commentary in a range of media formats to which students can have ready access during various stages of engagement with a literary work. As such, it can stimulate and enhance students' awareness and appreciation of literary devices and the author's craft.

Applications for review were identified through a detailed search of listings dedicated to hypermedia materials, such as the Multimedia and Videodisc Compendium (Pollack, 1994) and Multimedia '94 (Educational Resources, 1994), as well as vendor catalogs that included educational hypermedia. Fifty-four multimedia/hypermedia literature programs and/or program series were identified, and forty-five were acquired from their publishers and reviewed.

The applications we acquired were evaluated by twenty-five graduate students of literature education and/or instructional technology. Most were practicing teachers. Each was given two programs to evaluate and asked to complete a written content analysis of both programs by responding to questions in each criterion, and to provide numerical ratings for each of the first seven criteria according to such analyses. Reviewers were also asked to look for features potentially supportive of response-based practice (from the above list), and to describe those they found. The written evaluations were collected and reviewed for consistency by a group of four graduate students, at which time some changes were made in ratings that were inconsistent with responses to content questions or with the general consensus concerning such ratings. The evaluations were then again reviewed by the project directors who made some changes of their own.