Robert S. McLean <RMcLean@oise.on.ca>
Associate Professor
Ontario Institute for Studies in Education/University of Toronto
http://www.oise.on.ca/~rmclean
252 Bloor St. W., Toronto, Ontario M5S 1V6 Canada
As public and educator awareness of the Internet and the World Wide Web grows, students will increasingly submit their assignments as Web pages. As the educational and reference value of the Web increases, this is appropriate. Used appropriately, student assignment Web documents can benefit from links to the mushrooming collection of timely information resources that may only be available in this form. Students can work in a contemporary medium and practice online research and expression that will be useful to them as the world's intellectual property migrates to networked form.
However, faculty face a myriad of challenges. Those who have spent their career receiving assignments as term papers and similar tangible products may find that assessment of Web documents presents many problems. Assessing students' Web products is a major task and may be more time-consuming than assessing the traditional product. Web-based assignments will limit the environment in which the instructor can do the evaluations, requiring use of a computer and often a continuous connection to the Internet as well.
This paper raises some issues regarding assessment of Web-based assignments. It is based on experience evaluating the work of more than 100 graduate students in an "Internet Resources for Education" course offered several times in the last three years. Each student completed up to three assignments and submitted them as Web pages: (1) a personal home page, (2) a collection of Internet resources on a particular educationally significant topic, and (3) presentation of an "issue" concerning the use of the Internet in education. Students were encouraged to post their work publicly, for constructive critique by their coursemates and for contribution to the world's collection of Web pages. A selection of these projects from assignments 2 and 3 can be viewed at http://www.oise.on.ca/courses/1514/ and a recent course outline is at http://www.oise.on.ca/~rmclean/1514.html
In brief, this paper will urge that instructors should be clear about the acceptable characteristics and scope of Web-based assignments, specifying (or at least having a clear idea themselves) the extent to which evaluation can be influenced by the new characteristics of this medium, the form that the presentation can take, and the permissible extensions to basic Web facilities used. It then discusses many aspects of assessment dictated by the characteristics of the Web: intangibility, limitless linking, ephemeral content, instant alteration, unlimited copyable materials, etc. It also tries to suggest ways of coping with the dynamic nature and constant evolution of the Web, its services, and the inventiveness of students.
The ideas presented here are based on evolving experience, but readers will need to select those aspects that are relevant to their own instructional context.
The most important recommendation is that instructors be explicit about the expectations they have for Web-based assignment products. They should indicate limits and assessment criteria explicitly at the outset. This is always good advice; it is more important (and difficult) in an environment that is new for both professor and student, and that is constantly evolving. The expectations should anticipate a wide range of student expertise, facilities, and enthusiasm for the new medium, as well as accommodating technical developments and even new Web services that may be introduced during the course.
The Web and its technology are evolving rapidly. Some students are likely to be advantaged in terms of Web skills, the hardware and software to exploit it, availability of quality surfing time, and willingness to exploit the medium's characteristics to meet their educational goals. Other students will be relatively disadvantaged; a fundamental question is whether and how these technical and skill advantages will be credited in the evaluation. Even more challenging is the likelihood that several students will be significantly advantaged in comparison to the instructor; some are certain to have superior Web savvy. The extent to which this is relevant to the assignment and is to be rewarded may be an important consideration.
What is the contribution of form, function, and content? Except in courses that specifically teach the Web or Internet technology as topic, one would expect content to take preeminent importance. Subject matter ideas and their expression are most important. But "their expression" leads to a limitless set of possibilities when a Web document replaces the traditional term paper.
At a superficial level, the presentation possibilities are enhanced, since Web browsers take charge of the rendering of the document. Just as word processing has raised the presentation quality of term papers, evolution of Web technology has meant that course Web products can be more elaborate and have intricate and professional presentation. Community standards for expression are also evolving; just as the word-processed essay has displaced the hand-written one, Web technology raises expectations again.
Some Web phenomena are incidental to the learning intended in an assignment. To what extent will gratuitous use of the form and features of Web presentation be tolerated or rewarded? True, the Web offers rich possibilities for the presentation of ideas in ways that may communicate better and/or link to others' contributions to knowledge. Just as "a picture is worth a thousand words" (but surely that is an average, with tremendous variance), there is great opportunity for new, effective ways of presenting ideas using Web capabilities. Instructors will need to develop principles and rules for deciding when a Web feature contributes to the student's intellectual product and understanding, and when it is used gratuitously.
Given the uneven distribution of skills and creativity within a class, it is difficult to delimit all possibilities in the statement of the assignment. But it may be worthwhile, when setting the assignment, to try to indicate the extent to which presentation will influence the evaluation. Cautionary comments about the use of even simple Web enhancements, such as background color and images, or the use of nonstandard text and link colors may be warranted. For example, when evaluating dozens of assignment products, the evaluator may come to depend upon the link colors as an indication of visited links (red means we've seen this one) and disoriented if the student elects to reverse the colors (so red means not visited).
Further cautionary notes about the technical context in which the evaluation will be done may be in order, since students may have different computer equipment and the quality of document rendering may vary. Assumptions about window size and font availability may be violated. Certain HTML expressions (and errors) may be rendered differently or ignored in different browsers. In some cases, this may result in portions of the student's work being invisible or rendered in a manner they did not intend. One possibility is to specify a "reference platform and browser" that will be used for evaluation and to provide access to that combination for verification of student work.
The expressive possibilities of Web documents are expanding without bound; the attendant complexity is growing apace. Once one goes beyond simple text documents containing static images and hypertext links to other similar documents, the technical aspects of evaluation potentially explode.
Multimedia will require that the evaluator's browser be configured with the appropriate helper applications or necessary "plug-in" software extensions. To the extent that the browser market is fragmented and competitive advantage is seen in browser-specific enhancements, this challenge will worsen. Specification of acceptable helpers and plug-ins may be advised.
Even simple features we now take for granted on the Web--such as clickable graphics ("imagemaps"), forms, cgi-scripts, and search engines--pose significant problems for some of the strategies discussed below, since they rule out the evaluators' use of computer tools to make their task manageable. Features that depend on server-side processing will limit the ability to evaluate Web documents offline.
Hypertext links are a fundamental contribution of Web technology. The groundrules for their use in course assignments should be spelled out. Links can serve well as the analog of traditional references, providing the evaluator with immediate access to the information resources used by the student, and doing so in context. In general, links must be to whole Web documents, and the traditional role of page numbers is unfulfilled. Conventions for indicating exactly what part of the document is being referenced probably should be spelled out in the expectations.
It is also useful to develop a common understanding with students of the role of links, and how the role will be indicated. Links can function to support the student's conclusions, attribute direct or indirect quotes, provide general interest information, provide for further searching, etc. Since a link does not inherently present any indication of its function or relation to the current document, it is important that the relationship be explicit in the surrounding text or phrasing of the link.
Adding links when creating a Web document is now a trivial matter. Web search facilities can provide more URLs than could be visited by the evaluator. Thus links, by themselves, mean little. The evaluator will need to find evidence in the Web document to indicate that the student has critical knowledge of the linked document and that it contributes significantly to his or her understanding and expression of ideas.
Some links are not well behaved from an evaluation point of view. Access to some documents may be ephemeral--present when the student created the product, but gone when it is evaluated. Some documents may be generated by servers, on demand, and not retrievable later using the same URL. Access to some documents may require membership or authentication, or "magic cookie" information stored on the student's computer, which will prevent the evaluator from accessing the referenced document. Sometimes students will not be aware of the contextual requirements for access, since they may have been fulfilled in much earlier browsing, even days before visiting the document. Obviously, there are lots of opportunities for students to claim "but it worked for me!"
Of course, URLs can be wrong or outdated. They can suffer transcription faults, and may lead to altered, moved, or expired pages between the time the student document is created and when it is evaluated. The browser's "Get Document Info" capability can be used to adjudicate these occurrences.
What styles of the presentation will be accepted? The Web has blossomed with a myriad of styles. A simple list of links to Web sites might be at one extreme, while a term paper-like text presentation might represent the other extreme. To what extent are graphics permitted and encouraged? Should limits be specified that relate to performance considerations, such as bandwidth limitations?
What is acceptable in terms of depth of hypertext? The evaluator will tire of exploring a document with many levels of tiny subcomponents, but may also find a large single-document format unwieldy. As is true in general Web site design, each page should have a purpose and provide information itself, with clear navigation conventions. General expectations should be made explicit in the assignment.
Evaluating Web documents is complex, compared to traditional term papers. Term papers are tangible; they can be glanced through easily to get an overview of organization and content. No additional technology is required to do this, and the physical characteristics of the document and its rendering give some impressions of what to expect.
Web documents are generally revealed one page at a time, rendered on a computer screen by Web browser software. Typically, pages are browsed one at a time, by following embedded hypertextual links from one page to another. It is not uncommon for users to lose track of the overall organization of the document they started with. While this may be fun for recreational surfing of the Web, it can be a major problem when evaluating student course products.
One technique, if browser and screen size will accommodate it, is to open links in new windows, so that the higher-level document is still visible and can serve as orientation. With the drag-and-drop capabilities of some browsers and operating systems, a specific set of browser windows, representing top-level, second-level, tertiary, etc., links can be set up on the screen in fixed positions as a reminder of their relationships. Links at the top level can be dragged to the second level window for viewing (and so on for subsequent levels), providing contextual information.
Of course, it may be useful to prescribe certain limits of depth of links that will be allowed in assignments, such as a maximum of two (or three) levels of documents, with any further level being links to external reference documents.
Hypertext document technology, by design, encourages an unlimited set of possible browsing paths, while giving little or no indication of the source or context of each document. Thus, it is easy in evaluative browsing to find that one is assessing someone else's subdocument rather than the student's work. This can be avoided by close attention to the URLs of subpages. A student's work for a given assignment should be collected together in one directory, and it may be reasonable to assume or require that URLs that do not have the common base directory are not part of the assignment. This expectation should be announced ahead of time. However, when students do group work, the conventions and expectations for the range of URLs that may be used in the assignment should be announced.
Students often create their Web pages in their home Web directory, without imposing further structure on their site. As a result, casual links to the student's other Web materials may appear to be part of the assignment. Typical examples are a personal home page or resume and Web documents prepared for other courses; this can make a student product very impressive to the unwary evaluator. Clear directions about the hierarchical organization of the Web product for the course and use of the URL stem components to separate other documents will help make this distinction explicit.
In Web hypertext, all links are functionally similar: You click them and you go to a new document. The art of constructing useful Web documents includes making informative link descriptions, indicative of the content that will be found by traversing the link, so that the reader can assess the utility of accessing the linked document. But with just a browser, a link can only be assessed by (1) judging the description of the linked document given by the student, or (2) visiting the linked document and seeing for oneself. The former requires discipline by the student when creating the document and should be rewarded as an important aspect of articulate expression in this medium. The latter will quickly overload the evaluator.
A useful strategy may be to announce that the top-level page should give the complete essence of the Web product, sufficient in itself for assessment purposes, and that subdocuments should be used to develop the details, and that they may be assessed on a sampled basis rather than completely read.
Computer-generated reports of the relationships represented by the links might provide a partial alternative to extensive browsing. Since computers cannot "understand" the content of Web pages, such reports should be used in evaluation situations only for guidance of human browsing.
Nonetheless, it is quite feasible to generate reports that show the extent, structure, links, and technical richness of a collection of Web pages. Some of the metrics might include file size, number of links, number of unique links, types of files linked, cross-references of pages, access protocols employed, HTML level and tags used, acceptability of pages to page verification programs such as weblint, etc. A summary of some of these metrics can be used to alert the evaluator to characteristics that might be missed while browsing. For instance, a large number of links to one site might indicate that the student's search for information was quite limited, the modern equivalent of taking most references from one book.
Such metrics are also useful for cross-student comparisons; they form objective measures of the product. One needs to be careful about the interpretation of these statistics, but they can be used constructively to support appraisals from content examination and to refute appeals of the more subjective judgments that will normally form the main basis of assessment.
An individual's statistics and comparative group statistics can also contribute to the feedback to students, since these reports may have sufficient detail that they can be annotated informatively.
Eventually the evaluator will have to look at actual Web pages. The evaluator should keep in mind that the presentation may vary considerably from what the student saw or intended. Differences in fonts and type size settings, window size, processor speed, display resolution, color depth, browser type and level of HTML support, and even environmental variables will make it unlikely that the documents are rendered as the student saw them. Since the evaluator has generally not seen the documents rendered on other platforms, there isn't too much that can be done about detecting these differences except to be alert to the possibility of such problems.
In most educational settings, a transaction between the student and instructor marks the transfer of the assignment product (a term paper, say) to the teacher to evaluate. That act serves to identify the product and to freeze its characteristics. With a Web assignment, the URL identifies the product, but does not freeze its characteristics, since the student can continue to modify the product. However, in many educational situations it is necessary to identify and freeze the assignment at a particular time, such as the deadline.
The corresponding "handing in" transaction for a Web document may be via e-mail, or the student may declare the product's URL via a Web form. It may be useful to post URLs, titles and abstracts of student projects early in the course and encourage other students to contribute and critique the developing projects.
However, a definitive version is required at the assignment's deadline. Web documents can be changed almost instantly; a professor's assessment takes time.
It may be useful to require an explicit submission of the URL for an assignment. If this is done through a Web form and appropriate cgi script, the act could trigger various automatic events such as automatic archiving of the related pages, thus freezing their state for evaluation. The script could also identify the related URLs thought to be part of the submission, provide the list as a form, and interactively allow the student to be sure that the proper pages and related files are identified. This can obviate misunderstandings of what is included in the submission.
Since the instructor can also lose (or change) files, it may be useful to develop a third-party archiving service that could hold the definitive copy of the Web assignment for resolution of any eventual dispute. I am not aware of such a service, but it is quite feasible to set up such a service in an institution's computer center.
Each new technology brings creative requests for special consideration. Portions of the Internet and Web infrastructure may be unreliable from time to time. Conditions include one's Internet service provider being down, local communications problems, and unreliability of distant Internet sites used in the student's work. Usually, these should be minor annoyances, but if there are firm deadlines for submission of the assignment, some Internet-related events that preclude timely submission will no doubt appear.
One approach to establishing a definitive version at the time of the assignment deadline is to (automatically) make a copy of the Web site as of the deadline. I have used modifications of mirroring software for this purpose. At the announced deadline, a Unix script executes the mirroring software, which makes the definitive copy for evaluation purposes. Students are then free to continue to modify their versions, and the evaluator may or may not pay any attention to the changes.
Mirroring techniques are advantageous because the evaluation copy is independent of the original and hence could be stored on a local hard disk, and much of the evaluation work could be done offline. They also are independent of the Internet service provider, so students can mount their Web sites at the provider convenient to them.
Mirroring software has possible problems, however. In the versions we used, there were instances in which the mirroring software did not recognize certain simplifications of URLs that were acceptable to browser software. Thus there is the possibility that portions of the student's product may not get copied. There also were instances in which certain defective URLs resulted in incorrect or looping behavior.
Depth of mirroring must be considered: Which links are part of this student's document? The Web mirroring we used treated any links that had the same URL stem as leading to parts of the document. Links with differing URL stems were considered to be external references. As has been noted previously, this may not be what is desired if a number of students worked on the product and stored materials in their own directories or if a student used a more elaborate structure.
Recent "off-line" Web browser software might also be used for this harvesting process; one would need to check that the software's harvesting rules will catch the appropriate pages from each student's work.
The discussion so far has assumed that the student products are traditional Web documents that consist of static HTML documents of text and images, and thus a stable set of files can be downloaded. However, some students will produce much more sophisticated and dynamic Web "documents" that depend upon server-side processes for their generation or that respond to user input in some dynamic way.
For instance, one student produced a large database of materials and, rather than creating a big list of them all, created a very simple Web form that allowed access through a Perl script search engine. Other students might use imagemaps in ways that cannot be represented as a line of corresponding menu selections. Some might introduce mechanisms that rely on third party search services. These features rule out simple mirroring and cannot be adequately represented in the automatic statistics generation processes.
As Web capabilities and understanding grow, the complexity and inventiveness of students will also escalate. The evaluation alternatives here seem to require direct interaction with the student's site, explicit anticipation of such sites in the statement of requirements for the assignment, or possibly requiring a separate summary report that gives a description of what is behind the dynamic process (the database size and complexity, the search mechanism, etc.).
The introduction of Java, Javascript, and other client-side processes will further complicate the mirroring aspect, particularly where these processes operate in conjunction with server-side processes
Unfortunately, issues of originality and intellectual property rights often need to be considered when evaluating student work. The assignment specifications should be explicit about the expectations with respect to inclusion of others' text and multimedia in a Web document submission, and the extent to which intellectual property rights are expected to be respected. Considerations of copyright clearance, fair use, and royalty-free license of materials included in a student's work may be more important where the student work is made available for general access over the World Wide Web than when a document is simply handed in to the instructor.
Occasionally students will submit work that has been assembled, excerpted, copied, or borrowed from other sources, rather than their original work. Usually this is an academic violation, and the consequences should be stated in the assignment expectations.
Plagiarism may be a greater problem in Web-based products, since the Internet provides a world of materials to copy and paste into one's own work. Many instances of such plagiarism will be detected by experienced instructors, who sense differences in quality, style, and expression. Identification of the offending material may be aided by use of the Web's full-text search engines. Still, gathering evidence to substantiate a conclusion of plagiarism is labor-intensive at present. It would be interesting to have a search engine to which whole documents (rather than just words or phrases) could be submitted, which would identify Web documents that contain similar passages of text.
Evaluation of student assignments should encourage further learning as well as giving a mark. Traditionally, the instructor has written comments in the margins of the term paper, which is returned to the student. Web-based documents rule out marginal notation, but there would seem to be an opportunity to devise tools for annotation of the content of Web-based assignments. These might include forms-based checklists or rating scales that relate to the objectives of the assignment and have means of identifying the supporting instances in the student's work by simply pointing and clicking. Perhaps floating bubbles of comment could be provided for the evaluator's use, with the resulting Web document returned to the student, who would use a Web browser to read the comments. Frequently used evaluations might be facilitated by computer processes and Web form elements. Perhaps some of the current crop of Web authoring tools could be used in creative ways to provide comments on students' documents.
Three years' experience with marking Gopher and Web-based student assignments has presented many challenges. Web-based assignments currently take longer to evaluate and constrain the environment in which the instructor can do the evaluation. Computer tools to help in the process are currently rudimentary, and need to be assembled or modified extensively by the user.
However, Web-based assignments are here to stay; they correspond to an increasingly important mode of expression, have considerable motivational value, allow sharing of students' work beyond just the instructor's evaluation, relate more directly to reference sources that are also in the Web medium, and provide a range of innovative means of expression.
There already is a demand from some students to allow this mode of expression, and the challenge to instructors will be to set assignment expectations that encourage the most intellectually useful aspects of Web-based assignment completion, while still affording an evaluation process that is fair, contributes to further education, and is manageable.