Archive | Evaluation RSS feed for this section

Synthesis Post: What did I learn from the Annotated Bibliography?

First of all, let me say that after reading classmates’ bibliography posts, I’m convinced I must read this publication in its entirety. For this post, however, I chose to focus on readings reviewed by Maury and Summer. I selected Maury’s … Continue reading

Multimodal Writing Assessment_Annotated Bibliography

Eidman-Aadahl, Elyse, Kristine Blair, Dànielle Nicole DeVoss, Will Hochman, Lanette Jimerson, Chuck Jurich, Sandy Murphy, Becky Rupert, Carl Whithaus, and Joe Wood. “Developing Domains for Multimodal Writing Assessment: The Language of Evaluation, the Language of Instruction.” Eds.  Heidi A. McKee and Dànielle Nicole DeVoss. Digital Writing: Assessment and Evaluation. Logan, UT: Computers and Composition Digital Press/Utah State University Press, 2013. Web. 30 Jan 2014.

This article, written by the Multimodal Assessment Project (MAP) committee for the National Writing Project’s Digital Is…Initiative, identifies five domains they believe are “vital for the assessment of digital writing offers the possibility that the language of assessment can inform—and build upon—discussions more often associated with interaction, instruction, and text creation than with evaluation”: 1) artifact (“the finished product”), 2) context (“the world around the artifact, around the creation of the artifact, and how the artifact enters, circulates, and fits into the world”), 3) substance (“the content and overall quality and significance of the ideas presented”), 4) process management and technique (“the skills, capacities, and processes involved in planning, creating, and circulating multimodal artifacts”), and 5) habits of mind (“patterns of behavior or attitudes that reach beyond the artifact being created at the moment”). The authors elaborate on the definitions of these five domains by centering each one on a specific example of a student(s) project that they feel best exemplifies each domain. My favorite example was that of context with the example being a class/individual project of creating war poems based on The Things They Carried and the class’ microblog. Each example represents a multimodal project incorporated into a grade school classroom and the strengths and weaknesses that were demonstrated in the final “artifact.” MAP, with its emphasis on the five domains as the focal point, suggest that such projects could encourages students to become active producers of rhetoric and more aware of how they and others are using rhetorical strategies, and be a progressive step towards a better understanding of how to evaluate multimodal projects that go beyond the uniform writing skills that are being taught in accordance with standardized testing.

This article was very surprising in just how creative the grade school multimodal project instructions and productions were. The aim of the MAP committee’s project was interesting in that it founded itself upon multimodality, with an emphasis on how students and teachers were operating within rhetorical genres. I was heavily reminded of the reading by Miller (with her idea of fluid motivations for genres) and Popham (with her dissection of boundary genres and the cohesion and tension such meldings can bring). The Google Earth Historical Figures Tour (not quite the right name for the project, but still) was a fascinating boundary genre because students were drawing on skills learned in history and English courses, using technology as the go-between despite that level of technology being an unexpected surprise to the teachers who had simply expected students to use the more traditional PowerPoint. The article itself presented alternatives to the standardized writing assessments that are lacking in many areas that multimodal writing projects could unveil, such as a student’s growing awareness of situational needs that would play into the rhetorical choices they make with their work. By picking and choosing among existing projects being done in various schools (and even a camp setting at a university) the authors did a nice job of uncovering just how well K-12 grade students can learn to adjust and even thrive with writing assignments that go beyond the traditional print platform and the new skill sets that emerge when rhetorical forms are being engaged by youths who are learning to navigate what can be accomplished with advancements in technology while still retaining and honing skills that can be learned with more traditional modes of communication.

In terms of the Theories of Networks course, this article seems especially useful in that multimodal writing assignments create a network unto themselves between different kinds of technologies, students and teachers, students and students, and students, teachers, and larger audiences involved. One particular example in the article that stood out to me in terms of how a project discussed by the authors became more than just a project but a network of skills and participants/creators was that of the website created by the girls at a technology camp. The girls were broken into groups of 3 and given a short amount of time to create a website that reflected their own emerging digital identities but also that of the camp itself. By collaborating so regularly on specific projects, the girls became a network of shared skills and well as a collective intelligence about the project and camp as a whole. This network then grew larger as the website was put on display as the final project to visiting family members, but then attained a greater sense of network as the aim of the project was to be used for potential campers and their parents. The girls involved essentially created a network between themselves as existing campers, the camp leaders, and anyone interested in the camp as a camper or as the guardian of a potential camper. The multimodality of the project is a physical representation of the collaborative nature of the camp and the relationships between the campers that made the website possible.

Music with Which to Celebrate Multimodality

Assignment: Annotated Bibliography Part 1 – Entry


Zoetewey, Meredith W., W. Michele Simmons, Jeffrey T. Grabill. "Assessing Civic Engagement: Responding to Online Spaces for Public Deliberation." Digital Writing Assessment & Evaluation. Eds. Heidi A McKee and Danielle Nicole DeVoss. Logan, UT: Computers and Composition Digital Press/Utah State UP, 2013. Web. 3 Feb 2014.


The primary concern explored in this chapter is the evaluation of student-produced civic web sites; these web sites are defined as "community-based digital spaces that can be used to enable public deliberation."

The authors explain that there is an inherent difficulty with assessing civic web sites in that usefulness - the ultimate goal of a civic web site - can only be determined after the public has had a chance to utilize the site. However, a student will need a grade at the end of a semester before the public has had that chance to interact with the final project students produce. If the site cannot be evaluated based on its usefulness to the public, then what criteria and methods can be employed to determine a fair evaluation?

The authors are clear to distinguish between "usefulness" and "usability." Usability is a site that is easy to use, makes efficient use of graphics and interfaces. Usefulness is the idea that a site helps users "do better work"; the site supports learning and knowledge-making.

Civic web sites can be difficult to assess because traditional rubrics favor usability when evaluating digital products; concepts like simplicity, scanable content, and efficiency are privileged. However, civic web sites have a different purpose, which is to provide users with deeper educational content, unfettered access to data, and communicates in specialized vocabularies specific to the field. These criteria are often incompatible with traditional usability rubrics, requiring an alternative method for evaluation.

The authors explain their concept of "productive usability" which accounts for these advanced needs and is based on the features that users of civic web sites have identified as being important to the work they wanted to do. Productive usability is based on three main criteria: consideration of alternative use, consideration of technical literacy, and consideration of interactivity.

Alternative use is the idea that site creators should consider the multiple ways in which the information may be of use to the public, and they should plan for the kinds of alternative uses that might occur that differ from the creators' intentions. For example, the creators may want a civic web site that provides information for policy makers, but the site may also be visited by citizens looking to become involved in a cause. The site should be evaluated based on whether the creators accounted for the multiplicity of purpose in visiting the site.

Technical literacy is the idea that the civic web site is educational at its core, and if people visiting the site are to be able to participate in the discourse, they must have the fluency in the specific jargon used therein. There must also be an inclusion of full-text reports and other data that can be read and interpreted by the user. Technical literacy is the term applied to the jargon and data, and it should be evident on the site in links to data or the inclusion of glossaries.

Interactivity is the idea that visitors to the site must be engaged in order to participate in the discourse surrounding the particular civic issue being highlighted online. Interactivity is the pathway to engagement by building emotional and psychological connections to the issue. The site should encourage various forms of interaction to build the sense of engagement and ultimately action. Evidence might take the form of photographs designed to induce an empathetic response or a place devoted to user uploaded narratives.

Finding indicators of the three criteria above should also be combined with evidence of the students' process in creating the site. Process-based evidence might include journal-style work logs, which track the groups' discussion and implementation of the three criteria.

By evaluating a civic web site based on the indicators and process-based evidence of the three criteria of productive usability, instructors will be able to confidently assess student-produced digital spaces before the public has a chance to determine their usefulness outside the academic exercise.

Connections to Course Readings:

  • The article describes civic web sites as having "the potential to aid change in communities." This is reminiscent of Miller's work with genre. She explains that we can view genre as being able "to marshal linguistic resources for the sake of social action" (71 "Rhetorical Community"). She continues to argue that genres help "communities do their work and carry out their purposes" (75 "Rhetorical Community"). There are also connections to Popham's claim that "genres are the means by which things get done within a community, ideas get transmitted, and plans get made" (281-2). Both civic web sites and genres are significant because of the work they can perform with a community, for their ability to create action and change. 
  • This ability for the audience to perform social action is also linked to the claims by Bitzer. He argues that the "rhetor alters reality by bringing into existence a discourse of such a character that the audience, in thought and action, is so engaged that it becomes mediator of change” (4). Here, the rhetor can be understood as the makers of civic web sites. They bring the site into existence much like Bitzer's rhetorical discourse and the audience reached becomes engaged in the topic enough to foster change. The term engaged is significant especially since interactivity to increase engagement is on of the three criteria the article proposes to use for evaluation. Furthermore, the civic web site is evaluated based on usefulness - the concept that the user can "do better work". This doing of work by the audience is the same as Bitzer's argument that "rhetorical discourse produces change by influencing the decision and action of persons who function as mediators of change" (7). Civic web sites provide the tools and information needed for the audience to do work that will ultimately change a community.
  • The concept of alternative use is related to Popham's ideas about boundary objects. Popham states that "a boundary object serves the needs of multiple sites or multiple professions" (284). This is exactly what makers of civic web sites must take into consideration under the article's proposed evaluative criteria. The site should be agile enough to meet the needs of users that the site-creators may not have in mind as the initial audience; it should be a boundary object that can serve the needs of multiple users. There is also the Bazerman idea from the chapter "Systems of Genres and the Enactment of Social Intentions" embedded here that there is a "multiplicity of action" (90) in texts with multiple intentions and interpretations possible from the same speech-act (87). Alternative use suggests that civic web sites have multiplicity, the potential to be useful to the community in more than one way. 
  • We can also see connection to Foucault in the discussion. Consider the following quote from the article: "[The alternative use criterion] requires seeing the audience in context, considering how the information relates to a range of stakeholders, and positioning them as active participants, capable of exploring their own interests." In this sense, the civic web site becomes meaningful in terms of its relations. Foucault is interested in scholars' ability to "analyse the interplay of [concepts'] appearances and dispersion" (35). Creators of civic web sites are evaluated on how well they can analyze how information relates to various audiences, how the active participants will be positioned against the information provided, and where they will be dispersed throughout the field.
  • Foucault is also present in interactivity as the concept is grounded in the theory that there should be "no pre-set entry points or stopping points" (Mirel qtd. in Zoetewey). This is discussed in context of the notion that interactivity and engagement can be reached through user-guided explorations. The web site should be designed in such a way to support this free exploration without pre-set starts and stops. This is like Foucault's ideas about general history versus total description. He writes, "A total description draws all phenomena around a single center - a principle, a meaning, a spirit, a world-view, an overall shape; a general history, on the contrary, would deploy the space of dispersion" (10). By building in a principle of free exploration to enhance engagement, the civic web site is less structured or restrictive, much like the general history using the "space of dispersion". Users are allowed to move as they will through the dispersed space, leading himself or herself to the knowledge and connections important to them. 

Works Cited:

Bazerman, Charles. “Systems of Genres and the Enactment of Social Intentions”. Genre and the New Rhetoric. Eds. Aviva Freedman and Peter Medway. London: Taylor and Francis, 1994. 79-104. Print.

Bitzer, Lloyd F.. “The Rhetorical Situation”. Philosophy & Rhetoric 25 Selections from Volume 1 (1992): 1-14. Print.

Foucault, Michel. Archaeology of Knowledge. New York: Vintage Books, 2010. Print.

Miller, Carolyn R.. “Rhetorical Community: The Cultural Basis of Genre”. Genre and the New Rhetoric. Eds. Aviva Freedman and Peter Medway. London: Taylor and Francis, 1994. 23-42. Print.

Popham, Susan L.. “Forms as Boundary Genres in Medicine, Science, and Business”. Journal of Business and Technical Communication 19 (2005): 279-303. Print.

Zoetewey, Meredith W., W. Michele Simmons, Jeffrey T. Grabill. "Assessing Civic Engagement: Responding to Online Spaces for Public Deliberation." Digital Writing Assessment & Evaluation. Eds. Heidi A McKee and Danielle Nicole DeVoss. Logan, UT: Computers and Composition Digital Press/Utah State UP, 2013. Web. 3 Feb 2014.