{"id":12789,"date":"2013-01-07t16:30:36","date_gmt":"2013-01-07t21:30:36","guid":{"rendered":"\/\/www.imrbdigital.com\/?p=12789"},"modified":"2013-01-08t10:45:46","modified_gmt":"2013-01-08t15:45:46","slug":"getting-to-know-coursera-peer-assessments","status":"publish","type":"post","link":"\/\/www.imrbdigital.com\/2013\/01\/getting-to-know-coursera-peer-assessments\/","title":{"rendered":"getting to know coursera: peer assessments"},"content":{"rendered":"

by katie mcewen, graduate assistant<\/em><\/p>\n

\"\"<\/a>after getting started talking about assessment methods<\/a> in coursera, in general, today we\u2019ll turn our attention to peer assessments, in particular.\u00a0 peer assessments are designed to evaluate the kinds of unstructured output\u2014essays, projects, videos, music, art, design challenges, etc.\u2014a student might reasonably be required to complete in a traditional course. \u00a0but it is precisely these kinds of open-ended assignments that pose serious problems in a massive online setting more suited to automatically graded quizzes and programming assignments.\u00a0 who, after all, has got the time to read 10,000 essays?<\/p>\n

the answer, for coursera at least, is other students.\u00a0 and while peer assessments have garnered a fair bit of attention<\/a>, they\u2019ve also quickly enough underscored some of the more pervasive, and truly difficult, issues faced by the coursera model<\/a>.\u00a0 plagiarism<\/a> is perhaps only the most obvious one.\u00a0 peer assessments\u2014the ways they work, they ways they don\u2019t\u2014raise serious issues about creating and cultivating community online; navigating questions of authority\u00a0 and language proficiency; and about grading as a reflective practice.<\/p>\n

laura gibbs, an experienced teacher of online courses, provides a thoughtful response to her own experience with peer feedback<\/a> in the recent \u201cfantasy and science fiction\u201d class on her blog coursera fantasy<\/a><\/em>.\u00a0 there, she also addresses some of deeply problematic<\/a> aspects of grading in coursera.<\/p>\n

\"\"<\/a>
"a history of the world since 1300," instructions<\/figcaption><\/figure>\n

unlike automatically graded quizzes and programming assignments, peer assessments require a good-faith effort on the part of each student not only to submit original work in the proper format and the proper language (still largely english), but also to then anonymously evaluate the work of others attentively and constructively.\u00a0 so for each assignment submitted in a course, students are generally then asked to evaluate the work of up to 4 or 5 peers.\u00a0 that\u2019s not a negligible amount of work or time, especially in those courses with weekly or every-other-week peer-assessed assignments.<\/p>\n

for example, students are asked to write short essays in \u201ca history of the world since 1300;\u201d or work through a series of project briefs in \u201chuman-computer interaction;\u201d participate in assignments and design challenges in \u201cdesign: creation of artifacts in society;\u201d or formulate final projects for \u201cintroduction to sustainability.\u201d\u00a0 david owens, a professor at vanderbilt\u2019s graduate school of management, will try out group projects in his upcoming coursera course on \u201cleading strategic innovation in organizations.\u201d<\/p>\n

as we can see, peer assessment is a part of the course requirements for a wide spectrum coursera courses across disciplines, not just those dedicated to literature.\u00a0 in fact, of the 50 coursera courses opened between june to october 2012, 14 (or 28%) required at least one peer-graded assessment.\u00a0 and four of those 14 courses required only peer-graded assessments, with some unexpected titles in the mix: \u201chealth policy and the affordable care act,\u201d \u201cfantasy and science fiction,\u201d \u201ca history of the world since 1300,\u201d and \u201ccomputer architecture.\u201d<\/p>\n

given this, it\u2019s important to keep in mind that feedback on coursera is anonymous.\u00a0 that means you don\u2019t know whose work you\u2019re reviewing or who\u2019s reviewed your work.\u00a0 this makes any actual discussion of the feedback essentially impossible.\u00a0 want to follow up on a comment?\u00a0 or continue the discussion?\u00a0 not easy in a class of 50,000.\u00a0 \u201cmodern and contemporary american poetry\u201d gets around this structural problem by asking students to post their completed and graded essays in the forum (in addition to the peer-assessment system) for more feedback and more engaging discussion.<\/p>\n

\"\"<\/a>
\u201cdesign: creation of artifacts in society,\u201d assignment 1.1: good design<\/figcaption><\/figure>\n

but already, this question of anonymity in coursera\u2014which protects privacy while making it impossible to ask questions, or engage in a direct conversation, about the feedback\u2014points to larger issues of how privacy and pedagogy intersect online.\u00a0 how do we create sustainable online learning communities in coursera if students are not accountable to their peers or for their feedback?<\/p>\n

and when are students supposed to learn how to grade?<\/p>\n

it generally seems that the piece most often missing in peer assessments is not good-faith effort on the part of most students to submit and evaluate work.\u00a0 rather, it\u2019s that many (or even most) students simply do not have experience in evaluating the work of others.\u00a0 and how could they?\u00a0 while some courses offer peer-assessment training, this doesn\u2019t seem to have yet bridged the skill gap.\u00a0 so what are students really getting from this peer feedback?\u00a0 is it helping them to write better essays or to create more complex projects?<\/p>\n

part of the problem is, of course, that grading is difficult no matter the medium: online or face-to-face, seminar or lecture.\u00a0 neither is it a problem limited to students: many instructors likewise lack formal training in evaluating student work.<\/p>\n

and, here, we run up against another unspoken assumption at work in coursera: that grading is a relatively transparent, relatively straightforward process that can be \u201clearned\u201d quickly enough and well enough to be effective online.\u00a0 to guide students, coursera encourages instructors to develop detailed rubrics for evaluation.\u00a0 coursera also presents some limited data suggesting that their rubrics have improved over time.\u00a0 this data<\/a>, or rather the conclusions drawn from it, is far from conclusive.\u00a0 it could be that it is the students\u2019 ability to grade effectively improves over time, or even that only the more dedicated and skilled students continue to participate in peer grading.<\/p>\n

certainly, i don\u2019t doubt the value<\/a> of peer grading.\u00a0 nor do i doubt the ability of some, perhaps even many, coursera students to grade effectively and insightfully.\u00a0 i do, however, doubt that this is what actually<\/strong> happens in coursera.\u00a0 and initial anecdotal evidence — drawn from my own research<\/a> into common practices in coursera, as well as the experiences of others (see here<\/a>, here<\/a>, here<\/a>, and the comments here<\/a>) — would seem to indicate that it is not.<\/p>\n

because as we know, grading is often one of the most difficult aspects of teaching.\u00a0 it is a reflective practice, like any other we undertake in the classroom, changing over time, and requiring dedication, energy, and engagement.\u00a0 ideally, it would also include a commitment to helping others learn and improve.\u00a0 how do we work to cultivate this kind of community culture online?\u00a0 and what might we need to do differently to facilitate community online?<\/p>\n

\"\"<\/a>
"human-computer interaction," project brief 1: change<\/figcaption><\/figure>\n

the real sticking point, for me, isn\u2019t simply the issue of students grading effectively or ineffectively online.\u00a0 rather, it\u2019s that coursera doesn\u2019t quite acknowledge that there is an implicit pedagogy or ideology at work on the platform.\u00a0 which suggests that grading is work to be outsourced, that the division of academic labor operating in most large university programs in the us, like those where the coursera founders work\u2014in which professors teach, graduate students (or machines) grade\u2014is one worthy of replication online.<\/p>\n

by thus separating expertise and grading, coursera would seem to rely on an impoverished conception of grading, which privileges international perspectives over expert critique.\u00a0 the model of peer assessment supported by coursera folds together two assumptions: that peers can approximate or replace the kinds of substantive, constructive expert feedback critical to deeper understanding and that a grade is necessary to learn, full stop.\u00a0 even when credit is not granted<\/a>.<\/p>\n

so although there is no shortage of innovative projects assigned in coursera courses, which ask students to apply and expand their knowledge in exciting, creative, and challenging ways, there is still a lack of sustained conversation around what grading, or peer assessment, means in this online environment.<\/p>\n

next time, we\u2019ll continue our discussion to take a look at one outcome of grading in coursera: certification.<\/p>\n

image: “score cards<\/a>,” marcus hodges, flickr (cc)<\/p>\n","protected":false},"excerpt":{"rendered":"

by katie mcewen, graduate assistant after getting started talking about assessment methods in coursera, in general, today we\u2019ll turn our attention to peer assessments, in particular.\u00a0 peer assessments are designed to evaluate the kinds of unstructured output\u2014essays, projects, videos, music, art, design challenges, etc.\u2014a student might reasonably be required to complete in a traditional course….<\/p>\n","protected":false},"author":452,"featured_media":12843,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","_links_to":"","_links_to_target":""},"categories":[167],"tags":[191],"acf":[],"jetpack_featured_media_url":"https:\/\/cdn.vanderbilt.edu\/vu-wp0\/wp-content\/uploads\/sites\/59\/2018\/07\/09154438\/98844766_b4d0c2ed77_z.jpg","_links":{"self":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/posts\/12789"}],"collection":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/users\/452"}],"replies":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/comments?post=12789"}],"version-history":[{"count":0,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/posts\/12789\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/media\/12843"}],"wp:attachment":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/media?parent=12789"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/categories?post=12789"},{"taxonomy":"post_tag","embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/tags?post=12789"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}