Your SlideShare is downloading. ×
Is peer grading a valid assessment for mooc
Prochain SlideShare
Chargement dans... 5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Is peer grading a valid assessment for mooc


Published on

Published in: Éducation, Affaires, Technologies
0 commentaires
0 mentions J'aime
  • Soyez le premier à commenter

  • Be the first to like this

Aucun téléchargement
Total des vues
Sur Slideshare
À partir des ajouts
Nombre d'ajouts
Ajouts 0
No embeds

Signaler un contenu
Signalé comme inapproprié Signaler comme inapproprié
Signaler comme inapproprié

Indiquez la raison pour laquelle vous avez signalé cette présentation comme n'étant pas appropriée.

No notes for slide


  • 2. The Problem of Assessment Why is assessment needed in MOOCs? ◦ Differs a MOOC from an educational website ◦ Closes the loop Why is assessment difficult in MOOCs? ◦ Massive - workload ◦ Open – heterogeneous student population ◦ Online – technology affordances ◦ Course – reliability and validity emphasized
  • 3. Why peer grading might be the solution Automated Grading ◦ Pros: immediate, easy-to-setup ◦ Cons: no feedback, can’t assess complex assignments Teaching Assistant Grading ◦ Pros: accurate, meaningful feedback (maybe) can assess complex assignments ◦ Cons: expensive, takes time, “unsustainable” Local Learning Center (blended) ◦ Pros: accurate, meaningful feedback ◦ Cons: difficult to build collaboration, only accessible to certain students. Pros ?? Peer Grading
  • 4. Findings from Literature Review Peer Grading Reliability ◦ Measured by the agreement among peer graders ◦ Generally positive findings regarding reliability ◦ Calculated by Pearson r, Proportion of Variance, or Intraclass Correlation ◦ Sometimes mistaken for validity, much smaller body of research Peer Grading Validity ◦ Measured by the similarity between peer grading and instructor grading ◦ Positive findings reported in various contexts Peer Grading in MOOCs ◦ Few empirical studies ◦ Mixed experiences reported from both instructors and students
  • 5. Coursera’s Peer Grading System Each submitted assignment is randomly graded by 3-5 peers. Instructor provides the grading rubric and builds it into the Peer Grading System. Evaluation can be quantitative, qualitative, or both. Students can grade their own assignment. Median Score is used as the final peer grading score.
  • 6. The Context of Research (MGR)
  • 7. Research Questions: RQ1: Can Coursera’s peer review system provide a reliable and valid assessment of student assignments in a MOOC? RQ2: Does the use of median score provide a more valid assessment than the use of mean score when calculating the final peer grading scores? RQ3: What are the perceived effects of peer grading on students’ MOOC learning experience?
  • 8. Data Source The structure of peer grading assignment data in Coursera’s database
  • 9. Data Source Attributes of a submitted peer grading assignment
  • 10. Finding - Reliability Intraclass Correlation 95% Confidence Interval F Test with True Value 0 Lower bound Upper bound Value df1 df2 Sig Single Measures .262 .240 .284 2.774 1824 7300 .000 Average Measures .640 .613 .665 2.774 1824 7300 .000 Intraclass Correlation Coefficient (Case 1) for Peer Grading Scores (N=1825)
  • 11. Finding- Reliability Clarity of the Presentation Convincingness of the Story Quality of the Cartography Aesthetics of the Design Single Measures .216 .215 .176 .210 Average Measures .579 .578 .516 .571 Intraclass Correlation Coefficients (Case 1) for the Four Criterion Scores (N=1825)
  • 12. Finding-Validity instructor_grading peer_grading_median peer_grading_mean self_grading instructor_grading 1 .619** .662** .341** peer_grading_median 1 .952** .279** peer_grading_mean . 1 .464** self_grading 1 Pearson’s Correlation Coefficient between Instructor, Peer, and Self Grading Scores (N=93) **. Correlation is significant at the 0.01 level (2-tailed)
  • 13. Finding-Perceived Effects Question Questions (1-5 from strongly disagree to strongly agree) N Positive * Mean The peer review process helped me to grow and improve as a spatial thinker. 2121 63% 3.67 I received fair grades on my Lesson 5 Mapping Assignment from my peers. 1694 62% 3.75 The feedback my peers gave me on my Lesson 5 Mapping Assignment was useful. 1719 61% 3.69 The peer grading activity in this course made me feel more engaged in the course. 2027 63% 3.70 The peer grading activity in this course made me feel more connected with other students. 2039 57% 3.57 The peer grading activity in this course provided me with opportunities to review and/or reflect on the course content. 2044 72% 3.88 I would recommend keeping the peer grading assignment in future offerings of this course. 2185 70% 3.90 *. Agreed or strongly agreed responses from the students are considered as positive responses Students’ Ratings of the Seven Statements Regarding the Peer Grading Activity in MGR
  • 14. Conclusion The inter-rater reliability of peer grading is low. The use of multiple student graders can significantly increase the peer grading reliability. Peer grading seems to be a valid assessment of student assignment (both median-based or mean-based). Self-grading is less valid and tends to produce inflated scores. General perception towards the peer grading activity in MOOC is positive. Students think it is fair, useful, and beneficial to their MOOC learning experience.