Is peer grading a valid assessment for mooc
Prochain SlideShare
Loading in...5
×

Vous aimez ? Partagez donc ce contenu avec votre réseau

Partager
  • Full Name Full Name Comment goes here.
    Êtes-vous sûr de vouloir
    Votre message apparaîtra ici
    Be the first to comment
    Be the first to like this
No Downloads

Vues

Total des vues
1,268
Sur Slideshare
499
From Embeds
769
Nombre d'ajouts
5

Actions

Partages
Téléchargements
4
Commentaires
0
J'aime
0

Ajouts 769

http://www.scoop.it 642
http://hengluo.wordpress.com 120
http://www.google.fr 3
http://translate.googleusercontent.com 3
http://webcache.googleusercontent.com 1

Signaler un contenu

Signalé comme inapproprié Signaler comme inapproprié
Signaler comme inapproprié

Indiquez la raison pour laquelle vous avez signalé cette présentation comme n'étant pas appropriée.

Annuler
    No notes for slide

Transcript

  • 1. IS Peer Grading a Valid Assessment for MOOC? HENG “PATRICK” LUO & ANTHONY C. ROBINSON JOHN A. DUTTON E-EDUCATION INSTITUTE, PENN STATE UNIVERSITY
  • 2. The Problem of Assessment Why is assessment needed in MOOCs? ◦ Differs a MOOC from an educational website ◦ Closes the loop Why is assessment difficult in MOOCs? ◦ Massive - workload ◦ Open – heterogeneous student population ◦ Online – technology affordances ◦ Course – reliability and validity emphasized
  • 3. Why peer grading might be the solution Automated Grading ◦ Pros: immediate, easy-to-setup ◦ Cons: no feedback, can’t assess complex assignments Teaching Assistant Grading ◦ Pros: accurate, meaningful feedback (maybe) can assess complex assignments ◦ Cons: expensive, takes time, “unsustainable” Local Learning Center (blended) ◦ Pros: accurate, meaningful feedback ◦ Cons: difficult to build collaboration, only accessible to certain students. Pros ?? Peer Grading
  • 4. Findings from Literature Review Peer Grading Reliability ◦ Measured by the agreement among peer graders ◦ Generally positive findings regarding reliability ◦ Calculated by Pearson r, Proportion of Variance, or Intraclass Correlation ◦ Sometimes mistaken for validity, much smaller body of research Peer Grading Validity ◦ Measured by the similarity between peer grading and instructor grading ◦ Positive findings reported in various contexts Peer Grading in MOOCs ◦ Few empirical studies ◦ Mixed experiences reported from both instructors and students
  • 5. Coursera’s Peer Grading System Each submitted assignment is randomly graded by 3-5 peers. Instructor provides the grading rubric and builds it into the Peer Grading System. Evaluation can be quantitative, qualitative, or both. Students can grade their own assignment. Median Score is used as the final peer grading score.
  • 6. The Context of Research (MGR)
  • 7. Research Questions: RQ1: Can Coursera’s peer review system provide a reliable and valid assessment of student assignments in a MOOC? RQ2: Does the use of median score provide a more valid assessment than the use of mean score when calculating the final peer grading scores? RQ3: What are the perceived effects of peer grading on students’ MOOC learning experience?
  • 8. Data Source The structure of peer grading assignment data in Coursera’s database
  • 9. Data Source Attributes of a submitted peer grading assignment
  • 10. Finding - Reliability Intraclass Correlation 95% Confidence Interval F Test with True Value 0 Lower bound Upper bound Value df1 df2 Sig Single Measures .262 .240 .284 2.774 1824 7300 .000 Average Measures .640 .613 .665 2.774 1824 7300 .000 Intraclass Correlation Coefficient (Case 1) for Peer Grading Scores (N=1825)
  • 11. Finding- Reliability Clarity of the Presentation Convincingness of the Story Quality of the Cartography Aesthetics of the Design Single Measures .216 .215 .176 .210 Average Measures .579 .578 .516 .571 Intraclass Correlation Coefficients (Case 1) for the Four Criterion Scores (N=1825)
  • 12. Finding-Validity instructor_grading peer_grading_median peer_grading_mean self_grading instructor_grading 1 .619** .662** .341** peer_grading_median 1 .952** .279** peer_grading_mean . 1 .464** self_grading 1 Pearson’s Correlation Coefficient between Instructor, Peer, and Self Grading Scores (N=93) **. Correlation is significant at the 0.01 level (2-tailed)
  • 13. Finding-Perceived Effects Question Questions (1-5 from strongly disagree to strongly agree) N Positive * Mean The peer review process helped me to grow and improve as a spatial thinker. 2121 63% 3.67 I received fair grades on my Lesson 5 Mapping Assignment from my peers. 1694 62% 3.75 The feedback my peers gave me on my Lesson 5 Mapping Assignment was useful. 1719 61% 3.69 The peer grading activity in this course made me feel more engaged in the course. 2027 63% 3.70 The peer grading activity in this course made me feel more connected with other students. 2039 57% 3.57 The peer grading activity in this course provided me with opportunities to review and/or reflect on the course content. 2044 72% 3.88 I would recommend keeping the peer grading assignment in future offerings of this course. 2185 70% 3.90 *. Agreed or strongly agreed responses from the students are considered as positive responses Students’ Ratings of the Seven Statements Regarding the Peer Grading Activity in MGR
  • 14. Conclusion The inter-rater reliability of peer grading is low. The use of multiple student graders can significantly increase the peer grading reliability. Peer grading seems to be a valid assessment of student assignment (both median-based or mean-based). Self-grading is less valid and tends to produce inflated scores. General perception towards the peer grading activity in MOOC is positive. Students think it is fair, useful, and beneficial to their MOOC learning experience.