This document discusses a new approach called NP-Q (Notes as Peer-assessed Questions) to improve feedback quality in online self-assessment tests. NP-Q allows students to submit answers in phases, receive peer feedback, and have their contributions peer-assessed. Only the highest rated contributions are selected as feedback. An experiment found high student participation rates in writing explanations and peer assessment. NP-Q shows potential to efficiently generate qualitative, student-provided feedback at scale. Future work includes handling faulty peer reviews and open-ended questions.

1. Reflexive learning, socio-cognitive conflict and peer-
assessment to improve the quality of feedbacks in online tests
Franck Silvestre, Philippe Vidal, Julien Broisin
IRIT, University Toulouse III, France
EC-TEL 2015
September 2015
1

2. Context of research
• Computer-based assessment
• Online self-assessment tests
• Feedback
• Tsaap-Notes
2

3. The challenge
How to qualify students’ contributions in order to
provide enhanced feedback in online self-
assessment tests?
3

4. Benefits of online tests
• Facilitate the provision of frequent formative
assessments [12] [9]
• Provide learners with feedback
- any time,
- from everywhere,
- as often as they want [10]
4
Quality of feedback is a key factor of success for better
learning [2][7][8]

5. Issues about feedback
5
• It is provided by teachers
- teacher’s dialect
- time consuming
• The same feedback is provided
to all learners

6. Tsaap-Notes [18]
• Collaborative note-taking
system
• Micro-blogging, scope and
fragment
• Audience response system
• Notes are taken by students
after the presentation of the
results
6

7. The « Notes as feedback » approach
• To reuse questions asked during lectures
• To reuse notes taken by students
• In order to generate self-assessment tests
providing feedback coming from students notes
7
Semi-automatic

8. How does it work?
8

9. Result in Moodle
9

10. Benefits
• Creation of online self-assessment tests is easier
• Feedback is written in students’ dialect
• Promotion of students’ contributions
• Encouraging results from first experimentation [16]
10

11. Two limits
• The students’ notes are not evaluated nor filtered,
there is no ”quality check” of the feedback =>
qualitative limit
• Only few students participate in the writing of
explanations => quantitative limit
11

12. Toward the NP-Q feature
• Why more than 80% of students do participate in
interactive questions, whereas only 25% of them
take notes?
12

13. Success of interactive questions
Audience Response Systems provide
• formative assessments
• immediate feedback for each student
• positive impacts on learning outcomes
[4,6,14,18]
13

14. How to provide students with immediate feedback
about their explanations?

15. The NP-Q feature
• Submission of an answer in several phases
• Immediate feedback
• Peer assessment of the students’ explanations
• Only best graded contributions are selected for
feedback in online tests
15

16. Phase 1 - First answer submission
16

17. Phase 2 - Socio-cognitive conflict
17
Alternative answer
provided by a peer
The second
submission form

18. Phase 3 - Peer assessment
18
The presentation of
the results
The peer assessment
form

19. Final view
19
The list of graded
explanations

20. In Moodle
20

21. Focus on the algorithm for the explanations’ association
21
a1
[ 100; 2; 92 ]
a2
[ 0; 3 ; 24 ]
a3
[ 100; 3; 0 ]
a4
[ 100; 4 ; 65 ]
a5
[100; 3 ; 86 ]
a6
[ 40; 2 ; 153 ]
a7
[ 65; 1 ; 0 ]
a8
[ 100;2 ; 38 ]
a1
[ 100; 2 ; 92 ]
a4
[ 100; 4 ; 65 ]
a5
[ 100; 3 ; 86 ]
a8
[ 100; 2 ; 38 ]
a2
[0; 3 ; 24 ]
a6
[40; 2 ; 153 ]
A
AcAc a1
[ 100; 2 ; 92 ]
a3
[ 100; 3; 0 ]
a4
[ 100; 4; 65 ]
a5
[ 100; 3 ; 86 ]
a8
[ 100; 2 ; 38 ]
a2
[0; 3 ; 24 ]
a6
[40; 2 ; 153 ]
a7
[ 65; 1 ; 0 ]
CcCc
a1
[ 100; 2 ; 92 ]
a4
[100; 4 ; 65 ]
a5
[ 100; 3 ; 86 ]
a8
[100; 2 ; 38 ]
a2
[0; 3 ; 24 ]
a6
[ 2 ; 153 ]
Ac a1
[ 100; 2 ; 92 ]
a3
[ 100; 3; 0 ]
a4
[ 100; 4 ; 65 ]
a5
[ 100; 3 ; 86 ]
a8
[ 100; 2 ; 38 ]
a2
[0; 3 ; 24 ]
a6
[40; 2 ; 153 ]
a7
[65; 1 ; 0 ]
Cc
Ac
Cc
2
1
fg

22. First experimentation
• 2 students’ groups
• First year of Computer Science Master
• Two different courses (DCLL and MA)
• 3 sessions of 2 hours
• 5 NP-Q questions
22

23. Participation rate
23

24. Socio-cognitive conflict
24

25. Peer assessment
25

26. Conclusion
• Students contributions are now qualified
• High participation rate in written tasks
• Encouraging results from the first experimentation
26

27. Perspectives
• Consideration of faulty peer-assessments
• Consideration of open questions: how to associate
answers in this context?
• Scaling: big auditorium, MOOC
27

28. • Thanks for your attention!
• Questions?

29. Bibliography
1. Biggs, J., Tang, C.: Teaching for quality learning at university: What the student does. McGraw-Hill Education (UK) (2011)
2. Black,P.,Wiliam,D.:Assessmentandclassroomlearning.Assessmentineducation 5(1), 7–74 (1998)
3. Boud, D., Cohen, R., Sampson, J.: Peer learning in higher education: Learning from and with each other. Routledge (2014)
4. Caldwell, J.E.: Clickers in the large classroom: Current research and best-practice tips. CBE-Life Sciences Education 6(1),
9–20 (2007)
5. Dochy, F., Segers, M., Sluijsmans, D.: The use of self-, peer and co-assessment in higher education: A review. Studies in
Higher education 24(3), 331–350 (1999)
6. Gauci, S.A., Dantas, A.M., Williams, D.A., Kemm, R.E.: Promoting student-centered active learning in lectures with a
personal response system. Advances in Physiology Education 33(1), 60–71 (2009)
7. Hattie, J., Jaeger, R.: Assessment and classroom learning: A deductive approach. Assessment in Education 5(1), 111–122
(1998)
8. Higgins, R.: Be more critical: rethinking assessment feedback.. DYE. V, PC4000. (2000)
9. Miller, T.: Formative computer-based assessment in higher education: The effectiveness of feedback in supporting student
learning. Assessment & Evaluation in Higher Education 34(2), 181–192 (2009)
10. Nicol, D., Milligan, C.: Rethinking technology-supported assessment practices in relation to the seven principles of good
feedback practice. Innovative assessment in higher education pp. 64–77 (2006)
11. QTI, I.: Question and test interoperability (2006)
12. Ricketts, C., Wilks, S.: Improving student performance through computer-based assessment: Insights from recent
research. Assessment & evaluation in higher education 27(5), 475–479 (2002)
13. Scouller, K.: The influence of assessment method on students’ learning approaches: Multiple choice question
examination versus assignment essay. Higher Education 35(4), 453–472 (1998)
14. Shaffer, D.M., Collura, M.J.: Evaluating the effectiveness of a personal response system in the classroom. Teaching of
Psychology 36(4), 273–277 (2009)
15. Silvestre, F., Vidal, P., Broisin, J.: Tsaap-notes–an open micro-blogging tool for collaborative notetaking during face-to-
face lectures. In: Advanced Learning Tech- nologies (ICALT), 2014 IEEE 14th International Conference on. pp. 39–43. IEEE
(2014)
16. Silvestre, F., Vidal, P., Broisin, J.: Semi automatic generation of online tests providing feedbacks based on collaborative
note taking. In: Advanced Learning Technologies (ICALT), 2015 IEEE 15th International Conference on. IEEE (to appear)
17. Topping, K.: Peer assessment between students in colleges and universities. Review of educational Research 68(3), 249–
276 (1998)
18. Uhari, M., Renko, M., Soini, H.: Experiences of using an interactive audience response system in lectures. BMC Medical
Education 3(1), 12 (2003)