Some experiences from evaluating and stress testing digital examination systems
•
1 j'aime•520 vues
A talk given at Inspera 2018 in Oslo about digital examination from a university teacher perspective.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Some experiences from evaluating and stress testing digital examination systems
Similaire à Some experiences from evaluating and stress testing digital examination systems (20)
Plus de Per Olof Arnäs
Plus de Per Olof Arnäs (20)
Dernier
Dernier (20)
Some experiences from evaluating and stress testing digital examination systems
- 1. Some experiences from evaluating and stress-testing digital examination systems Per Olof Arnäs Chalmers University of Technology
- 2. I do not like exams. really
- 3. Why do we give exams?
- 4. Why do we give exams? To teach? To pass verdict?
- 5. Why do we give exams? To pass verdict? Yes, in a sense. We are obligated to evaluate knowledge.
- 6. Why do we give exams? To teach? Yes! We are also obligated to transfer knowledge.
- 7. Me I love the 21st century Senior lecturer inlogistics at ChalmersUniversity of Technology Co-produced two MOOCs in Logistics Research on thedigitalization of thetransportation industry Per Olof Arnäs
- 8. My beliefs Society does not want test-takers Understanding is muchmore important thanknowledge of facts Written exams is often a badway to ensure knowledge and understanding My job is to make mystudents understandthe subject better andfaster than I did as astudent myself Digital development is key to human evolution
- 9. Active learning Blended learning • Students are involved more than listening • Less emphasis is placed on transmitting information and more on developing students’ skills • Students are involved in higher-order thinking (analysis, synthesis, evaluation) • Students are engaged in activities (e.g. reading, discussing, writing) • Greater emphasis is placed on students’ exploration of their own attitudes and values (Bonwell and Eison 1991) Bonwell, C. C. and J. A. Eison (1991). Active Learning: Creating Excitement in the Classroom. 1991 ASHE-ERIC Higher Education Reports, ERIC. Garrison, D. R. and H. Kanuka (2004). "Blended learning: Uncovering its transformative potential in higher education." The Internet and Higher Education 7(2): 95-105. Blended learning uses a combination of face-to-face learning with asynchronous content (on the internet) and has a large transformative potential (Garrison and Kanuka 2004).
- 10. Why do we give exams? To teach? This means that the exam should be part of the learning process of the student!
- 11. The short version: It’s a definite upgrade from analog
- 12. The ultra short version: The Devil is in the details
- 13. The cost of things
- 16. Economy-of-Scale
- 17. Learning curve for the teacher
- 18. Fixed costs are going down
- 19. Administration cost is lowered (a lot)
- 22. Three processes Student Teacher Admin
- 23. Three phases Before During After
- 24. Students before exam Preparations Download software Do test exam HardwareOK? Borrow computer?
- 25. Students before exam + Ability to test system - Learning curve Some planning needed
- 27. Students during exam + Faster than analog No handwriting Copy/paste Higher quality of answers - Some prefer analog Connectivity issues Compatibility issues Math Images
- 29. Students after exam + Rich feedback Swift feedback Learning opportunity - ?
- 30. Teachers before exam Create examQuestions Feedback texts Solutions Prepare students Hardware? Software? Question types
- 31. Teachers before exam + Re-use questions Multiple teachers - Prepare students Some question types still missing Enter solution when entering question
- 32. Teachers during exam Stand-by Answer questions Assist in onboarding
- 33. Teachers during exam + Message to all students via system - ?
- 35. Teachers after exam + Multiple graders Automatic grading Fast grading of essays Rich feedback! - Need for mouse Want to add same feedback texts to all students PDF-comments Sorry, but this sucks!
- 36. Admin before exam Room(s) People Extra hardware Outlets Wifi Training Support Management
- 37. Admin before exam + Centralised management - Special rooms Trained personnel Computer lending
- 38. Admin during exam Security Support Wifi security Visible screens Onboarding Lending of computers Offline support
- 39. Admin during exam + Digital overview Secure - New type of security needed Connectvity issues Power failures
- 41. Admin after exam + Fraction of workload compared to analog - Manual transfer of results to LMS Infinite archive
- 42. Some ideas GradingCreatingNew, non-additive question types Solution when creating Low friction UI Library of feedback texts Show total score “Smart”support
- 43. Question type: “Correct path” Check all correct alternatives (none, one, or more). 3 points if all are correct, -1 point per error (min 0 points) A. Alternative 1 B. Alternative 2 C. Alternative 3 D. Alternative 4 E. Alternative 5 •The student needs to find the “correct path” and gets penalties for each failure until 0 •Tests many things at once. •Should be automatically graded (not possible today). •Differentiates students •Hard to guess
- 44. Question type: “Correct path” Check all correct alternatives (none, one, or more). 3 points if all are correct, -1 point per error (min 0 points) •The student needs to find the “correct path” and gets penalties for each failure until 0 •Tests many things at once. •Should be automatically graded (not possible today). •Differentiates students •Hard to guess TRUE FALSE Statement A Statement B Statement C Statement D Statement E
- 45. Error^2 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Error 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Risky behaviour does not pay Question type: Mean Squared Error A statistical risk function where an estimator (=the student) is tested. For each question, the error (between 0 and 1) is squared The mean of all the errors form the Mean Squared Error - MSE
- 46. Usage of results Question 1 Question 2 Question 3 MSE Student 1 0 0 0,6 0,12 Student 2 1 1 1 1 Student 3 0 0,8 0,4 0,27 MSE 0,33 0,55 0,51 Rank 3 1 2 Theirresults My results
- 47. Feedback films Second loop Statistics Answers
- 48. Mean Squared Error - Goals Double loop Ensure understanding Lowworkloadfor me Scalable Honesty Reflection Think - don’t guess Quick feedback Should be digital!
- 49. Grading - some ideas Word cloud from essays “Flagged” words Highlight in answers Canned feedback “Learn” during grading Verbal/videocommenting Secondopinion Send question to colleague
- 50. Grading - some ideas See statistics Per question Diagrams See totals for anonymous students
- 51. Insights - digital exams… + Save money Free up time Are scalable Can be used to increase learning - Need physical infrastructure Need new competences Difficult to go back…
- 52. Why do we give exams? To teach! This means that the exam must be part of the learning process of the student!
- 53. Some experiences from evaluating and stress-testing digital examination systems Per Olof Arnäs Chalmers University of Technology per-olof.arnas@chalmers.se Thank you!